Lee Tech's Top 10 Data Center Operating Mistakes, Part 2 updated for an architecture order

July 26, 2010 Dave Ohara

I wrote on Lee Tech's Top 10 Data Center Mistakes, and made the point of reordering the 10 mistakes for an architecture approach.

I like the list, but I would change the ordering to create an architecture approach for looking at the issues. 8, 2, 10, 4, 1, 9, 3, 6, 7, 5 is a quick pass at an order I would choose, but I admit this is 3 minutes of thinking about it.

You can go to the updated post here.

Dave, you make two really great points:

I think the value of a team discussion is enormous as each customer is different both in requirements, constraints and at what point in the process they currently find themselves. This discussion would bring out multiple view points and arrive at an even better place than they had started.

I had not thought in terms of architecture when sequencing the list, and your three minute analysis is great. I would make one small change by moving 4 to follow 3 but could see the argument on both sides.

So if you think in terms of building a foundation on which each would progressively build, quality should be engineered into every system and process to ensure you can provide a sustainable solution. Too many data centers have been subjected to enormous impacts that could have been avoided either during construction or operation. Often, I hear about data centers no more than five years old having enormous infrastructure issues. So if I wrote the paper over again, I think I would adopt your approach and sequence the list as follows:

Big Mistake #8:
Failure to develop and implement Quality Systems

Big Mistake #2:
Relying too much on data center design

Big Mistake #10:
Thinking you can build a best in breed program as quickly as a data center

<the remaining 7>

Click this link to get the other rest of the order.

I could have asked to have this embedded as a comment, but I don't spend much time supporting comments on GreenM3.

Lee Kirby - thanks for updating your document.

Meditating, Higher Consciousness, Situation Awareness, Seeing Things in Data Centers most miss

July 26, 2010 Dave Ohara

I had one of my local high tech friends come by for an afternoon chat. It was the first time he had come over, and we spent a couple of quiet hours discussing all kinds of topics including data centers. My friend appreciated the Zen and quiet time I get working from home. Whenever someone suggests travel, I try to figure out how I can skip the plane flight. :-)

The next day I got up in the morning, and started writing.

This is the view to the south at the same time the sun was rising above.

As the evening was approaching, and the moon was almost full. I was using the quiet time to meditate, and try some higher consciousness exercises.

The concept of higher consciousness rests on the belief that the average, ordinary human being is only partially conscious due to the character of the untrained mind and the influence of 'lower' impulses and preoccupations. As a result, most humans are considered to be asleep (to reality) even as they go about their daily business.

As the night progressed, I decided to have my own outdoor movie and brought out my laptop to stream a Netflix movie, "The Sensei"

A DIFFERENT KIND OF MARTIAL ARTS FILM

A Different Kind of Martial Arts Film: D. Lee Inosanto’s ‘The Sensei’ battles prejudice and homophobia in 1980s small town in Colorado

If there’s one thing D. Lee Inosanto is no stranger to, it’s martial arts. Her father is martial arts legend Dan Inosanto, her godfather was the late Bruce Lee (whom she refers to simply as “Uncle Bruce”), and Inosanto herself is a highly trained martial artist who has worked as a stunt person on projects from Buffy the Vampire Slayer to Face/Off.

Then the next morning it hit the connection between the method of higher consciousness and situation awareness.

Situation awareness, or SA, is the perception of environmental elements within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future. It is also a field of study concerned with perception of the environment critical to decision-makers in complex, dynamic areas from aviation, air traffic control, power plant operations, military command and control, and emergency services such as fire fighting and policing; to more ordinary but nevertheless complex tasks such as driving an automobile or motorcycle.

Situation awareness (SA) involves being aware of what is happening around you to understand how information, events, and your own actions will impact your goals and objectives, both now and in the near future. Lacking SA or having inadequate SA has been identified as one of the primary factors in accidents attributed to human error (e.g., Hartel, Smith, & Prince, 1991; Merket, Bergondy, & Cuevas-Mesa, 1997; Nullmeyer, Stella, Montijo, & Harden, 2005). Thus, SA is especially important in work domains where the information flow can be quite high and poor decisions may lead to serious consequences (e.g., piloting an airplane, functioning as a soldier, or treating critically ill or injured patients).

There are some people I am working with to apply the ideas of situation awareness (a higher level of consciousness) to the data center. Here is a situation awareness demo using Geographic Information System (GIS) information from ESRI.

Part of the fun things I am working on is with people who have a situation awareness, a higher level of consciousness of what is going on data centers. The challenge we have is discussing things we discover with people "who don't know what they don't know."

A Chinese Proverb states.

He who knows not, and knows not that he knows not, is a fool...shun him.

He who knows not, and knows that he knows not, is willing...teach him.

He who knows, and knows not that he knows, is asleep...awaken him.

He who knows, and knows that he knows, is wise...follow him.

How many data center errors/mistakes could be prevented if systems and methods are set up to support a situational awareness. A consciousness of what what is going on.

Eric Gallant, a Lee Tech employee commented on this concept.

Eric Gallant said...

Excellent thought Dave. Prior to getting into the data center industry, I operated nuclear power plants for the Navy. After countless hours standing watch in the engine room, my senses became tuned to the environment. I could recognize a change in the pitch of a steam turbine, detect the slightest hint of acrid odor from a switchgear section and identify suspicious vibrations in the deck plates through the soles of my boots. My analog senses were often more useful than the abundant digital meters and detectors that monitored plant conditions.
The same phenomenon can be found in experienced data center professionals. I’ve seen data center operators sprint from their offices before the first alarm sounds because of a barely perceptible change in the quality of the light. I’ve even seen an engineer diagnose a bad CRAH shaft bearing by pressing his forehead against the front of the running machine.
As a managers and engineers we focus on metrics. As the old saw goes, “you can’t manage what you can’t measure.” And a good deal of engineering is knowing how, what and why to measure. Perhaps we focus on the measurable and quantifiable to the detriment of our more visceral abilities.

Top 10 Mistakes/opportunities in Data Center Operations , my #1 implement quality system

July 23, 2010 Dave Ohara

Lee Technologies has put out a paper on Top 10 mistakes in Data Center Operations. They previously posted on Top 9 Data Center Design Mistakes, and I posted commenting my #1.

Having spent 30 years in high tech working at HP, Apple, Microsoft and consulting I've seen my share of mistakes. You can choose the "ignorance is bliss" strategy and as long as no one else notices, things are fine. Or you can look at a mistake as an opportunity.

Mistakes Merely Opportunities in Disguise
OfficePRO magazine, November/December 1998 issue

Accept your mistakes, accept yourself, and turn blunders and missteps into lessons learned

By Dr. Gene Sharratt and Eldene Wall, CEOE

Mistakes are a part of life. We all make mistakes, but the real mistake is not to learn from them. How can mistakes be turned into opportunities? Effective office professionals acknowledge that errors happen. Most importantly, they learn from their mistakes and move forward.

Each person reacts to mistakes differently, but it's natural to feel angry and disappointed when errors are made. While these are normal responses, your reaction to mistakes largely determines what you learn from them. Some people criticize and belittle themselves for their errors longer than necessary, which can be counterproductive to professional growth.

Why are mistakes so painful? Whether a huge and costly mistake, or a relatively minor one, individuals often feel a strong sense of personal failure. While criticism is usually painful and can even be traumatic, the personal disappointment a person feels can be devastating.

It can be hard to address mistakes as few want to discuss the topic as millions of dollars are spent in data centers, and too many people have seen people dismissed or unfairly punished for mistakes made.

One way to break through this barrier is look at the Top 10 mistakes in data center operations as a guide to run an inventory on where you are at.

Lee Kirby the paper author starts with a piece of data center wisdom.

How can you avoid making major mistakes when operating
and maintaining your data center(s)? The key lies in the
methodology behind your operations and maintenance
program. All too often, companies put immense amounts
of capital and expertise into the design of their facilities.
However, when construction is complete, data center
operations are an afterthought. This whitepaper explores
the top ten mistakes in data center operations.

For those of you who want to know what the top 10 are, here is the summary.

Big Mistake #1:
Not including your operations team in facility design

Big Mistake #2:
Relying too much on data center design

Big Mistake #3:
Failure to correctly address the staffing
requirement

Big Mistake #4:
Failure to train and develop your talent

Big Mistake #5:
Failing to consistently drill and test skills

Big Mistake #6:
Failure to overlay your operations program with
documented processes and procedures

Big Mistake #7:
Failure to implement appropriate processes and
procedures

Big Mistake #8:
Failure to develop and implement Quality Systems

Big Mistake #9:
Failure to use software management tools

Big Mistake #10:
Thinking you can build a best in breed program as
quickly as a data center

I like the list, but I would change the ordering to create an architecture approach for looking at the issues. 8, 2, 10, 4, 1, 9, 3, 6, 7, 5 is a quick pass at an order I would choose, but I admit this is 3 minutes of thinking about it.

You can use the Lee Tech paper in a staff meeting to discuss the Top 10 data center operations mistakes made by others and create your own order, where you are at, and whether the areas warrant investment.

I would start by asking whether you have a quality system (item #8) in place, and whether the quality group is rewarded for finding mistakes and providing early feedback.

Many companies err in thinking that process, once proven, is infallible.
Continuous improvement is the only way to ensure your data center
operations are efficient, reliable and cost effective. A program for quality
systems consists of two principles:
• Quality Assurance (QA): processes to ensure that errors are not
introduced into the system
• Quality Control (QC): measures taken at various stages of the
process to proactively identify problems that could potentially lead
to system failure

Or you can go with "ignorance is bliss" strategy

BTW, eliminating mistakes is another way to reduce the environmental impact for a greener data center. Look at the environmental impact of BP's mistake. Fewer mistakes made the less environmental impact.

Data Center Wisdom, learning from past mistakes

July 22, 2010 Dave Ohara

Wisdom can be interpreted by people in different ways. A classic definition is

Wisdom is a deep understanding and realizing of people, things, events or situations, resulting in the ability to choose or act to consistently produce the optimum results with a minimum of time and energy

MSNBC has an article about the CIA bringing back a retired spy

Retired spy hired to run CIA clandestine service

by ADAM GOLDMAN, MATT APUZZO

WASHINGTON — The CIA brought one of its most experienced spies out of retirement to run the far-flung U.S. intelligence network, the agency said Wednesday.

John D. Bennett witnessed the emergence of al-Qaida in Africa in the 1990s and was on the front lines of the war on terrorism as the station chief in Pakistan. From his seat in Islamabad, he oversaw the unmanned Predator drone program, which has become the agency's most successful weapon against terrorism.

"John has impeccable credentials at the very core of intelligence operations — espionage, covert action, and liaison," CIA Director Leon Panetta said. "He has been at the forefront of the fight against al-Qaida and its violent allies."

What is a bit controversial is he surpassed the professional managers who were jockeying for the job.

Until his retirement in May, Bennett, 58, was the agency's most senior station chief, having served four tours in that position, including in Pretoria, South Africa. In taking this new post, he leapfrogged several other senior CIA officials who had been angling for the coveted job.

WSJ covers Boeing bringing back retired engineers.

The mission of these retired elder statesmen is to brainstorm with current Boeing engineers and project managers. But their advice often comes packaged in blunt terms that reflect their willingness to call things as they see them.

As a result, Boeing's past managers occasionally have been derided as "bean counters," and engineers on a failed project have been jestingly accused of "smoking marijuana." That's language not often heard in the buttoned-down Boeing empire.

One good thing about guys who have retired, they have no fear of upsetting the system as they have already quit.

Quick-witted and sturdy, Mr. Sutter has a no-holds-barred attitude and strong opinions about the company's past, present and future. Asked whether the Sonic Cruiser—a short-lived proposal in 2001 for a large jet that would travel at nearly the speed of sound—was a real effort or simply a smokescreen to confuse European rival Airbus, Mr. Sutter answered without hesitation. "It was a real effort," he said, "by people who were smoking marijuana."

Ideally if you are going to green a data center, you would want to look for data center wisdom. But, where do you find the wisdom? Internal groups, consultants, and vendors who know part of their job is to hide the mistakes.

Maybe it is time to a lesson from ancient Chinese wisdom.

Ancient Chinese Wisdom, of Mencius and Others: Learning From Mistakes

In Ancient China, correcting one's mistakes was considered a moment of revelation and a key element in one's character development

Epoch Times StaffCreated: Dec 3, 2008Last Updated: Dec 10, 2008

Mencius (372 – 289 BCE), an ancient Chinese philosopher, held that human beings were inherently moral. (Public domain image)

Older generations of Chinese people believed that even a saint could commit an error. Ancient texts mirror the adage that no man is infallible of making mistakes, but learning from them is one of man's greatest virtues. This leads to the betterment of one's conduct. Confucius was wont to say, “If you know you made a mistake and don't correct it, then you have really made a mistake.”

To err is human, but to recognize the error and correct it and learn from it gains respect from others.

Neglecting to do so, or even hiding the mistake, leads to loss of self-respect and the respect of others. Zi Chang, one of Confucius' students, said: “Human beings' mistakes are as blatantly obvious as a lunar or solar eclipse; everyone notices them, and when they disappear, people marvel in awe.”

Luckily I have hanged around enough data center people and technology projects to figure out where the wisdom is and isn't.

Wise men may not be learned; learned men may not be wise.

Message to the CFO, green the data center by asking for a multi-tier design, saving 15 - 20% - HP Video

July 22, 2010 Dave Ohara

HP has a video that illustrates some concepts for greening the data center. I don't think many of you have seen this video as YouTube shows there are only 64 views.

1) Multi-tier design. I am amazed at how many times people create data center space without thinking of the high rent and low rent district. The high availability space is more expensive and business units should be charged more to put IT services in these areas. If you don't provide a choice, then everyone will pay a higher cost. HP says they can save 15 - 20% data center costs with a mulit-tier design.

2) Modeling. Data Center Cost Optimization modeling is demo'd as an HP critical infrastructure solution. This starts at 3:30 min into the video.

HP Optimized Data Centers Simulation
Data Centers require a huge investment to design, build,
and then operate. It is critical to your business success
that your Data Center operates at peak efficiency to
achieve its business goals. Optimized Data Centers
Simulation from HP will help you do exactly that.

Two good ideas to improve data center design is to use multi-tiers in one facility and to model the design choices.

How many of your are cornered into one way with out the ability to see your choices?