Top 10 Mistakes/opportunities in Data Center Operations , my #1 implement quality system

July 23, 2010 Dave Ohara

Lee Technologies has put out a paper on Top 10 mistakes in Data Center Operations. They previously posted on Top 9 Data Center Design Mistakes, and I posted commenting my #1.

Having spent 30 years in high tech working at HP, Apple, Microsoft and consulting I've seen my share of mistakes. You can choose the "ignorance is bliss" strategy and as long as no one else notices, things are fine. Or you can look at a mistake as an opportunity.

Mistakes Merely Opportunities in Disguise
OfficePRO magazine, November/December 1998 issue

Accept your mistakes, accept yourself, and turn blunders and missteps into lessons learned

By Dr. Gene Sharratt and Eldene Wall, CEOE

Mistakes are a part of life. We all make mistakes, but the real mistake is not to learn from them. How can mistakes be turned into opportunities? Effective office professionals acknowledge that errors happen. Most importantly, they learn from their mistakes and move forward.

Each person reacts to mistakes differently, but it's natural to feel angry and disappointed when errors are made. While these are normal responses, your reaction to mistakes largely determines what you learn from them. Some people criticize and belittle themselves for their errors longer than necessary, which can be counterproductive to professional growth.

Why are mistakes so painful? Whether a huge and costly mistake, or a relatively minor one, individuals often feel a strong sense of personal failure. While criticism is usually painful and can even be traumatic, the personal disappointment a person feels can be devastating.

It can be hard to address mistakes as few want to discuss the topic as millions of dollars are spent in data centers, and too many people have seen people dismissed or unfairly punished for mistakes made.

One way to break through this barrier is look at the Top 10 mistakes in data center operations as a guide to run an inventory on where you are at.

Lee Kirby the paper author starts with a piece of data center wisdom.

How can you avoid making major mistakes when operating
and maintaining your data center(s)? The key lies in the
methodology behind your operations and maintenance
program. All too often, companies put immense amounts
of capital and expertise into the design of their facilities.
However, when construction is complete, data center
operations are an afterthought. This whitepaper explores
the top ten mistakes in data center operations.

For those of you who want to know what the top 10 are, here is the summary.

Big Mistake #1:
Not including your operations team in facility design

Big Mistake #2:
Relying too much on data center design

Big Mistake #3:
Failure to correctly address the staffing
requirement

Big Mistake #4:
Failure to train and develop your talent

Big Mistake #5:
Failing to consistently drill and test skills

Big Mistake #6:
Failure to overlay your operations program with
documented processes and procedures

Big Mistake #7:
Failure to implement appropriate processes and
procedures

Big Mistake #8:
Failure to develop and implement Quality Systems

Big Mistake #9:
Failure to use software management tools

Big Mistake #10:
Thinking you can build a best in breed program as
quickly as a data center

I like the list, but I would change the ordering to create an architecture approach for looking at the issues. 8, 2, 10, 4, 1, 9, 3, 6, 7, 5 is a quick pass at an order I would choose, but I admit this is 3 minutes of thinking about it.

You can use the Lee Tech paper in a staff meeting to discuss the Top 10 data center operations mistakes made by others and create your own order, where you are at, and whether the areas warrant investment.

I would start by asking whether you have a quality system (item #8) in place, and whether the quality group is rewarded for finding mistakes and providing early feedback.

Many companies err in thinking that process, once proven, is infallible.
Continuous improvement is the only way to ensure your data center
operations are efficient, reliable and cost effective. A program for quality
systems consists of two principles:
• Quality Assurance (QA): processes to ensure that errors are not
introduced into the system
• Quality Control (QC): measures taken at various stages of the
process to proactively identify problems that could potentially lead
to system failure

Or you can go with "ignorance is bliss" strategy

BTW, eliminating mistakes is another way to reduce the environmental impact for a greener data center. Look at the environmental impact of BP's mistake. Fewer mistakes made the less environmental impact.

Data Center Wisdom, learning from past mistakes

July 22, 2010 Dave Ohara

Wisdom can be interpreted by people in different ways. A classic definition is

Wisdom is a deep understanding and realizing of people, things, events or situations, resulting in the ability to choose or act to consistently produce the optimum results with a minimum of time and energy

MSNBC has an article about the CIA bringing back a retired spy

Retired spy hired to run CIA clandestine service

by ADAM GOLDMAN, MATT APUZZO

WASHINGTON — The CIA brought one of its most experienced spies out of retirement to run the far-flung U.S. intelligence network, the agency said Wednesday.

John D. Bennett witnessed the emergence of al-Qaida in Africa in the 1990s and was on the front lines of the war on terrorism as the station chief in Pakistan. From his seat in Islamabad, he oversaw the unmanned Predator drone program, which has become the agency's most successful weapon against terrorism.

"John has impeccable credentials at the very core of intelligence operations — espionage, covert action, and liaison," CIA Director Leon Panetta said. "He has been at the forefront of the fight against al-Qaida and its violent allies."

What is a bit controversial is he surpassed the professional managers who were jockeying for the job.

Until his retirement in May, Bennett, 58, was the agency's most senior station chief, having served four tours in that position, including in Pretoria, South Africa. In taking this new post, he leapfrogged several other senior CIA officials who had been angling for the coveted job.

WSJ covers Boeing bringing back retired engineers.

The mission of these retired elder statesmen is to brainstorm with current Boeing engineers and project managers. But their advice often comes packaged in blunt terms that reflect their willingness to call things as they see them.

As a result, Boeing's past managers occasionally have been derided as "bean counters," and engineers on a failed project have been jestingly accused of "smoking marijuana." That's language not often heard in the buttoned-down Boeing empire.

One good thing about guys who have retired, they have no fear of upsetting the system as they have already quit.

Quick-witted and sturdy, Mr. Sutter has a no-holds-barred attitude and strong opinions about the company's past, present and future. Asked whether the Sonic Cruiser—a short-lived proposal in 2001 for a large jet that would travel at nearly the speed of sound—was a real effort or simply a smokescreen to confuse European rival Airbus, Mr. Sutter answered without hesitation. "It was a real effort," he said, "by people who were smoking marijuana."

Ideally if you are going to green a data center, you would want to look for data center wisdom. But, where do you find the wisdom? Internal groups, consultants, and vendors who know part of their job is to hide the mistakes.

Maybe it is time to a lesson from ancient Chinese wisdom.

Ancient Chinese Wisdom, of Mencius and Others: Learning From Mistakes

In Ancient China, correcting one's mistakes was considered a moment of revelation and a key element in one's character development

Epoch Times StaffCreated: Dec 3, 2008Last Updated: Dec 10, 2008

Mencius (372 – 289 BCE), an ancient Chinese philosopher, held that human beings were inherently moral. (Public domain image)

Older generations of Chinese people believed that even a saint could commit an error. Ancient texts mirror the adage that no man is infallible of making mistakes, but learning from them is one of man's greatest virtues. This leads to the betterment of one's conduct. Confucius was wont to say, “If you know you made a mistake and don't correct it, then you have really made a mistake.”

To err is human, but to recognize the error and correct it and learn from it gains respect from others.

Neglecting to do so, or even hiding the mistake, leads to loss of self-respect and the respect of others. Zi Chang, one of Confucius' students, said: “Human beings' mistakes are as blatantly obvious as a lunar or solar eclipse; everyone notices them, and when they disappear, people marvel in awe.”

Luckily I have hanged around enough data center people and technology projects to figure out where the wisdom is and isn't.

Wise men may not be learned; learned men may not be wise.

Message to the CFO, green the data center by asking for a multi-tier design, saving 15 - 20% - HP Video

July 22, 2010 Dave Ohara

HP has a video that illustrates some concepts for greening the data center. I don't think many of you have seen this video as YouTube shows there are only 64 views.

1) Multi-tier design. I am amazed at how many times people create data center space without thinking of the high rent and low rent district. The high availability space is more expensive and business units should be charged more to put IT services in these areas. If you don't provide a choice, then everyone will pay a higher cost. HP says they can save 15 - 20% data center costs with a mulit-tier design.

2) Modeling. Data Center Cost Optimization modeling is demo'd as an HP critical infrastructure solution. This starts at 3:30 min into the video.

HP Optimized Data Centers Simulation
Data Centers require a huge investment to design, build,
and then operate. It is critical to your business success
that your Data Center operates at peak efficiency to
achieve its business goals. Optimized Data Centers
Simulation from HP will help you do exactly that.

Two good ideas to improve data center design is to use multi-tiers in one facility and to model the design choices.

How many of your are cornered into one way with out the ability to see your choices?

Tip for building a better data center, work with the best salesperson

July 22, 2010 Dave Ohara

I have often joked that one of things people don't understand in the data center industry is many times the vendor selected goes to who has the best salesperson. The experienced data center professionals know the reality of what they need and are careful of sales tactics. With multi-million dollar equipment purchases the commissions are huge.

Ironically, some of the most arrogant people are the ones who are most vulnerable as a good salesperson can assess the arrogant ego easily and play them well.

exaggerating or disposed to exaggerate one's own worth or importance often by an overbearing manner

A good salesperson has the following skills.

* Cherish the client at all times

* Treat clients as you would your best friend

* Listen to clients and decipher their needs

* Make (or give) clients what they need

* Price your product to its dollarized value (in other words don't sell price sell the value received from purchasing)

* Give your clients more than they expect

* Thank each client sincerely and often

I was reading James Hamilton's blog about his boat and he makes an excellent point most miss.

If you do plan to request price quotes, be aware that if you approach a company without choosing a salesperson, one is assigned to you and this can be difficult to change later. Get feedback from other owners and explicitly choose one to work with before approaching the builder. The salesperson can have a major impact on the project, particularly if you plan major customizations.

In your effort to get the best design and best equipment, how many of you look for the best salesperson?

Changing the Site Selection game, being in Control, leveraging Mike Manos's post

July 21, 2010 Dave Ohara

Mike Manos is extremely busy now and doesn't blog as much as he used to, but when he does post he still gets lots of traffic. Mike and I were laughing once when a corporate data center blog was discussing proudly how many people they reached in a month with their blog. Mike said, "I get that much in less than a week." Which brings up a good point of how your view changes if you knew what you don't know. If they had known Mike gets 10x more traffic than them in a month, they'd wonder how influential they really were, and whether they are successful or not.

I always enjoy reading Mike's post, figuring out ways to use less words than he does, but also reading what Mike is trying to say, but hasn't put it down in words. Luckily leveraging multiple discussions with Mike, I can take a pretty stab at what he was thinking of when he was writing.

Mike posted July 14 on Site Selection.

Site Selection,Data Center Clustering and their Interaction

July 14, 2010 by mmanos

I have written many times on the importance of the site selection for data centers and its growing importance when one considers the regulatory and legislative efforts underway globally. Those who make their living in this space know that this is going to have a significant impact on the future landscape of these electronic bit factories. The on-going long term operational costs over the life of the facility, their use of natural resources (such as power) and what they house and protect (PII data or Personally Identifiable Information) are even now significantly impacting this process for many large global firms, and is making its way into the real estate community. This is requiring a series of crash courses in information security, power regulation and rate structures, and other complex issues for many in the Real Estate community.

What I think Mike is trying to say is it is much easier to build a better performing low cost data center with the right site. But, few understand the complex relationships that affect data center performance. I've always found it naive and over simplistic when companies and consultants use a long list of weighted criteria as the method to pick a site, assuming the highest score is the best site. This works for those who the most complex math they are comfortable with is multiplication and addition, but think about this hundreds of millions of dollars of CAPex, OPex, and IT equipment will be spent over a data center lifetime, and you are going to make the decision based on addition and multiplication?

I believe modeling techniques should be applied to ask the question "what is the right site?" And, Mike has his own mental models of what is right and wrong.

The right site for what? Pick 3 - 5 data center designs that you think you would want to build and use them as models to represent what you intend to build. If you have built the model with enough detail you should see the relationships that Mike talks about.

Tying into the power conversation is that of water. With the significant drive for economization (whether water based or air-based) water continues to be a factor. What many people don’t understand is that in many markets the discharge water is clean to dump into the sewage system and to ‘dirty’ to discharge to retention ponds. This causes all kinds of potential issues and understanding the underlying water landscape is important. The size of the metropolitan sewage environments, ability to dig your own well efforts, the local water table and aquifer issues, your intended load and resulting water requirements, how the local county, muncipality, or region views discharge in general and which chemicals and in what quantities is important to think about today. However, as the use of water increases in terms of its potential environmental scrutiny – water is quickly rising on the site selection radar of many operators and those with long term holds.

I have friends who designed a waterless cooling system in Australia due to the drought conditions. There was a cost associated for this data center design vs. cooling towers, but when you looked at the total system it was the right design.

If you really want to be advanced you can use semantic models.

Savanna is a model-driven analysis solution for solving complex problems. The magic of Savanna is in defining models that address what’s relevant to your problem at any given point in the analysis process. Savanna’s semantic models are driven by the Thetus Publisher architecture, enabling information synthesis by offering users the unique ability to derive meaning from information sets and to bridge the gaps in information. Savanna’s innovative, mind-mapping interface provides intuitive tools for approaching analysis from a point that frames the problem rather than one that starts from the information out.

Warning this technique works, but few have the capabilities to operate in this way. Using semantic models change the game as you focus on the problems and questions to ask, and enables you to see things others cannot.

To take control of site selection you need to have data center designs in mind for what you are building. If you are Mike Manos you can see the relationships of the site to the data center designs and how the system will operate.

If you can't do what Mike Manos does, then be prepared to make lots of mistakes even if you hire experts. Because you are not in control and are being told what to do. Do you think you can be a good cook by hiring a bunch of experts to tell you what to do? You need to be in control. Use data center designs to take control.

If you walk into a site selection consultant and say here are five data center designs I am looking at find me sites that support these designs. When you find me a site tell me which design works best and worst on the site. You'll find out whether the consultant can do more than addition and multiplication, and whether they really understand what a good data center site is.