Data Center Wisdom, learning from past mistakes

July 22, 2010 Dave Ohara

Wisdom can be interpreted by people in different ways. A classic definition is

Wisdom is a deep understanding and realizing of people, things, events or situations, resulting in the ability to choose or act to consistently produce the optimum results with a minimum of time and energy

MSNBC has an article about the CIA bringing back a retired spy

Retired spy hired to run CIA clandestine service

by ADAM GOLDMAN, MATT APUZZO

WASHINGTON — The CIA brought one of its most experienced spies out of retirement to run the far-flung U.S. intelligence network, the agency said Wednesday.

John D. Bennett witnessed the emergence of al-Qaida in Africa in the 1990s and was on the front lines of the war on terrorism as the station chief in Pakistan. From his seat in Islamabad, he oversaw the unmanned Predator drone program, which has become the agency's most successful weapon against terrorism.

"John has impeccable credentials at the very core of intelligence operations — espionage, covert action, and liaison," CIA Director Leon Panetta said. "He has been at the forefront of the fight against al-Qaida and its violent allies."

What is a bit controversial is he surpassed the professional managers who were jockeying for the job.

Until his retirement in May, Bennett, 58, was the agency's most senior station chief, having served four tours in that position, including in Pretoria, South Africa. In taking this new post, he leapfrogged several other senior CIA officials who had been angling for the coveted job.

WSJ covers Boeing bringing back retired engineers.

The mission of these retired elder statesmen is to brainstorm with current Boeing engineers and project managers. But their advice often comes packaged in blunt terms that reflect their willingness to call things as they see them.

As a result, Boeing's past managers occasionally have been derided as "bean counters," and engineers on a failed project have been jestingly accused of "smoking marijuana." That's language not often heard in the buttoned-down Boeing empire.

One good thing about guys who have retired, they have no fear of upsetting the system as they have already quit.

Quick-witted and sturdy, Mr. Sutter has a no-holds-barred attitude and strong opinions about the company's past, present and future. Asked whether the Sonic Cruiser—a short-lived proposal in 2001 for a large jet that would travel at nearly the speed of sound—was a real effort or simply a smokescreen to confuse European rival Airbus, Mr. Sutter answered without hesitation. "It was a real effort," he said, "by people who were smoking marijuana."

Ideally if you are going to green a data center, you would want to look for data center wisdom. But, where do you find the wisdom? Internal groups, consultants, and vendors who know part of their job is to hide the mistakes.

Maybe it is time to a lesson from ancient Chinese wisdom.

Ancient Chinese Wisdom, of Mencius and Others: Learning From Mistakes

In Ancient China, correcting one's mistakes was considered a moment of revelation and a key element in one's character development

Epoch Times StaffCreated: Dec 3, 2008Last Updated: Dec 10, 2008

Mencius (372 – 289 BCE), an ancient Chinese philosopher, held that human beings were inherently moral. (Public domain image)

Older generations of Chinese people believed that even a saint could commit an error. Ancient texts mirror the adage that no man is infallible of making mistakes, but learning from them is one of man's greatest virtues. This leads to the betterment of one's conduct. Confucius was wont to say, “If you know you made a mistake and don't correct it, then you have really made a mistake.”

To err is human, but to recognize the error and correct it and learn from it gains respect from others.

Neglecting to do so, or even hiding the mistake, leads to loss of self-respect and the respect of others. Zi Chang, one of Confucius' students, said: “Human beings' mistakes are as blatantly obvious as a lunar or solar eclipse; everyone notices them, and when they disappear, people marvel in awe.”

Luckily I have hanged around enough data center people and technology projects to figure out where the wisdom is and isn't.

Wise men may not be learned; learned men may not be wise.

Message to the CFO, green the data center by asking for a multi-tier design, saving 15 - 20% - HP Video

July 22, 2010 Dave Ohara

HP has a video that illustrates some concepts for greening the data center. I don't think many of you have seen this video as YouTube shows there are only 64 views.

1) Multi-tier design. I am amazed at how many times people create data center space without thinking of the high rent and low rent district. The high availability space is more expensive and business units should be charged more to put IT services in these areas. If you don't provide a choice, then everyone will pay a higher cost. HP says they can save 15 - 20% data center costs with a mulit-tier design.

2) Modeling. Data Center Cost Optimization modeling is demo'd as an HP critical infrastructure solution. This starts at 3:30 min into the video.

HP Optimized Data Centers Simulation
Data Centers require a huge investment to design, build,
and then operate. It is critical to your business success
that your Data Center operates at peak efficiency to
achieve its business goals. Optimized Data Centers
Simulation from HP will help you do exactly that.

Two good ideas to improve data center design is to use multi-tiers in one facility and to model the design choices.

How many of your are cornered into one way with out the ability to see your choices?

Tip for building a better data center, work with the best salesperson

July 22, 2010 Dave Ohara

I have often joked that one of things people don't understand in the data center industry is many times the vendor selected goes to who has the best salesperson. The experienced data center professionals know the reality of what they need and are careful of sales tactics. With multi-million dollar equipment purchases the commissions are huge.

Ironically, some of the most arrogant people are the ones who are most vulnerable as a good salesperson can assess the arrogant ego easily and play them well.

exaggerating or disposed to exaggerate one's own worth or importance often by an overbearing manner

A good salesperson has the following skills.

* Cherish the client at all times

* Treat clients as you would your best friend

* Listen to clients and decipher their needs

* Make (or give) clients what they need

* Price your product to its dollarized value (in other words don't sell price sell the value received from purchasing)

* Give your clients more than they expect

* Thank each client sincerely and often

I was reading James Hamilton's blog about his boat and he makes an excellent point most miss.

If you do plan to request price quotes, be aware that if you approach a company without choosing a salesperson, one is assigned to you and this can be difficult to change later. Get feedback from other owners and explicitly choose one to work with before approaching the builder. The salesperson can have a major impact on the project, particularly if you plan major customizations.

In your effort to get the best design and best equipment, how many of you look for the best salesperson?

Changing the Site Selection game, being in Control, leveraging Mike Manos's post

July 21, 2010 Dave Ohara

Mike Manos is extremely busy now and doesn't blog as much as he used to, but when he does post he still gets lots of traffic. Mike and I were laughing once when a corporate data center blog was discussing proudly how many people they reached in a month with their blog. Mike said, "I get that much in less than a week." Which brings up a good point of how your view changes if you knew what you don't know. If they had known Mike gets 10x more traffic than them in a month, they'd wonder how influential they really were, and whether they are successful or not.

I always enjoy reading Mike's post, figuring out ways to use less words than he does, but also reading what Mike is trying to say, but hasn't put it down in words. Luckily leveraging multiple discussions with Mike, I can take a pretty stab at what he was thinking of when he was writing.

Mike posted July 14 on Site Selection.

Site Selection,Data Center Clustering and their Interaction

July 14, 2010 by mmanos

I have written many times on the importance of the site selection for data centers and its growing importance when one considers the regulatory and legislative efforts underway globally. Those who make their living in this space know that this is going to have a significant impact on the future landscape of these electronic bit factories. The on-going long term operational costs over the life of the facility, their use of natural resources (such as power) and what they house and protect (PII data or Personally Identifiable Information) are even now significantly impacting this process for many large global firms, and is making its way into the real estate community. This is requiring a series of crash courses in information security, power regulation and rate structures, and other complex issues for many in the Real Estate community.

What I think Mike is trying to say is it is much easier to build a better performing low cost data center with the right site. But, few understand the complex relationships that affect data center performance. I've always found it naive and over simplistic when companies and consultants use a long list of weighted criteria as the method to pick a site, assuming the highest score is the best site. This works for those who the most complex math they are comfortable with is multiplication and addition, but think about this hundreds of millions of dollars of CAPex, OPex, and IT equipment will be spent over a data center lifetime, and you are going to make the decision based on addition and multiplication?

I believe modeling techniques should be applied to ask the question "what is the right site?" And, Mike has his own mental models of what is right and wrong.

The right site for what? Pick 3 - 5 data center designs that you think you would want to build and use them as models to represent what you intend to build. If you have built the model with enough detail you should see the relationships that Mike talks about.

Tying into the power conversation is that of water. With the significant drive for economization (whether water based or air-based) water continues to be a factor. What many people don’t understand is that in many markets the discharge water is clean to dump into the sewage system and to ‘dirty’ to discharge to retention ponds. This causes all kinds of potential issues and understanding the underlying water landscape is important. The size of the metropolitan sewage environments, ability to dig your own well efforts, the local water table and aquifer issues, your intended load and resulting water requirements, how the local county, muncipality, or region views discharge in general and which chemicals and in what quantities is important to think about today. However, as the use of water increases in terms of its potential environmental scrutiny – water is quickly rising on the site selection radar of many operators and those with long term holds.

I have friends who designed a waterless cooling system in Australia due to the drought conditions. There was a cost associated for this data center design vs. cooling towers, but when you looked at the total system it was the right design.

If you really want to be advanced you can use semantic models.

Savanna is a model-driven analysis solution for solving complex problems. The magic of Savanna is in defining models that address what’s relevant to your problem at any given point in the analysis process. Savanna’s semantic models are driven by the Thetus Publisher architecture, enabling information synthesis by offering users the unique ability to derive meaning from information sets and to bridge the gaps in information. Savanna’s innovative, mind-mapping interface provides intuitive tools for approaching analysis from a point that frames the problem rather than one that starts from the information out.

Warning this technique works, but few have the capabilities to operate in this way. Using semantic models change the game as you focus on the problems and questions to ask, and enables you to see things others cannot.

To take control of site selection you need to have data center designs in mind for what you are building. If you are Mike Manos you can see the relationships of the site to the data center designs and how the system will operate.

If you can't do what Mike Manos does, then be prepared to make lots of mistakes even if you hire experts. Because you are not in control and are being told what to do. Do you think you can be a good cook by hiring a bunch of experts to tell you what to do? You need to be in control. Use data center designs to take control.

If you walk into a site selection consultant and say here are five data center designs I am looking at find me sites that support these designs. When you find me a site tell me which design works best and worst on the site. You'll find out whether the consultant can do more than addition and multiplication, and whether they really understand what a good data center site is.

Complexity of Data Center Construction Projects requires challenging the status quo, site selection example

July 20, 2010 Dave Ohara

I met with a friend who has worked in the high tech start-up environment and currently works in a company that has a data center presence, and he is thinking of a career change over to data centers. We spent a couple of hours discussing many topics, a who's who in the industry, dynamics of how the industry works, and systemic problems in the current ways.

A systemic problem is a problem due to issues inherent in the overall system,^[1]^[2] rather than due to a specific, individual, isolated factor. Contrast with pilot error, user error or mistake.

A change to the structure, organization or policies in that system could alleviate the systemic problem. On an Ishikawa diagram (fishbone diagram) of cause-and-effect links, the source of the problem can be said to be a common cause, rather than a special cause.

My friend showed me some jobs he was thinking of applying for, and some ideas started to gel together on problems the data center industry have.

One, data centers are computers. Google threw this idea out there.

Google: The Data Center Is the Computer

By Stacey Higginbotham Jun. 15, 2009, 11:47am PDT 1 Comment

61

As folks increasingly store and access information online, the data centers powering cloud services need to be managed more like a single computing entity rather than a bunch of servers, according to a Google white paper (Google calls it a mini-book) released today.

The paper lays out the concept of warehouse-scale computers (which we have previously referred to as both web-scale computing and mega data centers), specifically how to build out the infrastructure to support Internet services managed across thousands of servers. Google’s Luiz Barroso, a distinguished engineer, and Urs Hölzle, SVP of operations, both of whom help manage and build out Google’s data center, lay out their definition of WSCs:

Most data center projects are projects where fiefdoms battle for control of budget and resources - facilities, real estate, IT ops, application groups, and various users.

Most projects start with a data center project approval at the executive level and an approval of a budget. Then the real estate group takes control on site selection. Consultants are hired for the site selection process to figure out the right data center site. Lots of hours are spent interviewing people making the project more complex and more expensive.

Each group maneuvers to be the owner of a piece, and it sets you down a path without an overall design.

How many site selection companies are thinking like Mike Manos's post on site selection?

Site Selection,Data Center Clustering and their Interaction

July 14, 2010 by mmanos

I have written many times on the importance of the site selection for data centers and its growing importance when one considers the regulatory and legislative efforts underway globally. Those who make their living in this space know that this is going to have a significant impact on the future landscape of these electronic bit factories. The on-going long term operational costs over the life of the facility, their use of natural resources (such as power) and what they house and protect (PII data or Personally Identifiable Information) are even now significantly impacting this process for many large global firms, and is making its way into the real estate community. This is requiring a series of crash courses in information security, power regulation and rate structures, and other complex issues for many in the Real Estate community.

...

The Generation mix of that area has a large nuclear component which has little to no carbon impact, and generates long term stability in terms of power cost fluctuations. According to Phil Horstmann, President of Ascent, their is tremendous interest in the site and one of the key draws is the proximity of its nearby neighbor. In the words of one potential tenant ‘Its like the decision to go to IBM in the 80s. Its hard to argue against a location where Microsoft or Google has placed one of its facilities.’

How can a data center site be picked without a data center design? Do you pick a data center design and then figure out sites? If you have not picked your cooling system design, how do you know the affect of the water supply is and the impact of water on your design.

Tying into the power conversation is that of water. With the significant drive for economization (whether water based or air-based) water continues to be a factor. What many people don’t understand is that in many markets the discharge water is clean to dump into the sewage system and to ‘dirty’ to discharge to retention ponds. This causes all kinds of potential issues and understanding the underlying water landscape is important. The size of the metropolitan sewage environments, ability to dig your own well efforts, the local water table and aquifer issues, your intended load and resulting water requirements, how the local county, muncipality, or region views discharge in general and which chemicals and in what quantities is important to think about today. However, as the use of water increases in terms of its potential environmental scrutiny – water is quickly rising on the site selection radar of many operators and those with long term holds.

The industry standard is you pick a site, spent million dollars plus on site selection consultants, buy a site after a long evaluation period. Then, the rest of the system never says a word about you picking a bad site.

I get a good laugh whenever people criticize another person's site selection.

One of the ways you can change site selection is to bring in the top data center construction companies to evaluate your site before you make your final selection. "here are the 2 - 3 sites I am looking at to build a data center. Can you tell me the pros and cons for the sites?"

I got this idea talking to data center construction executive when he said they had won a new construction project and thank god the customer didn't pick the site in another state that was a crappy site. "Why don't you get involved before a site is selected?" After more chatting, figured out the site selectors want to close the deal with a site selection process they control. Bringing in data center construction experts decreases their control and shifts the control to the customer. This is bad for the commercial real estate process. We need to remove conflicts on interest by keeping suppliers who would influence the decision process isolated. BS!!!

If you take the idea to the next step, why not invite the electrical contractors to the site as well. Electrical systems are the dominant cost in a data center project. Site characteristics will affect the electrical systems and the costs. Why not get the electrical contractors input too. Cooling and water would be easy to add.

Hope this gets you thinking on how to change your site selection process.

A holistic approach thinking of the data center as a computer is better than following the status quo. A process of hiring a bunch of experts who have limited knowledge on how data centers are computers.