See the Cargo Cult Science in Data Centers

Now you may assume you that the data center world is all built on science and facts.  But, the Data Center World is no different than the rest of the world, and unfortunately those who benefit from managing your perception may have little interest in science.

Richard Feynman presented a talk on Cargo Cult Science.

I think the educational and psychological studies I mentioned are examples of what I would like to call cargo cult science. In the South Seas there is a cargo cult of people. During the war they saw airplanes land with lots of good materials, and they want the same thing to happen now. So they've arranged to imitate things like runways, to put fires along the sides of the runways, to make a wooden hut for a man to sit in, with two wooden pieces on his head like headphones and bars of bamboo sticking out like antennas--he's the controller--and they wait for the airplanes to land. They're doing everything right. The form is perfect. It looks exactly the way it looked before. But it doesn't work. No airplanes land. So I call these things cargo cult science, because they follow all the apparent precepts and forms of scientific investigation, but they're missing something essential, because the planes don't land.

If you want a way to detect the Cargo Cult Science in data centers, a pretty good indicator is whether you can find what Richard Feynman tell the graduates to do to not be Cargo Cult Scientists.  How are they credible.

It's a kind of scientific integrity, a principle of scientific thought that corresponds to a kind of utter honesty--a kind of leaning over backwards. For example, if you're doing an experiment, you should report everything that you think might make it invalid--not only what you think is right about it: other causes that could possibly explain your results; and things you thought of that you've eliminated by some other experiment, and how they worked--to make sure the other fellow can tell they have been eliminated.

Details that could throw doubt on your interpretation must be
given, if you know them. You must do the best you can--if you know
anything at all wrong, or possibly wrong--to explain it. If you
make a theory, for example, and advertise it, or put it out, then
you must also put down all the facts that disagree with it, as well
as those that agree with it. There is also a more subtle problem.
When you have put a lot of ideas together to make an elaborate
theory, you want to make sure, when explaining what it fits, that
those things it fits are not just the things that gave you the idea
for the theory; but that the finished theory makes something else
come out right, in addition.

An example of the Cargo Cult Science is IO Data Centers Last Snowflake that 2013 will be the last non-standardized snowflake.  There are many others, but this is one of the more timely ones that just got picked up.

Uptime: CEO of IO reaffirms prediction of the “snowflake’s” last day

Slessman sticks by prediction of end of non-standardized data centers

The world will not see a single non-standardized “snowflake” data center built after May 2013.

There may be people excited about this, but where is the data to support this claim?

When i looked at Slessman's transcript from 2011 where he made this claim.  There is not a single mention of quality or errors in the talk.  What data is shown that every IO data center is the same.   Huh. IO data centers has redefined physics where there are no errors or quality issues with any of their build outs, so every data center is the same.

So, let's go with a brand that prides themselves on quality and german engineering - BMW.

BMW builds 450 cars a day in its South Carolina plant and I am going to point to a bunch of things that make me believe BMW more than IO Data Centers.

BMW has an analysis center to figure out what is wrong.

Analysis Center

At the Analysis Center, we ignore the BMW mystique, look past the dazzling lines and impeccable paint job. We strip down the BMW and take an honest look at our work. The naked truth in all its beauty is revealed.

The 60,000-square-foot Analysis Center is a fully functioning laboratory that allows us to examine and test every weld, every dimension, and every component on vehicles as they come off the production line. The Analysis Center covers three key areas of vehicle development: Functional Analysis, Manufacturing Analysis and Customer Feedback.

BMW has a 1MW data center to support its operations and analysis.

The BMW South Carolina factory is described in this National Geographic Video.  What is not shown in this short video is the camera inspection equipment and many others to detect quality and errors in manufacturing. 

The laws of physics are tough to beat and it is really hard to make hundreds of complex products be exactly the same.  The products are not the same, they are all a bit different.  The issue is whether the products perform within specifications and meet quality standards.

You can believe the Cargo Cult Science that 2013 will be the last snowflake and they will all be the same after that, but you may be like the villagers doing all the those things to bring the cargo planes back.  Wearing wooden headhphones, bulding runways and putting wooden planes out.

Or you can go with the view that every data center is a bit different and I want an engineering science backed team who can adapt the manufacturing process to give a quality performing product. 

Are you picking up the habits of the slowest player? Maybe you shouldn't follow the thought leaders

Many of you play Golf.   I don't.  i decided playing golf gave people (including in-laws) the right to grab 4 hours of my time was not something I had interest in.  Maybe if golf was 1 1/2 hrs I would play.

Watching Kevin Na play golf is painful, and few would follow his example.  Although all players have been frustrated by a slow group in front of them.  Almost all amateurs pick up some habit of the pros.

WSJ has a post on this problem.

But the Tour's pace of play is a problem for the rest of golf, since the pros serve as amateurs' primary role model for how the game should be played. We buy the clubs, balls and golf fashions that we do largely based on the pros' example, and the same goes for how everyday players line up putts, take practice swings, throw grass in the air and dither around the course like they're being paid by the hour. Survey after survey show that slow play is a major factor in creating ex-golfers.

Now, as much as you may think this is wrong.  Keep in mind who the tour serves.  The players.

The reasons why the Tour is unlikely to change its current pace-of-play system anytime soon are many and interconnected, but here's a good one to start with: meaningfully speeding up play would, in effect, penalize the Tour's slowest members where it hurts them the most, in their wallets. And the Tour, lest we forget, exists primarily for the benefit of its members.

Think about it.  Just because you are watching someone present at a data center conference should you follow their habits?  Many data center conferences, the #1 customer is the vendor and their needs.  A pro golf player's # 1 revenue is his sponsor money, not the winnings.

Consider Charles Barkley's controversial statement that he is not a role model.  Who is your role model for data centers?  The guys who have vendors sponsoring their performance should have you questioning whether it is best for you.

Building a Room to Think, high ceilings work

After three days in the bay area it is nice to get home.  Home is a place to reflect on the week's activities. To meditate.

Meditation is thought of as a self improvement, but I also use the technique of meditation to think of the data center industry.

Meditation is any form of a family of practices in which practitioners train their minds or self-induce a mode of consciousness to realize some benefit.^[1][2][3] 

Last week with the DCD Seattle event I got a chance to chat with a bunch of folks who were in town and it turns out some were in town longer than they expected as they flew out from the East Coast and where then going to Uptime Symposium. Fieldview Solutions's John Consoli was one of those who was sticking around so we decided to grab lunch on Friday.  I told him to take a cab over to my house, and we could go to lunch and i would give him a ride back to his hotel.  I gave Fred the 3 minute tour as a complete tour of the house, office, and beach house can take an hour, and is a workout walking down the 200 steps to the beach and back.

When I ran into Fieldview Solutions's CEO Fred Dirla in Santa Clara, he heard about my house and 30ft ceilings.  Actually the ceiling is 13ft, not 30. 

I spend more time than I expected in this room, even choosing to work in the room.  Why? I think being in a tall ceiling feels good.  WSJ has a post on the concept.

Today, it turns out, the real cutting edge of architecture has to do with the psychology of buildings, not just their appearance. Recently, scientists have begun to focus on how architecture and design can influence our moods, thoughts and health. They've discovered that everything—from the quality of a view to the height of a ceiling, from the wall color to the furniture—shapes how we think. 

...

It's not just color. A similar effect seems to hold for any light, airy space. In 2006, Joan Meyers-Levy, a marketing professor at the University of Minnesota's school of management, studied the relationship between ceiling height and thinking style. She demonstrated that, when people are in a high-ceilinged room, they're significantly better at seeing the connections between seemingly unrelated subjects. In one experiment, undergraduates came up with nearly 25% more connections between different sports, such as chess and basketball, when sitting in a loft-like space than in a room with an 8-foot ceiling. Instead of focusing on particulars, they were better able to zoom out and see what various things had in common.

I found the research paper by Joan Meyers-Levy on ceiling height.

...

We believe that the effects produced by high or low ceilings actually occur because such ceiling heights increase or
decrease vertical room volume, which in turn stimulates
alternative concepts and types of processing. Indeed, this
logic corresponds with Hall’s (1966, 77) earlier discussed
thesis that chapels versus cathedrals communicate our theorized (i.e., confinement vs. freedom-related) associations
“by virtue of the space they enclose.”

FYI, we did not specify 13ft ceilings.  It was the height we needed to make the garage above us be level with the road.  We have no regets having little choice, and made the height one of the main features of the room, and a great place to think.

Greenpeace's claims vs. the facts, Apple fan digs up some facts

Greenpeace and Apple are having a PR war regarding the Maiden Data Center.

Here is a post I found that the data center community is going to like.  The author goes beyond the claims made and looks at the logic of when things occur vs. the perception of events.

...

Furthermore, the Maiden land deal didn’t “just happen” over the course of a few fevered weeks following Greenpeace’s April charm offensive. Anyone that’s ever purchased land, especially agriculturally zoned land like Apple acquired to site their second solar array, knows it takes months to negotiate a deal, gain regulatory approval and process the sale.

Apple’s been working on this for months if not years and there’s a paper trail a mile wide showing exactly what they’ve been up to if anyone had chosen to follow it.

If Greenpeace is a public watchdog, what exactly have they been watching? My guess is they’ve been breathlessly watching themselves on TV.

The author then gets more critical.

In “How Clean Is Your Cloud,” Greenpeace claimed Apple’s NC data center would use 60MW of power and that the overwhelming majority of that would be coal-generated electricity from Duke Energy. However, Apple’s public statements and public filings clearly state the facility would use just 20WM — a difference of 300 percent.

Digging up more facts about energy composition.

Moreover, Greenpeace overstated Duke Energy’s reliance on coal (55 percent), citing 2007 data, despite the ready availability of 2010 data filed with regulators showing a greatly reduced coal footprint (45 percent) in its generating mix.

i wonder if the author of this post lives in NC.

And, the NCUC approval process included a public comment period and public hearings — Apple was on the ball and ready to deliver 100 percent green-powered data centers months ago and information showing that was readily available to the public.

Few including the media have spent this time researching the facts that exist in public disclosure.

I like green. I seek out green products and work hard to reduce, reuse and recycle. I despise Greenpeace…

AOL's 3 steps to save $5mil in IT Operations

I met a bunch of Mike Manos's IT team from AOL in Santa Clara.  They were in town for Uptime Symposium and we invited the AOL team to meet some of the thought leaders in the bay area.  We gave Mike's team a bunch of Mike Manos's stories as there were a lot of people who used to work in Mike's group at Microsoft and Digital Realty Trust. Plus his brother Steve Manos was there as well.

I've gotten to know Mike as I used to help him on his presentations.  I met the presenters Chisty and Julie, so besides sharing the slides and writing a post, let me take a stab at telling the AOL story in a different way.

The opening slide is this.

After going through the slides i would try "AOL's 3 steps to save $5mil in IT Operations"

Clearing the Cruft is not going to get people's attention.  Saving $5mil does.

The results are summarized here early on.  This is good.

The order of the three steps presented are AOL Cloud, Power Absurdity, and Power Hog.

I would change the order. Put Power Hogs first.  Find those wasteful things.

Some of these power hogs are not only wasteful, but also absurd.  Project Absurdity after Power Hogs

What is left should try to go to AOL cloud.  The last step is the stuff left over gets moved to the cloud.

Although I bet you there are a bunch of VMware consultants who will gladly move the Power Hogs and Power Absurd to the cloud as well.  The more VMs the more they make.

The list of steps taken are good, but I would reorder them and use it is a good chance to make fun of their absent leader Mike Manos.  Without an executive sponsor a project this is impossible.  I wonder if Mike calls his staff Donkeys for not taking up the challenge.

Excellent slide with energy savings and carbon reduction.

Overall this is a well thought out presentation.  I only suggest some changes in the order of the content, changing the title, and adding some humor.

Good job AOL team.  Hope to see you soon.