News of Facebook's Raining inside its Data Center leaks 2 years after the event

When I was at the first Open Compute Summit I was sitting with a data center executive and when he heard about the air misters used in Facebook's 1st data center he said they were going to run into problems.  We don't hear about misters in Facebook's data center any more and news has leaked out what may have helped convince putting humidity into the data center has issues.

The Register broke the news in an interview with Facebook VP Jay Parikh.

Facebook's first data center ran into problems of a distinctly ironic nature when a literal cloud formed in the IT room and started to rain on servers.

Though Facebook has previously hinted at this via references to a "humidity event" within its first data center in Prineville, Oregon, the social network's infrastructure king Jay Parikh told The Reg on Thursday that, for a few minutes in Summer, 2011, Facebook's data center contained two clouds: one powered the social network, the other poured water on it.

"I got a call, 'Jay, there's a cloud in the data center'," Parikh says. "'What do you mean, outside?'. 'No, inside'."

There was panic.

"It was raining in the datacenter," he explains.

Facebook has made some changes to weatherproof components in the data center.

Some servers broke entirely because they had front-facing power supplies and these shorted out. For a few minutes, Parikh says, you could stand in Facebook's data center and hear the pop and fizzle of Facebook's ultra-lean servers obeying the ultra-uncompromising laws of physics.

Facebook learned from the mistakes, and now designs its servers with a seal around their power supply, or as Parikh calls it, "a rubber raincoat."

Following Eric Schmidt's fact check idea, shows Al Gore is either Blind or just trying to make his point

Al Gore spoke at the "How Green is Internet?" event at Google, and at the 1:10 mark he says that he doesn't see the word "climate" in the word cloud for the event.  Which then gets Al Gore launching into how important the climate is.

During Eric Schmidt's talk he starts out saying how great Google is because "anybody says something slightly off ... you check. What a way to live your life. ... someone says something, and you say oh that's kind of a surprise let me check and see if that is true."

Al Gore starts his talk looking for a way to say how people don't pay attention to climate.  Eric Schmidt starts his talk with the culture of questioning what others say.

Below is the word cloud that Al Gore refers to.  In alphabetic order right after change and before cost is the word "climate".  Climate is small, and Al Gore says it may be too small to see.  Al said he couldn't see it.  The list is alphabetical order, not random.

This illustrates a point that Eric Schmidt is to imagine what could happen with climate change if people had a culture to check the facts others say.

Some people are big Al Gore fans.  I think I am more of a fan of facts and data. :-)  I can see "climate" pretty clearly in the above graphic.  How many attendees or watchers of Al Gore's video would accept as a fact that "climate" was missing or too small to read.

Google gains access to Waze users, Waze gets access to Google's WW Infrastructure

It's official Google has acquired Waze.  I found this statement most interesting.

We’ll also work closely with the vibrant Waze community, who are the DNA of this app, to ensure they have what’s needed to grow and prosper.

In the old days you would say the top things people would go to the web was for weather.  I don't think Google has acquired a weather company.

I find myself accessing Waze way more than than looking at the weather.  I can look outside and see the weather and get a good idea what is going on within 10-20 miles.

I don't see the traffic issues until it hits me in the face.

The one thing that Google has over Facebook and others who were bidding for Waze is more servers WW than any other company.  The speed at which Waze can grow and expand and not worry about capacity, availability, and performance is beyond what they could have done on their own.  So even though the development does not move, you can expect Waze type of functionality to show up in Google Maps soon.

The Waze product development team will remain in Israel and operate separately for now. We’re excited about the prospect of enhancing Google Maps with some of the traffic update features provided by Waze and enhancing Waze with Google’s search capabilities.

And you know what this is a good use of Internet technology to allow users to better plan their driving.  Taking the shortest routes most likely saves significant fuel.  I think I see a future Google sponsored study of how Waze saves the millions of gallons of fuel a year. :-)

Strategy for getting the most out of an industry event. Case Example Google's Energy Summit

I don't know about you, but it can hard to focus on a day long industry event.  You start out ready, then your phone rings, txt msgs, and e-mail distracts you.  Some of the presentations are good.  Some have room for improvement.  Sometimes you just get bored and your mind drifts.

Let me walk through an example of what I choose as a strategy for Google's How Green is the Internet? event.  Here is the full agenda.

First I got there early.  The event started at 9a.  I got there at 8:45a - to allow for Mtn View traffic, get a close parking spot (you were told to park 1/2 mile away and bus you in, I parked 300 ft away on the street), maximize time to socialize, and chat with the event staff.

Out of the full list of speakers.  There were three I really wanted to listen to.  Urs Hoelzle's 10 min intro.  Jonathan Koomey's 20 min infrastructure of the Internet.  And, Eric Schmidt's 15 minute new digital age.  Al Gore and the rest were lower priority.  Active listening can be tiring.  I was looking for patterns that connected what Urs, Jon, and Eric were saying.  I figured what Al Gore would be saying would be his own agenda that really wasn't connected to the other three.  Also, in my opinion Urs, Jon, and Eric were some of the smartest guys in the room with a breadth of testing ideas and concepts.

Here is Eric's talk.

And, Jon's talk

Urs introduction isn't posted.  Urs did spend an hour with the media to answer questions and that session was not recorded.

The other three people I focused on where the Google event organizer, the executive sponsor, and media contact.  These three were making sure the event worked.  Chatting with them gives perspective on how to interpret the presentations and their opinion on the event and presentations.

When I was at 7x24 Exchange I was daily chatting the executive board of 7x24, the event staff, and even the photographer.  I do the same at Open Compute Summit and DatacenterDynamics.

This comes back to my journey through 10 data center conferences post.  The four conferences I mention above are all the ones where I feel like I use this strategy, and I get the most out of.

Can you see the influence and factions in the data center industry?

Part of what I have an interesting time is discussing the influence and factions in the data center industry.  It is hard to discuss the specifics in a blog entry especially when I want to build on a public disclosure.  I started writing this post a couple of days ago and didn't decide to publish until Chris Crosby threw up his inbreeding post.

 For our industry to continue to grow and mature we need to better understand the needs, not the buzzword wants, of our customers. At a time when we should be more open as an industry, it seems that we are only becoming more insular. While this may make us better banjo players, it doesn’t help us improve the product that we offer to our customers in any meaningful way.

So, let me discuss ideas from my own observations and general concepts.

I just posted on my 10 data center journey.  And one of the top criteria for attending an event is how is easy it to network.  Most conferences I exchange maybe 20 business cards at the most, many times only 10 or less.  When going to sessions, I'll quietly watching from a corner spot or if I want to really focus on the presenters I got up front where I can take pictures and I am not distracted by the audience.

One of the principles I use is to focus on location.  When I go to bfast I look for who is there and where is a good spot to see the people coming in for Bfast.  This allows me to see many people coming in and connect with people as they scope where to sit.  When I go to the bar I look for friends already there and where is a good place to get an overall view of the people.  When I go to receptions, I go early and look for the best table to sit/stand to watch the people.  If you try to wander around to talk to the people it can be frustrating and draining.  My strategy is to find the best spots to hang with some of the best people and just watch what happens.

The funny thing about this strategy is when you apply it to social situations it can look like you are uncomfortable as you don't move around and mingle.  You sit/stand and watch.  I've been lectured by my wife and her friends that I am introverted and it is hard for people to know me.  At 7x24 Exchange I was telling one of my friends this story, and he laughed so hard that I was being accused of being anti-social, introverted, and hard to talk to.  Well the truth is sometimes when I do get in social situations and I am not in a conversation, then I'll start working on thought experiments designing new features, iterating on different ways to solve the problem, more information I need, and I am perfectly happy just sipping on my wine watching people.  

Back to data center conferences, part of the observation is seeing those groups who stick together as a company.  I went to one social in LV at a hotel suite during Data Center World.  I got there mid party, which made it hard to find the right spot to watch people. I saw a friend by the food in the corner away from all the people and chatted.  The host came on by and we were poking of fun of the guys who were wearing the company polo shirts all collected together.  <ALERT: possible inbreed group.>  Then he said it was really hot as he came over to try and turn down the thermostat which was already set to 60 degrees.  I figured out I was in the 2 x 4 ft section of the suite that was comfortable as I was right next to the air conditioning vent.  New lesson, sometimes the best spot to be in a crowded party is where the air conditioning vents.  The cool air can work well to get people to come by and cool off.

When a large group of company employees from a vendor are together, then the thing to watch is their clients are most likely close by or their suppliers that they favor.  Seeing who interacts with who and spends time together gets you an idea who can work together.  One of the biggest mistakes made is when suppliers are picked that really don't get along with each other.  Sometimes worse case they really hate each other, and given the client has made the choices, they can't do anything about it.

There are those who are hunting for new business which is rarely successful, but often enough that some sales people will interject themselves wherever they think they have something they can.  This is when I am glad that I don't buy anything which means people leave me alone.  And, being a blogger it can work to scare off people who don't want to talk to press.  Sometimes I wonder if my friends use my press shield capabilities to keep people away.  :-)  Notice how the press badges are usually red.  Warning.  Don't talk to this person unless you are press trained and be careful what you say.

Some people are looking for new jobs.  At some point there are conversations on who has changed jobs, who is looking, who is hiring.  Do you see the talent?  <squirrel>  Who has had a reorganization which disrupts staff and execution.