Electrostatic Discharge (ESD), Humidity and the Data Center

June 11, 2010 Dave Ohara

I was reading DataCenterKnowledge's post on Yahoo's Data Center revolution, and one area that I want to elaborate on is electrostatic discharge, humidity and IT equipment.

It’s been a long journey from third-party colocation space to the Yahoo Computing Coop. The journey started by addressing the accepted range of 45 to 55 percent humidification for data center space. “Where did this belief come from?” Noteboom wondered.

Noteboom went searching for references, and says he found only a few academic scenarios warning of humidity issues. For practical arguments for tightly controlled humidity for computer rooms, Noteboom said he had to go back to the era of punch-card data entry, when concerns focused on whether excess humidity might wilt the punch cards and make them difficult to process properly.

“In 2005 we built a data center without humidification control,” said Noteboom. “It was filled with 25,000 servers. We were worried about static, and had all these protocols for handling equipment. We don’t have them anymore. We’ve operated through an entire lifecycle in this data center, and we’ve seen no impact.”

In 1980 as a summer intern I worked at HP on the idea that electrostatic discharge was causing component failures. I was working with guys like Dick Moss who was an ESD expert.

Richard Y. Moss
Hewlett-Packard Co.,
Palo Alto, CA, USA

Two of the questions most asked by manufacturing managers are "How good must my ESD-control program be?" and "How much money can it save me?" These are reasonable questions, considering that managers must justify their decisions in terms of return-on-investment or payback period.

Unfortunately, the first question is difficult to answer and the second almost impossible. One rarely knows the extent of ESD damage in a process until after it has been eliminated.

There is a bunch of stuff I learned 30 years ago, zapping parts with ESD, testing parts for failure, digging holes in plastic IC components, poring fuming sulfuric acid to burn the plastic away, and looking at integrated circuits to see where failures were occurring. We knew places like Fort Collins, CO had a higher electrostatic discharge risk due to low humidity, and you addressed the problem with wrist grounding straps, anti-static work surfaces, anti-static floor treatments, and air ionizers to remove the statically charged particles.

Quite frankly I've thought it dumb that the Data Center industry is obsessed about humidity which was most likely caused by a concern over electrostatic discharge, and potentially paper punch cards. There are much more cost effective ways to remove ESD risk than humidity. And, even a high humidity doesn't completely eliminate ESD.

If you want to address ESD then following guidelines like this.

Principles of electrostatic safe handling

There are two simple principles we can use to protect ESD sensitive components from ESD damage:

Only handle sensitive components in an ESD Protected Area (EPA) under protected and controlled conditions

Protect sensitive devices outside the EPA using ESD protective packaging

How many other dumb things are being done because of its been done in the past?

OK this is rant for the day. I am calmer now.

Spring Cleaning in the Data Center, address that cable mess

June 11, 2010 Dave Ohara

The general rule of thumb is network gear comprises 10% of the power use in the data center, but the network cannot affect your power consumption by throttling IO on servers, and blocking air flow.

We've all seen this photo as cable nightmare with a massive cooling impact to the IT equipment.

As part of the conversation I had with Rackforce's Brian Fry was their strategy to use Cisco Nexus switches with only two cables per server.

Cisco has their own article on the cable problem that is closer to what can occur than the above photo.

Cleaning up the cable mess is not a pleasant task, but it does have huge impacts.

Experts estimate that as much as 15 percent of the cost of data center equipment now goes to cabling. Multiple networks with their own sets of cables force data centers to buy extra equipment just to manage all the necessary connections, Borovick says.

More cables also means more energy demands. Cisco estimates companies can trim 20 to 30 percent off their power bill by simply reducing the number of cables and other equipment connecting their servers.

Industry experts also say that more simplicity in cabling is necessary for virtualization, the great hope for modernizing data centers for 21st century communications. "You need to clean up before you can move," Borovick says.

She says the situation has gotten out of control because the data center has evolved as three technological islands of different networks and various servers and data storage systems, each with their own communications technologies.

As it gets close to the first day of Summer, you've postponed the spring cleaning at home. Cable messes are one of those things that get postponed as well.

Microsoft's Christian Belady says time to move on, PUE was the past, Time to optimize the WHOLE

June 11, 2010 Dave Ohara

I've had the pleasure of getting to know the two HP engineers who started PUE, Christian Belady at Microsoft and Chris Malone at Google. here is the HP presentation from March 2007 that Christian gave.

This references the Sept 2006 paper Chistian and Chris co-wrote on PUE.

PUE is over 4 years old.

Matt Stansberry has an interview with Christian to discuss what is next. And starts with discussing PUE.

Your data center efficiency metric PUE has been public for about four years. What do you think of adoption at this point?
Christian Belady: I think it's pretty good. If you talk to anyone running a major data center operation, they're using PUE. There are perceived issues with it, comments about people cheating on PUE reporting. But who cares?

Then jumps to what is the next metric.

What do you think about the next level of efficiency metric: measuring the useful work of the data center? Will that kind of metric be available in the future?
C.B.: This is going to take a lot of effort. Look at PUE and see how difficult it was to get buy-in from various stakeholders. A data center productivity metric will be an order of magnitude more difficult to get broad acceptance. It's a very complex metric, and I'm very supportive of it, but it will be really hard to get agreement.

What is Christian working on in Microsoft Research?

But I'm all about the interfaces: the big opportunities are not to dive deep in one area, but to look across disciplines. In my new team, the eXtreme Computing Group, I get to look at the opportunities across hardware, software, applications and security interfaces. What can we do if we really stripped all our legacy IT requirements? What if we blurred the lines between these disciplines and developed a new cloud ecosystem from ground zero. What could that ecosystem look like? How can we see an order of magnitude reduction in cost.

My interest always lies way out in the future. How do we change the game? All the guys in this series have demonstrated that they have made significant changes in the industry. My challenge to all of us is [to ask] how we take the next big step. That's what excites me more and that is what I am working on. Stay tuned!

It is interesting to see how two guys who started down the path of a data center metric like PUE now work on the bigger data center picture.

If you are down in the weeds, maybe you should pop your head up and look at the bigger picture and figure out how what you do affects the whole.

Google, Microsoft, Apple, and Amazon are thinking about the whole, and many others do too. Also, these are the people who know how to green their data center as they understand the environmental issues and how it affects the long term sustainability of their operations.

Who will be next for Government Regulation? Google, Facebook or Apple

June 11, 2010 Dave Ohara

One side affect of the Microsoft anti-trust action is the governments of world feel good about taking on technology companies with regulation. You go throughout history and technologies at first were not regulated - auto mfg, power generation, oil & gas, and healthcare. And, there is still people arguing there needs to be more legislation in these areas.

Microsoft fought the battle with the DOJ and EU and besides paying lots of money, is government regulated. SEC has a degree on financial regulation, but doesn't touch on the latest issues with Google, Apple, and Facebook.

You look in the news at Google's Wifi.

Privacy group to go to police over Google Wi-Fi data collection

Privacy International likens mistake to tapping phone without consent

By Carrie-ann Skinner, PC Advisor UK
June 10, 2010 11:26 AM ET

Privacy International has revealed it will approach the Police regarding Google's recent admission it had mistakenly collected data from unsecured Wi-Fi networks for the past three years.

The search enginer's error came to light after the German data protection authority audited the Wi-Fi data collected by Street View cars for use in location-based products such as Google Maps for mobile.

Apple's ad policy.

Earlier today we reported that Apple (NASDAQ:AAPL) had amended itsiPhone Developer Agreement, specifically clause 3.3.9 related to advertising networks. The amendment was written in such a way that it allowed Steve Jobs to make good on a promise he made at D8 to open up the SDK for third parties to collect data for services such as advertising. But at the same time it was worded very carefully so as to exclude Google’s AdMob mobile advertising network from the iPhone eco-system.

This was met yesterday with lots of grumbling from Google, and gnashing of teeth from Admob’s CEO. Whilst the wording of the clause excludes any rival smart phone maker, Google is the only real threat on the horizon that Apple probably cares about right now.

And bloggers like danah boyd make the point on Facebook being a utility, and utilities get regulated as they are monopolies. Being a Microsoft employee she knows what is like to be a monopoly.

Facebook is a utility; utilities get regulated

From day one, Mark Zuckerberg wanted Facebook to become a social utility. He succeeded. Facebook is now a utility for many. The problem with utilities is that they get regulated.

Yesterday, I ranted about Facebook and “radical transparency.” Lots of people wrote to thank me for saying what I said. And so I looked many of them up. Most were on Facebook. I wrote back to some, asking why they were still on Facebook if they disagreed with where the company was going. The narrative was consistent: they felt as though the needed to be there. For work, for personal reasons, because they got to connect with someone there that they couldn’t connect with elsewhere. Nancy Baym did a phenomenal job of explaining this dynamic in her post on Thursday: “Why, despite myself, I am not leaving Facebook. Yet.”

danah goes on, and makes points about society, utilities and choice.

I hate all of the utilities in my life. Venomous hatred. And because they’re monopolies, they feel no need to make me appreciate them. Cuz they know that I’m not going to give up water, power, sewage, or the Internet out of spite. Nor will most people give up Facebook, regardless of how much they grow to hate them.

Your gut reaction might be to tell me that Facebook is not a utility. You’re wrong. People’s language reflects that people are depending on Facebook just like they depended on the Internet a decade ago. Facebook may not be at the scale of the Internet (or the Internet at the scale of electricity), but that doesn’t mean that it’s not angling to be a utility or quickly becoming one. Don’t forget: we spent how many years being told that the Internet wasn’t a utility, wasn’t a necessity… now we’re spending what kind of money trying to get universal broadband out there without pissing off the monopolistic beasts because we like to pretend that choice and utility can sit easily together. And because we’re afraid to regulate.

And here’s where we get to the meat of why Facebook being a utility matters. Utilities get regulated. Less in the United States than in any other part of the world. Here, we like to pretend that capitalism works with utilities. We like to “de-regulate” utilities to create “choice” while continuing to threaten regulation when the companies appear too monopolistic. It’s the American Nightmare. But generally speaking, it works, and we survive without our choices and without that much regulation. We can argue about whether or not regulation makes things cheaper or more expensive, but we can’t argue about whether or not regulators are involved with utilities: they are always watching them because they matter to the people.

The political win to regulate Google, Apple, or Facebook is huge. You can argue whether it is right or not, but the fact is a group of politicians see regulating one of the companies as a career milestone that will set them for life, and allow them to break into private industry making more money or guaranteeing them more influence.

EPA vs. Carbon motivated Congress Members debating who sets Energy Policy

June 10, 2010 Dave Ohara

Alaska (Oil) and West Virginia (Coal) Senators are making statements that congress should set energy policy and not the EPA, and are attacking the EPA's scientific findings.

EPA Stripped of Authority Would Be Threat to Climate Change Bill

Posted by Bridgette Outten in The District23 hours ago

President Barack Obama is planning to veto a bill that — if it passes Congress — would strip the Environmental Protection Agency of the authority to regulate greenhouse gas emissions and derail efforts for a climate change bill.

Alaska Senator says.

GOP Alaska Sen. Lisa Murkowski will ask senators to approve the measure Thursday, which willreportedly signal the Senate’s official disagreement with the EPA’s finding that carbon is a danger and needs to be regulated. The proposal uses a technique that can’t be filibustered and is an official mechanism to disagree with the rulings of executive branch agencies, according to reports.

Support for the Murkowski is coming from both sides of the aisle as lawmakers dispute the EPA’s right to set energy policy.

West Virginia says.

“I have long maintained that the Congress — not the unelected EPA — must decide major economic and energy policy,” said Democratic Sen. Jay Rockefeller of West Virginia in a press release. “EPA regulation will have an enormous impact on the economic security of West Virginia and our energy future.”

The White House says.

The White House said this week that Murkowski’s proposal would “undermine the administration’s efforts to reduce the negative impacts of pollution and the risks associated with environmental catastrophes, like the ongoing BP oil spill.”

The EPA says.

The Miami Herald reported that EPA Administrator Lisa Jackson “had even harsher words:”

“She called the oil spill a ‘tragic reminder of the hazards of our oil addiction’ and accused Murkowski of undermining the agency’s efforts to zero in on large emitters, not small ones.

‘It would take away EPA’s ability to take action on climate change,” Jackson said. “And it would ignore and override scientific findings, allowing big oil companies, big refineries and others to continue to pollute without any oversight or consequence. Finally, it will result in exactly zero protections for small businesses.’”

As the NYTtimes reports the battle is for jobs in States with a high carbon impact.

Republicans voted in unison, with some arguing that the emission program would suffocate millions of jobs and others asserting that EPA's plan is an unparalleled power shift toward "unelected bureaucrats," weakening Congress. Altogether 47 lawmakers, including six Democrats, supported moving forward with a vote to reverse the agency rules.

This is an interesting consequence of 2 Senators from each State vs. the House of representatives for the population. The States get to argue for their own livelihood vs. the population at large.

Some think the Climate Bill is too hard this year, but next it could be done.

One Dem: Climate bill next year

But there's evidence for less optimism: 47 senators signaled discomfort with a federal policy reducing greenhouse gases. Six of them are Democrats, a margin of mutiny that, if transferred to a vote on climate legislation, would likely spell disaster.

They include: Sens. Evan Bayh of Indiana; Mary Landrieu of Louisiana; Ben Nelson of Nebraska; Jay Rockefeller of West Virginia; and Blanche Lincoln and Mark Pryor, both of Arkansas. All come from states that lean heavily on fossil fuels.

The final tally, combined with heightening campaign partisanship, seems to have convinced some cap and trade supporters that the climb is too steep this year.

"I think it's difficult to pass a big bill a few months before a big election," Sen. Dianne Feinstein (D-Calif.) said after the vote. "But I think it can get done next year."

Where should you build your next data center? What colocation facilities will be in high demand.

If you aren't thinking of a carbon impact and a green data center strategy you better get started soon.