How big is a planned data center? Look at the Air Permitting for Diesel Generators, Microsoft receives permit for 13 generators in Quincy, WA

November 7, 2010 Dave Ohara

I have been having fun researching the top data center users, seeing how much power they use. One of the techniques I use is publicly disclosed information for USA data centers from the State Ecology groups who issue air permits for diesel generators.

Here is a news report on Washington State's ecology department issuing a permit for 13 diesel generators.

Generators OK'd at Microsoft data center in Quincy

The state Ecology Department granted Microsoft a permit to install 13 backup power generators for the expansion of its data center in Quincy in central Washington.

The Associated Press

QUINCY, Wash. —

The state Ecology Department granted Microsoft a permit to install 13 backup power generators for the expansion of its data center in Quincy in central Washington.

The diesel-powered generators would be used in case of an electricity outage.

The Columbia Basin Herald reports neighbors are concerned about air pollution from the generators. The department is requiring Microsoft to meet regularly with the school district and notify it when the generators will be running.

The state Ecology department is making this easier, creating their own news releases.

Microsoft Columbia Data Center receives air quality permit to expand

SPOKANE — The Washington State Department of Ecology (Ecology) has issued the final permit allowing Microsoft’s Columbia Data Center in Quincy to expand operations.

The final permit, called a “notice of construction” order, follows a five-week public review and comment period that included a public hearing. The final permit contains revisions to address the concerns people expressed for air quality.

The notice of construction order is a formal approval document that allows Microsoft to install 13 new backup generators for use during power failures to support the facility’s data servers. The generators are powered by diesel engines. When all the new generators are installed, they will add the capacity for an extra 32.5 megawatts of backup electricity. That’s in addition to the 60 megawatts already available from the existing 24 generators installed in 2008.

Diesel engine exhaust particulate is a toxic air pollutant. Because of this, Ecology required a thorough evaluation of the health risks posed by the expansion project. This evaluation, called a “third-tier review of the health impact assessment,” required approval by Ecology Director Ted Sturdevant before the generators could be installed. Sturdevant approved the third-tier review on Aug. 20, 2010, and recommended approval of the project pending input from the public.

Based on the nearly 40 questions and comments received, Ecology changed the final permit to enable the community of Quincy to be well-informed about scheduled times and duration of maintenance testing of the generators.

“The final permit is built on sound science and a thorough, expert review,” said Greg Flibbert, Ecology’s project manager for the Columbia Data Center permit. “The questions we received from the public were well thought out and helped us make sure we had covered all the bases for protecting air quality while also ensuring that the local economy can gain from the benefits of Microsoft’s expansion.”

Part of the process for Air Permitting is public comment cycle like the Microsoft Quincy permitting.

Public invited to comment on draft permit for Microsoft Columbia Data Center expansion

SPOKANE - The Washington State Department of Ecology (Ecology) invites the public to comment on a proposed “notice of construction” order (permit) for the expansion of the Microsoft Columbia Data Center, in Quincy. The notice is a formal approval document that allows the company to install 13 new backup generators for use during power failures to support the facility’s data servers. The generators are powered by diesel engines.

Diesel engine exhaust particulate is a toxic air pollutant. Because of this, Ecology required a thorough evaluation of the health risks posed by the expansion project. This evaluation is called a “third-tier review of the health impact assessment” and the director of Ecology must approve it before the generators are installed.

The Microsoft Columbia Data Center was built in 2008 after Ecology approved a permit for installing and operating 24 electrical generators, capable of producing 60 megawatts of emergency backup electrical power. The expansion would add 32.5 megawatts of backup electricity.

We'll how big the Dell data center is in its first phase when they get their air permits.

Facebook’s Topping-Out Ceremony at Prineville Data Center

November 4, 2010 Dave Ohara

I posted on a data center topping-out ceremony I attended and got a chance to write on topping-out beam.

Facebook just posted their own pictures of their topping out ceremony on Nov 1.

Part of the ceremony is the tree.

Here is what I found regarding the tree tradition.

An interesting piece of trivia is the symbolism for the Christmas tree.

The custom of decorating the uppermost point of the structure with an evergreen tree is a tradition that predates the structural-steel industry in America by hundreds of years and has old Northern European roots. Although the topping out tree has ancient roots there is no consensus among modern ironworkers as to what exactly the tree symbolizes, or when and how it came to be used by the ironworkers. According to The Ironworker, the union's official publication, "for some the evergreen tree symbolizes that the job went up without a loss of life, while for others it's a good luck charm for the future occupants"(1984:11). Other accounts attribute the tree as signifying simply that "we [ironworkers] did it" (Kodish, 1989:2).

Little scholarship has been published on this custom. Most of what has been published has appeared in newspapers, popular magazines and engineering trade journals. One can get a feel for the age and scope of such tree rituals from James Frazer who discusses tree worship extensively in The Golden Bough. (Indeed, the title of the book itself is an allusion to tree worship.) For example, in Chapter Ten, "Relics of Tree-Worship in Modern Europe," Frazer reports that it was common practice in spring or early summer for the people to go into the woods and cut branches and fasten them to every house (1922:139). Frazer further remarks, "The intention of these customs is to bring home to the village, and to each house, the blessings which the tree-spirit has in its power to bestow" (1922: 139). The evergreen tree's ability to survive the harsh Northern European winter must have made it a powerful life-affirming symbol.

Transactions/Watt go to zero, Paypal is latest victim of outage

November 4, 2010 Dave Ohara

Thinking about green in the data center can have people focusing on the their PUE and energy efficiency of their servers, but few are implementing transactions per watt dashboards. It is rare to find someone who discusses data centers and the energy efficiency of their transactions. The MPG of their data center.

Paypal CTO discusses their Oct 29 outage.

At around 08:07 am PT today, a network hardware failure in one of our data centers resulted in a service interruption for all PayPal users worldwide. Everyone in our organization was immediately engaged to identify the issue and get PayPal back up and running. We were not able to switch over to our back up systems as quickly as planned. We partially restored service by approximately 8:45 am PT and the issue was fully resolved by 9:24 am PT. A second service interruption started at around 11:30 am PT and was partially resolved at 11:55am with full recovery at 12:21pm.

When I read this description in reminds me of the Star Trek scene where Sulu can’t the Enterprise into Warp.

Hikaru Sulu: The fleet has cleared spacedock, Captain. All ships ready for warp.
Christopher Pike: Set a course for Vulcan.
Hikaru Sulu: Aye-Aye, Captain. Course laid in.
Christopher Pike: Maximum warp. Punch it.
Hikaru Sulu: [One by one, the rest of the star fleet jumps into warp drive, leaving the Enterprise behind. Sulu frowns at the console, puzzled]
Christopher Pike: Lieutenant, where is Helmsman McKenna?
Hikaru Sulu: He has lungworms, sir. He couldn't report to his post. I'm Hikaru Sulu.
Christopher Pike: And you are a pilot, right?
Hikaru Sulu: Very much so, sir.
Hikaru Sulu: [he trails off, hitting buttons]
Hikaru Sulu: Uh, I'm not sure what's wrong here.
Christopher Pike: Is the parking brake on?
Hikaru Sulu: Uh, no. I'll figure it out. I'm just...
Spock: Have you disengaged the external inertial dampener?
Hikaru Sulu: [Embarrassed. Without looking at anyone, he punches in the correct sequence] Ready for warp, sir.
Christopher Pike: Let's punch it.

From 8:07a to 8:45a there was no Warp drive for Paypal. Dozens of people looking at displays. Why are transactions not completing? We have power. Services are live. Is the parking break on?

StorefrontBacktalk provides more details.

Two major technology glitches in a row knocked PayPal offline on Friday (Oct. 29), preventing the alternative payment giant from processing any E-tailer transactions for 80 minutes. First a network hardware failure shut down all PayPal payments. Then the backup plan failed when a handoff to a secondary datacenter didn’t go smoothly.

StorefrontBacktalk provides a timeline of outage, switch to backup data center, switch back to primary, repeat outage issue, then back up. Then provides these words.

Like American Eagle, PayPal had a fallback plan. But it didn’t work the way it was supposed to. And though it had a technical plan (that didn’t work) for dealing with the outage, like Wal-Mart, PayPal didn’t have any plan at all for quickly notifying the people most affected (Wal-Mart’s store personnel, PayPal’s biggest E-Commerce partners).

The lesson about failed backup plans just keeps getting bigger. Yes, improbable failures can happen. When they do, failover plans can fail. And when that happens, you need a plan already in place to warn those affected in real time.

I predict over the next 5 years we will see an outage at scale that will cripple a company permanently. We saw this last year with T-Mobile Sidekick outage, and imagine it on a bigger scale.

T-Mobile Sidekick Data Outage Turns Into Epic Customer Data Fail

By Laura Northrup on October 11, 2009 4:00 PM

This time last week, we thought of the T-Mobile Sidekick data outage as a mere inconvenient outage, but a temporary one. We grossly misunderstimated how badly T-Mobile and Danger/Microsoft could screw things up.

It turns out that their promise that service would be restored "soon" actually meant "never."

Want to avoid the risk. Invest in better people and processes. Technology is what you use, not the answer to the problem.

The Green Data Center Battle, setting the standards

November 2, 2010 Dave Ohara

Many think the green data center topic is not important and have moved on to other issues, but consider this article from the Harvard Business Review.

Winning in the Green Frenzy

by Gregory Unruh and Richard Ettenson

Don’t let your competitors control what “sustainable” means in your industry.

Right now somebody, somewhere, is defining what sustainability means for your industry, business, and products. Almost everywhere you look—textiles, communications, agriculture, autos, high tech—green competition is shifting from a race to launch ecofriendly products to a battle over what constitutes a green product in the first place. The definition can vary from one industry, business, or product class to the next. But whatever your business, if you’re not engaged in the debate and in shaping the rules, you risk being assessed against sustainability standards you can’t meet. Worse, you may be left behind by a shrewd competitor that has strategically positioned itself as a certified paragon of the new green ideal.

HBR points to the coffee industry as an example of the battle.

Producing sustainability standards is a multiplayer melee we call the green frenzy, because it is like a feeding frenzy in the wild—a tooth-and-claw competition among a growing pack of stakeholders including environmental activists, think tanks, bloggers, industry associations, consultants, and your rivals, all clamoring to establish and impose their own green standards.

In the coffee industry, for example, more than a dozen standards currently compete, affecting everything from pesticide use to workers’ housing to bird friendliness. (Just one of these, the Rainforest Alliance sustainable agriculture certification for coffee production, has some 100 criteria.) Each of the various standards has a constituency working to define the benchmarks for “sustainable coffee.” Some are backed by nonprofits such as the Audubon Society and TransFair, others by companies such as Starbucks and Nestlé.

Imagine Greenpeace being one of the most vocal groups on what green data center standards should be.

What's in Dell's move to Quincy, WA? Competing with IBM and HP?

November 2, 2010 Dave Ohara

Wenatchee World has the story on Dell coming to Quincy, WA for a data center.

Secret's out? Dell headed here

Blog: Everyday Business

By Mike Irwin

November 2, 2010

Dell, the world's third-largest computer company, has purchased property in Quincy that could become the site of the Columbia Basin town's newest data center, the Seattle Daily Journal of Commerce reported this morning.
Underway for months, the top-secret purchase — known as Project Roosevelt — would add the computer manufacturer to Quincy's list of tech industry standouts, such as Microsoft and Yahoo.

DataCenterKnowledge has more details.

Officials with the Port of Quincy have confirmed that the deal is related to Project Roosevelt, the code name for a data center that could include grow to 250,000 square feet over time. The initial specifications call for 7 megawatts of power, ramping up to 30 megawatts over time. The search also considered sites in Douglas County.

Why is Dell building 7 MW with growth for 25 MW? Perot systems is the competitor of HP/EDS and IBM services. Perot systems doesn't have the data center inventory that HP and IBM has.

Converged infrastructure is a hot topic, and it is much easier to sell a converged infrastructure solution when you have the data center contract. The Dell data center is most likely a cost effective solution for hosting, gives a low carbon data center alternative, and becomes a showcase for Dell/Perot.

Dell announced today the acquisition of Cloud Solution SaaS company Boomi.

Dell to Acquire Boomi; Adds Industry’s No. 1 Integration Cloud™ Solution to SaaS Capabilities

Date : 11/2/2010

Round Rock, Texas

Dell today announced it has agreed to acquire Software-as-a-Service (SaaS) integration leader Boomi to help businesses reap the full value of cloud computing. Powered by its revolutionary AtomSphere technology, Boomi offers the industry’s only pure SaaS application integration platform that takes the cost and complexity out of integrating applications by allowing easy transfer of data between cloud-based and on-premise applications with no appliances, no software and no coding required.