Green Data Center Blog

Where the Clouds Meet The Ground

The Economist has a feature on Cloud computing.  An audio is available here.

Nicholas Carr summarizes The Economist article.

The Economist tours the cloud

October 25, 2008

The new issue of The Economist features a good primer on cloud computing, written by Ludwig Siegele, which looks at trends in data centers, software, networked devices, and IT economics and speculates about the broader implications for businesses and nations. A free pdf of the entire report is also available.

Siegele notes that the hype surrounding the term "cloud computing" may have peaked already - Google searches for the phrase have fallen after a big spike in July - but that "even if the term is already passé, the cloud itself is here to stay and to grow. It follows naturally from the combination of ever cheaper and more powerful processors with ever faster and more ubiquitous networks. As a result, data centres are becoming factories for computing services on an industrial scale; software is increasingly being delivered as an online service; and wireless networks connect more and more devices to such offerings." The "precipitation from the cloud," he concludes (milking the passé metaphor one last time), "will be huge."

Part of the report is a specific feature on Data Centres.

CORPORATE IT

Where the cloud meets the ground

Oct 23rd 2008
From The Economist print edition

Data centres are quickly evolving into service factories

Illustration by Matthew Hodson

IT IS almost as easy as plugging in a laser printer. Up to 2,500 servers—in essence, souped-up personal computers—are crammed into a 40-foot (13-metre) shipping container. A truck places the container inside a bare steel-and-concrete building. Workers quickly connect it to the electric grid, the computer network and a water supply for cooling. The necessary software is downloaded automatically. Within four days all the servers are ready to dish up videos, send e-mails or crunch a firm’s customer data.

This is Microsoft’s new data centre in Northlake, a suburb of Chicago, one of the world’s most modern, biggest and most expensive, covering 500,000 square feet (46,000 square metres) and costing $500m. One day it will hold 400,000 servers. The entire first floor will be filled with 200 containers like this one. Michael Manos, the head of Microsoft’s data centres, is really excited about these containers. They solve many of the problems that tend to crop up when putting up huge data centres: how to package and transport servers cheaply, how to limit their appetite for energy and how to install them only when they are needed to avoid leaving expensive assets idle.

But containers are not the only innovation of which Mr Manos is proud. Microsoft’s data centres in Chicago and across the world are equipped with software that tells him exactly how much power each application consumes and how much carbon it emits. “We’re building a global information utility,” he says.

Engineers must have spoken with similar passion when the first moving assembly lines were installed in car factories almost a century ago, and Microsoft’s data centre in Northlake, just like Henry Ford’s first large factory in Highland Park, Michigan, may one day be seen as a symbol of a new industrial era.

US CTO Candidates, GOOG, AMZN, OR MSFT -Purpose Develop Environmental Friendly Tech

BusinessWeek writes on the idea of Obama’s idea for a US CTO.

The Short List for U.S. Chief Technology Officer

Barack Obama has pledged to name a cabinet-level CTO to oversee a job-creating national broadband buildout if he's elected. Big names abound

By Tom Lowry

Barack Obama says that the U.S. is not doing nearly enough to create jobs through technology. Shortly after he launched his campaign, the Illinois Senator promised that if elected, he would create the first-ever Cabinet-level post of chief technology officer. The economic crisis has since made it certain that a White House CTO would become one of Obama's most important advisers, should he triumph in November.

Candidates for the job are Google, Amazon, and Microsoft.

Among the candidates who would be considered for the job, say Washington insiders, are Vint Cerf, Google's (GOOG) "chief internet evangelist," who is often cited as one of the fathers of the Internet; Microsoft (MSFT) chief executive officer Steve Ballmer; Amazon (AMZN) CEO Jeffrey Bezos; and Ed Felten, a prominent professor of computer science and public affairs at Princeton University.

Broadband is one focus as well as a $50 billion VC fund for environmental friendly technology.

A White House CTO would be expected to help create incentive programs to expand broadband's reach, particularly tax credits for smaller carriers. But the tech czar would almost certainly be deeply involved in overseeing a federally-backed $50 billion venture capital fund that Obama has proposed to develop more environmentally friendly technology.

It is interesting to imagine a US CTO with one of these high tech executives. Tech company politics are taken to a new level.

An Application Architect who Cares about Energy Efficiency

Pat Helland put up his post about a week at Foo Camp. 

What is Foo Camp? Pat gives an entertaining summary.

A Weekend at Foo Camp

Well...  I was lucky enough to get invited to Foo Camp (which was last weekend) and I figure "What the heck!  Let's do it!".  

Foo Camp is held at the O'Reilly headquarters in Sebastopol, CA which is 1 to 2 hours (or more depending on traffic) north of San Francisco.  It is an invitation only event whose name stands for "Friends Of O'Reilly" and involves about 300 diverse and interesting individuals from different walks of the computer industry (and related industries).  Tim O'Reilly host it at their headquarters and supplies very nice buffet food, showers, rest rooms, and meeting rooms.  I was informed that the best way to enjoy the event is to camp there which means either pitching a tent on their lawn or finding an available space in a meeting room or hallway to throw a sleeping bag.  It was an option to get a hotel room in town.  Now... I haven't camped in about 20 years so this required some thinking... 

I concluded I had two options:

1. Stay in a hotel room and ensure I remained sober enough to drive at the end of the evening, OR
2. Buy a tent at REI in Santa Rosa and then hit the high-end liquor store in Santa Rosa to buy enough whiskey to lubricate a serious subset of the 300 attendees.

Naturally, I chose the second option and bought six bottles (some of my favorites and others I hadn't tried). 

It turns out James Hamilton and Pat Helland are friends and are putting their heads together on Energy Efficiency software. James is left, Pat is center, and Jesse Robbins is right.

 

But, back to the Energy Efficiency software.  Here are Pat's comments.

Sometimes, it was hard to select between the various cool sessions.   I remember the following ones:

• Data center power -- James Hamilton (my friend from Microsoft) and Jeff Hammerbacher who leads the Facebook data team (but will be leaving soon).  Both James and Jeff were filled with information about running large and dynamic data centers.   The power issues for data centers have been on my mind the last few years and I find that James is a wonderful font of knowledge.   I most definitely love that he is at Microsoft and my friend... I plan to come pepper him with additional questions in the months to com.   Jeff, also, has tons of knowledge from supporting the data needs of Facebook as it has undergone its explosive growth.  This was a fun and invigorating discussion in which I met an attendee, Roger Magoulas who is a research director at O'Reilly.   I have a feeling that there will be opportunities for me to work with Roger, too.
• Parallel Programming -- Kerry Hammil of Microsoft Research.   We had a fun discussion of the difficulties of getting applications (and, indeed, their libraries and OSes) to be parallel.  There were about 25 great and interesting people participating in this group and, not surprisingly, I participated, too.     This was such a lively discussion for me that it ended up in the hallway and we skipped the next session.

Here is an application architect who cares about data center power and parallel (multicore) programming. I hope we see more people like Pat show up.

Jesse Robbins in the above picture has an interesting background.

• Technical Program Manager at Amazon.com
• Manager - IT Operations at Amazon.com
• Task Force Leader at World Shelters Task Force 1

• Systems Engineer at Amazon.com
• Firefighter/EMT (Intern) at Palo Alto Fire Department

Giving Cloud Storage a Try

If you have been thinking about giving Cloud Storage a try here is a blog post on Amazon.com’s Simple Storage Service.

Amazon S3 (Simple Storage Service) is an easy and inexpensive Internet hard-drive from Amazon Web Services (AWS) with absolutely no limits.

Now some people assume that Amazon S3 is a storage service meant primarily for web start-ups who store data in-the-cloud but that’s not correct because just about anyone (home users included) can benefit from S3.

For instance, you may backup your large music collection or even your entire computer hard-drive on S3. Similarly, bloggers can use Amazon S3 to store web images without worrying too much about their bandwidth bills.

If you never had a chance to explore Amazon S3 before, read the following guide that makes S3 simple even for non-geeks. It has all the information and tools you would need to quickly get started with Amazon S3.

The Basic Requirements:

To use Amazon S3 service, you’ll need an Amazon account (yes, it’s free; you pay only if you buy something from them) and an Amazon AWS S3 account.

Now go to AWS Access Key Identifiers and generate a unique Access Key ID + Secret Access Key pair that is required for using Amazon S3.

The post continues with a variety of tools you can use.

Questions for Amazon Web Services Evangelist

If you are interested in Amazon Web Services, here is video of a presentation by Mike Culver.

I have a meeting with Mike in a week to discuss Amazon Web Services. Mike and I used to work together, and we have been running into each other at the Pro Club for the past 2 years.  We both left Microsoft 2 years ago, and Mike went straight to Amazon to  be an evangelist for AWS.

I’ll write a blog post after my meeting with Mike. If you have any questions you would like me to ask, feel free to comment on this post, and I’ll see what I can do to get answers.  Keep in mind, Mike is not in the data center operations group, but in the evangelism of web services.

One question I have for Mike are there any other tools besides Service Health Dashboard for AWS operations?

Two Different opinions, Why Amazon Went Down

You all know amazon.com suffered a 2 hour outage on Friday, suffering a 4.59% stock drop, lost sales, and damage to its reputation. I found 2 different opinions which give you some ideas on what happened.

GigaOm has a factual analysis looking for cause and effect.

So what happened? Let’s look at the facts.

• Traffic to https://www.amazon.com was getting there. So DNS was configured properly to send traffic to Amazon’s data centers. Global server load balancing (GSLB) is the first line of defense when a data center goes off the air. Either GSLB didn’t detect that the main data center was down, or there was no spare to which it could send visitors.
• When traffic hit the data center, the load balancer wasn’t redirecting it. This is the second line of defense, designed to catch visitors who weren’t sent elsewhere by GSLB.
• If some of the servers died, the load balancer should have taken them out of rotation. Either it didn’t detect the error, or all the servers were out. This is the third line of defense.
• Most companies have an “apology page” that the load balancer serves when all servers are down. This is the fourth line of defense, and it didn’t work either.
• The HTTP 1.1 message users saw shows something that “speaks” HTTP was on the other end. So this probably wasn’t a router or firewall.

This sort of thing is usually caused by a misconfigured HTTP service on the load balancer. But that would happen late at night, be detected, and rolled back. It could also happen from a content delivery network (CDN) not retrieving the home page properly.

So my money’s on an AFE or CDN problem. But as Berman notes, Amazon’s store is a complex application and much of their infrastructure doesn’t follow “normal” data center design. So only time (and hopefully Amazon) will tell.

Site operators can learn from this: Look into GSLB, and make sure you have geographically distributed data centers (possibly through AWS Availability Zones.) It’s another sign we can’t take operations for granted, even in the cloud.

and WSJ blogger has a more entertaining version.

“The Amazon retail site was down for approximately 2 hours earlier today (beginning around 10:25) - and we’re bringing the site back up.

Amazon’s systems are very complex and on rare occasions, despite our best efforts, they may experience problems. We work to minimize any disruption and to get the site back as quickly as possible.

Amazon’s web services were not affected nor were our international sites.”

The statement doesn’t explain what went wrong, however. Here are some possibilities the Business Technology Blog has come up with and — in honor of tomorrow’s Belmont Stakes — our carefully-calculated odds that it’s what caused the problem.

* An explosion, fire or some other mishap at one of Amazon’s data centers: 5 to 1
* A faulty software upgrade: 7 to 1
* A so-called denial-of-service attack that tries to overwhelm the site with traffic: 10 to 1
* The site was broken into the by same guys who “RoXed Comcast”: 100 to 1
* A rush on Amazon’s Kindle e-book reader: 1,000 to 1
* Sharks with laser beams on their heads: 1,000,000 to 1

Most people will focus on what needs to happen to prevent this from happening again. Or you can accept the fact that no matter what you do, outages will happen, and it is part of an online business. So, the challenge is how you handle the customer relations of an outage. The smart companies address both issues with a good analysis of the long term effects.  This coincidentally aligns with companies who think about green/sustainability action in the long term.  The last thing you want to do in going green is be reactive. 

Amazon Web Services, Service Health Dashboard

Found a cool AWS Web Site, reporting Service Health.

Amazon guys are thinking of customer support.

Amazon Web Services publishes our most up-to-the-minute information on service availability in the table below. Check back here any time to get current status information, or subscribe to an RSS feed to be notified of interruptions to each individual service. If you are experiencing a real-time, operational issue with one of our services that is not described below, please inform us by clicking on the "Report an Issue" link to submit a service issue report.

Jun 4, 2008

Report an Issue

Current Status
Details
RSS

Amazon Elastic Compute Cloud (API)
Service is operating normally.

Amazon Elastic Compute Cloud (Instances)

[RESOLVED] Severe Weather - Instances Unavailable

more

12:29 PM PDT We are currently experiencing severe weather near one of our Amazon Web Services locations. We are investigating impact and will provide updates on the dashboard.

1:01 PM PDT During the transition to backup generator power, we had a small number of EC2 instances in a single Availability Zone shut themselves down. We will be restoring these instances and will post here when this is complete. We continue to monitor the situation closely.

2:26 PM PDT A brief update to let you know that all instances affected by this issue have been restored. We'll also be following up with additional information about this event.

Amazon Flexible Payments Service
Service is operating normally.

Amazon Mechanical Turk (Requester)
Service is operating normally.

Amazon Mechanical Turk (Worker)
Service is operating normally.

Amazon SimpleDB
Service is operating normally.

Amazon Simple Storage Service (EU)
Service is operating normally.

Amazon Simple Storage Service (US)

[RESOLVED] Severe Weather - timeouts and 503s

more

1:52 PM PDT Amazon S3 US customers may currently be experiencing an elevated rate of timeouts and 503 errors due to the severe weather events near one of our locations. We are investigating.

1:56 PM PDT Quick update to let you know that we are currently in the process of recovering.

2:14 PM PDT Recovery is proceeding, though some customers may still be seeing timeouts and/or errors at this stage. We'll post another update shortly.

2:35 PM PDT Amazon S3 US has recovered and system-wide performance has returned to normal. We'll be posting here with more information about this event once we've completed our investigation.

Amazon Simple Queue Service
Service is operating normally.

and, here is an example of an incident report.

Amazon.com Data Center

I've been searching for information on amazon.com data centers, and there isn't a whole lot out there.  Found one Turner Construction project with reference to one facility.

Amazon.com Data Center
Seattle, WA
This phased fast-tracked demolition, renovation and structural upgrade of approximately 10,000 sq. ft. basement and approximately 18,000 sq. ft. of first floor in existing downtown five-story office building. Electrical support for data center included construction of Seattle City Light electrical transformer vault, N+1 redundant power and cooling, and structural reinforcement of the existing structure for seismic upgrades. In addition open workspace and offices were carved out of the area. Intensive just-in-time pre-construction sessions with the design team, city utilities and Amazon.com staff has made it possible to design, obtain permit and substantially complete in record time - five months from formal Notice to Proceed to an operational data center

There is the UC Berkeley/Stanford paper.

Despite significant efforts in the field of Autonomic Computing,
system operators will still play a critical role in administering
Internet services for many years to come. However,
very little is know about how system operators work,
what tools they use and how we can make them more efficient.
In this paper we study the practices of operators in a
large-scale Internet service Amazon.com and propose a new
set of tools for operators. The first tool lets the operators
explore the health of system components and dependencies
between them; the other monitors the actions of operators
and automatically suggests solutions to recurring problems.

I think I need to talk to some other data center friends who work in Seattle to find more about amazon.com.

Amazon and Google, Winning the Battle

And, the news continues about amazon and google, claiming these 2 will be the winners as the US Internet matures. Reuters reports on a new study.

SAN FRANCISCO (Reuters) - An Internet analyst for a major Wall Street firm argues in a new report that Google Inc and Amazon.com Inc will be long-term winners, while Yahoo and IAC InterActiveCorp fall by the wayside and eBay Inc becomes a merger target.

Sanford C. Bernstein analyst Jeffrey Lindsay argues in a 310-page report entitled "U.S. Internet: The End of the Beginning" to be published on Tuesday that Google and Amazon are best placed to withstand the current economic downturn.

"We expect two players to continue to perform strongly, Google and Amazon," Lindsay writes. "Both Google and Amazon.com are still racking up annual growth rates in the 30-40 percent range, with only a relatively modest slowdown in sight."

James Hamilton's Comparison Google Application Engine vs. Amazon Web Services

James Hamilton attended the Google IO conference and posts his comparison of Google Application Engine (GAE) vs. Amazon Web Services (AWS).

Last week at Google IO, pricing was announced for Google Application Engine. Actually it was blogged the night before at: http://googleappengine.blogspot.com/2008/05/announcing-open-signups-expected.html.

The prices are close to identical with Amazon AWS although GAE differs substantially from the AWS offerings.  The former offers a easy to use Python  execution environment whereas Amazon offers the infinitely flexible run-this-virtual-machine model. Clearly the Amazon model costs more to provide so, by that measure, AWS pricing is somewhat better:

Google Application Engine Pricing:

· $0.10 - $0.12 per CPU core-hour

· $0.15 - $0.18 per GB-month of storage

· $0.11 - $0.13 per GB outgoing bandwidth

· $0.09 - $0.11 per GB incoming bandwidth

· From: http://googleappengine.blogspot.com/2008/05/announcing-open-signups-expected.html

Compared with AWS Pricing:

· $0.10 - $0.80 per VM hour (depending upon resources allocated)

· $0.15 per GB-month of storage

· $0.100 - $0.170 per GB outgoing bandwidth

· $0.100 per GB incoming bandwidth

· From: http://www.amazon.com/S3-AWS-home-page-Money/b/ref=sc_fe_l_2?ie=UTF8&node=16427261&no=3435361&me=A36L942TSJ2AJA and http://www.amazon.com/EC2-AWS-Service-Pricing/b/ref=sc_fe_l_2?ie=UTF8&node=201590011&no=3435361&me=A36L942TSJ2AJA.

There are some important differences that make the pricing comparison somewhat biased in a couple of ways. Two important differences: 1) as mentioned above, Amazon gives an entire virtual machine so EC2 is much more flexible than GAE both in that it can run arbitrary applications in arbitrary languages and that it supports all execution models whereas GAE only supports HTTP request/response.  Another key difference is the storage subsystem.  In the numbers above, we’re comparing the Amazon blob store (S3) with the more structured storage model offered by GAE.  The more comparable AWS SimpleDB pricing is considerably higher than the GAE storage pricing. SimpleDB charges $1.50 GB/month in addition to machine usage and network transmission costs.  GAE is offering much more affordable semi-structured storage and the GAE storage model actually supports data types rather than having to force everything to character format.

And, James makes a good point about the Amazon as a retailer vs. google.  Amazon understands as any good retailer, Wal-mart, Costco - the pennies added up, and are important given your low margins.

GAE is still free to start with under 5M page views/month and up to ½ GB storage for free.  Obviously this helps developers get started without strings and that’s a good thing. But, more importantly, it avoids Google from having to go to the expense of billing very small values.  In a weird sort of way, I’m more impressed with AWS billing $0.04 on some accounts in that it shows there billing system is incredibly lean. Scaling down billing is hard, hard, hard.