Green Data Center Blog

Move to Cloud Computing, Saves 80% Operating Cost

Here is a blog entry about a guy who thinks about green a lot and discusses his move to Amazon Web Services’s EC2.  He discusses servers, green data centers, and how he saves 80% of his operating cost by moving to the cloud.

Cloud Computing: Truly Green Data Centers

Category: Conservation, Save Electricity, Technology – Tom Harrison – 10:54 am

There has been a lot written about how much power is consumed by the computers that drive the Internet. A lot has been written about “green” data centers. But I think there’s a far more significant trend when, combined with more efficient computers and data centers, will make a 10x or greater reduction in power demand possible: cloud computing.

Data centers, brown or green are huge buildings — they are truly incredible places, with thousands of computers owned by multiple companies. I have negotiated the contracts for “co-location” in a number of data centers: you pay for floor space, bandwidth, and power and get a facility that has great connectivity, power that never goes out, and a carefully cooled environment for the computers. This blog, and most other websites are located at such data centers. Little sites like this one share a “slice” of a server with a number of others. Large sites like the ones we have at the Internet companies I have worked at have our own computers and other equipment “co-located” in data-centers.

Here is his bottom line.

The Virtual Green Cloud is Here

So look at cloud computing from an overall energy standpoint — we use about 80% less server power, thus need that much less cooling and that many fewer computers. Add virtualization and we use even fewer physical resources so the “embedded energy” of the hardware is even less.

The only losers in this whole new paradigm: the data centers (at least the ones who haven’t gone to cloud computing on their own) and the computer manufacturers.

Solar panels on the roof, more efficient cooling, and other “green data center” features are great improvements. But by enabling us to stop wasting power and resources our business, which is just one of many thousands like it, will save money and be green in a cloud computing environment.

Amazon’s James Hamilton Presentation

James Hamilton continues his focus on energy efficiency at Amazon after leaving Microsoft. 

AFCOM Western Washington Chapter Meeting

Earlier today I presented Where Does the Power Go and What to do About it at the Western Washington Chapter of AFCOM. I basically presented the work I wrote up in the CIDR paper: The Case for Low-Cost, Low-Power Servers.

The slides are at: JamesHamilton_AFCOM2009.pdf (1.22 MB).

The general thesis of the talk is that improving data center efficiency by a factor of 4 to 5 is well within reach without substantial innovation or design risk.

Controlling Your Cloud Computing with an iPhone

AWS has a post about their new iPhone Console Application.

iPhone Console for EC2

This is a very brief post to call your attention to yet more innovation in the Amazon Web Services ecosystem: in this case an iPhone console application that monitors and controls your Amazon EC2 environment. David Kavanagh and company cooked this up over at directThought.

My mind immediately went to "Sitting in Maui, umbrella drink by the pool, time to add a few more instances to my Amazon EC2 server fleet by tapping on the iPhone. Ahhh..." Then reality struck -- it's snowing outside the hotel I'm in.

The underlying client toolkit (cTypica) is licensed under the Apache 2.0 License.

You can preview the application here. Now is your chance to provide input on what will be a very useful tool for AWS users who have an iPhone.

Mike

This post will give me an excuse to ping Mike (at amazon) as I haven’t chatted with him for a while.

Microsoft Architect Joins Amazon Web Services

Mary Jo Foley reports on James Hamilton leaving Microsoft to join Amazon Web Services.

Microsoft datacenter architect defects to Amazon

Posted by Mary Jo Foley @ 1:42 pm

James Hamilton, an architect on Microsoft’s Data Center Futures team, has decided to leave Microsoft for Amazon.com, as TechFlash first reported.

Hamilton is known at Microsoft for his work to popularize the container-model for datacenters. He was a member of Chief Software Architect Ray Ozzie’s Live Platform Services development team.  Hamilton’s interests included “multi-tenant hosted systems, the management of very large scale systems, massively parallel data management systems, database security, and unstructured data management.” Before joining the Live Core team, Hamilton was the general manager of the Microsoft Exchange Hosted Services team, and before that, SQL Server Architect and leader of the Security and Incubation Team.

Hamilton will be a Vice President on the Amazon Web Services team. (I asked Amazon for more specifics about Hamilton’s new role but received no word back.)

Amazon has been doing everything it can to prep for Microsoft’s entry into the cloud-hosted development space. Microsoft layed out its plans there in late October, when its executives detailed officially the company’s Azure cloud platform.

I am curious if Microsoft bloggers will comment on James Hamilton leaving.

Given James’s seniority at Microsoft, I would expect Amazon paid top dollar to get him to leave. I’ve met James and had some discussions with him, but his roles have been more in research than data center design.  It will be interesting watching Amazon’s activities as they add more data center resources.  It seems like every week I get someone asking how to get into the amazon data center account, and I tell them I think Amazon is one of the toughest account to get into, and haven’t found a good access yet.

The data center world is small, and it will be easy to pick out James with this long hair, and tall stature.  You can look for him at data center conferences.

Here is James own post.

James Hamilton Joins Amazon.com

I’ve resigned from Microsoft and will join the Amazon Web Services team at the start of next year. As an AWS user, I’ve written thousands of lines of app code against S3, and now I’ll have an opportunity to help improve and expand the AWS suite.

In this case, I’m probably guilty of what many complain about in bloggers: posting rehashed news reported broadly elsewhere without adding anything new:

· http://techhermit.wordpress.com/2008/12/11/microsofts-container-progenitor-to-leave/

· http://www.techflash.com/microsoft/Key_data_center_architect_leaves_Microsoft_for_Amazon36057114.html

· http://www.datacenterknowledge.com/archives/2008/12/12/james-hamilton-leaving-microsoft/

· http://blogs.zdnet.com/microsoft/?p=1765

· http://www.theregister.co.uk/2008/12/12/amazon_goes_containers/

· http://blog.seattlepi.nwsource.com/amazon/archives/156853.asp

· http://www.readwriteweb.com/jobwire/2008/12/james-hamilton-leaves-micfroso.php

· http://blogs.msdn.com/stevecla01/archive/2008/12/12/dick-hardt-in-james-hamilton-out.aspx

Job changes generally bring some stress, and that’s probably why I’ve only moved between companies three times in 28 years. I worked 6 years as an auto-mechanic, 10 years at IBM, and 12 years at Microsoft. Looking back over my 12 years at Microsoft, I couldn’t have asked for more excitement, more learning, more challenges, or more trust.

I’ve had a super interesting time at Microsoft and leaving is tough, but I also remember feeling the same way when I left IBM after 10 years to join Microsoft. Change is good; change challenges; change forces humility; change teaches. I’m looking forward to it even though all new jobs are hard. Onward!

AWS Cloudfront, Amazon Creates an Alternative to Akamai & Limelight

Amazon Web Services just announced Cloudfront CDN.

Amazon CloudFront is a web service for content delivery. It integrates with other Amazon Web Services to give developers and businesses an easy way to distribute content to end users with low latency, high data transfer speeds, and no commitments.
Amazon CloudFront delivers your content using a global network of edge locations. Requests for your objects are automatically routed to the nearest edge location, so content is delivered with the best possible performance. Amazon CloudFront works seamlessly with Amazon Simple Storage Service (Amazon S3) which durably stores the original, definitive versions of your files. Like other Amazon Web Services, there are no contracts or monthly commitments for using Amazon CloudFront – you pay only for as much or as little content as you actually deliver through the service.

Cloudfront has a calculator too.

Here is the AWS blog entry on Cloudfront.

Like all of our other services, CloudFront was designed with ease of use in mind from the very beginning.  There are no minimum usage commitments, no monthly fees, and no need to even talk to us. Here's what you do:

1. Sign up for CloudFront.
2. Put your most frequently accessed static content into an Amazon S3 bucket and mark it as publicly readable.
3. Create a new CloudFront Distribution using a single REST-style POST call. Capture the domain name returned by the call.
4. Generate fresh URLs for your content using the domain name from step 3 and hand them out. By using our CNAME support you can even make the content appear as if it is coming from your own domain. You can associate up to 10 CNAMEs with each distribution.

Amazon’s New Oregon Data Center

DataCenterKnowledge reports on Amazon.com’s new Oregon Data Center.

Amazon Building Large Data Center in Oregon

November 7th, 2008 : Rich Miller

Amazon.com appears to be the tenant in a large data center rising on the banks of the Columbia River in Oregon, joining Google in harnessing the region’s cheap energy resources to power huge cloud computing data centers.

The $100 million data center is being built in Boardman, Oregon in the Port of Morrow, a 9,000 acre industrial park. Plans call for three large buildings on the site, according to The Oregonian, which reports that representatives of Amazon have attended local meetings to discuss permits for the site. The first building is underway and will be 116,000 square feet.

I was starting to wonder if Amazon had some secret to keep us from knowing what is was building.

Amazon is historically tight-lipped about the location of its data centers. But the rapid growth of its Amazon Web Services (AWS) cloud computing platform long ago exceeded the excess capacity in the data centers supporting the company’s retail operations. As AWS grows, Amazon will need to continue adding dedicated infrastructure to add capacity. Amazon recently said its S3 cloud storage service was now storing 29 billion objects.

But, you can’t build big data centers without the insiders finding out or the local press.  Especially when you are looking for cheap and/or green power for your data center.

The Columbia River basin has large resources of hydro electricity generated from dams along the river. This cheap, clean power was a factor in Google’s decision to build a huge data center in The Dalles, Oregon and has fueled the tranformation of  Quincy, Washington from a small farming town into a data center hub with new facilities from Microsoft and Yahoo.

A 10-megawatt power substation is being built adjacent to the new data center in Boardman, the Oregonian reports. The paper said a $320,000 Oregon Department of Transportation grant was awarded to the port to connect the data center to local utilities. The grant will cover about half the total cost of the additional power infrastructure.

We’ll see if Amazon changes and follows the Microsoft and Google model to share some of their best practices.

NYTimes AWS Cloud Computing Mistake Cost $240

Nicholas Carr discusses what is possible with Cloud Computing.

The new economics of computing

November 05, 2008

Are we missing the point about cloud computing?

That question has been rattling around in my mind for the last few days, as the chatter about the role of the cloud in business IT has intensified. The discussion to date has largely had a retrospective cast, focusing on the costs and benefits of shifting existing IT functions and operations from in-house data centers into the cloud. How can the cloud absorb what we're already doing? is the question that's being asked, and answering it means grappling with such fraught issues as security, reliability, interoperability, and so forth. To be sure, this is an important discussion, but I fear it obscures a bigger and ultimately more interesting question: What does the cloud allow us to do that we couldn't do before?

The history of computing has been a history of falling prices (and consequently expanding uses). But the arrival of cloud computing - which transforms computer processing, data storage, and software applications into utilities served up by central plants - marks a fundamental change in the economics of computing. It pushes down the price and expands the availability of computing in a way that effectively removes, or at least radically diminishes, capacity constraints on users. A PC suddenly becomes a terminal through which you can access and manipulate a mammoth computer that literally expands to meet your needs. What used to be hard or even impossible suddenly becomes easy.

His example is the NYTimes.

My favorite example, which is about a year old now, is both simple and revealing. In late 2007, the New York Times faced a challenge. It wanted to make available over the web its entire archive of articles, 11 million in all, dating back to 1851. It had already scanned all the articles, producing a huge, four-terabyte pile of images in TIFF format. But because TIFFs are poorly suited to online distribution, and because a single article often comprised many TIFFs, the Times needed to translate that four-terabyte pile of TIFFs into more web-friendly PDF files. That's not a particularly complicated computing chore, but it's a large computing chore, requiring a whole lot of computer processing time.

Fortunately, a software programmer at the Times, Derek Gottfrid, had been playing around with Amazon Web Services for a number of months, and he realized that Amazon's new computing utility, Elastic Compute Cloud (EC2), might offer a solution. Working alone, he uploaded the four terabytes of TIFF data into Amazon's Simple Storage System (S3) utility, and he hacked together some code for EC2 that would, as he later described in a blog post, "pull all the parts that make up an article out of S3, generate a PDF from them and store the PDF back in S3." He then rented 100 virtual computers through EC2 and ran the data through them. In less than 24 hours, he had his 11,000 PDFs, all stored neatly in S3 and ready to be served up to visitors to the Times site.

The total cost for the computing job? Gottfrid told me that the entire EC2 bill came to $240. (That's 10 cents per computer-hour times 100 computers times 24 hours; there were no bandwidth charges since all the data transfers took place within Amazon's system - from S3 to EC2 and back.)

An interesting part of the story on the original blog which was confirmed by an Amazon Web Services evangelist is the NYTimes had to run the cloud computing job twice, as the first had errors.

I then began some rough calculations and determined that if I used only four machines, it could take some time to generate all 11 million article PDFs. But thanks to the swell people at Amazon, I got access to a few more machines and churned through all 11 million articles in just under 24 hours using 100 EC2 instances, and generated another 1.5TB of data to store in S3. (In fact, it work so well that we ran it twice, since after we were done we noticed an error in the PDFs.)

So, it cost the NYTimes an extra $240 for their mistake as they had to run the job a second time.

Amazon Promotes Frustration Free (Greener) Packaging

How big is Green?  Amazon chooses to make Frustration Free (Greener) Packaging highlighted on its home page today.

One of the products featured is this pirate ship.

In addition Amazon has an environment site that points to a study online shopping is is more environmentally friendly than traditional shopping.

Economic and Environmental Implications of Online
Retailing in the United States
H. Scott Matthews1 and Chris T. Hendrickson2

Abstract

The advent of the Internet and e-commerce has brought a new way of marketing and selling many products, including books. The system-wide impacts of this shift in retail methods on cost and the environment are still unclear. While reductions in inventories and returns provide significant environmental savings, some of the major concerns of the new e-commerce business models are the energy and packaging materials used by the logistics networks for product fulfillment and delivery. In this paper, we analyze the different logistics networks and assess the environmental and cost impacts of different delivery systems. With a return (remainder) rate of 35% for best-selling books, ecommerce logistics are less costly and create lower environmental impacts, especially if private auto travel for shopping is included. Without book returns, costs and environmental effects are comparable for the two delivery methods.

Load Testing From The Cloud, A Killer App?

There are debates over the usefulness of cloud' computing for enterprise.  Amazon Web Services blog has a post on how one company is using AWS to create load testing to test other web sites.

SOASTA - Load Testing From the Cloud

I met Tom Lounibos, CEO of SOASTA, at the Palo Alto stop of the AWS Start-Up Tour. Tom gave the audience a good introduction to theirCloudTest product, an on demand load testing solution which resides on and runs from Amazon EC2.

Tom wrote to me last week to tell me that they are now able to simulate over 500,000 users hitting a single web application. Testing at this level gives system architects the power to verify the scalability of sites, servers, applications, and networks in advance of a genuine surge in traffic.

Here are a few of their most recent success stories:

• Hallmark tested their e-card sites in preparation for the holiday season, and are ramping up testing to over 200,000 simultaneous users using CloudTest.
• Marvel Entertainment is doing extensive cloud testing in order to get ready for the release of the sequel to IronMan.
• A division of Procter & Gamble is using cloud testing to get ready for new releases of their web site.

Soasta also has a monitoring capability.

See Across Your Entire Web Application Infrastructure.

Resource Monitoring

Monitoring is the ability to monitor a resource (hardware, network, load balancer, firewall, Web server, database, application server, content management system, etc.) and capture usage information about that resource. Resource monitoring is a key component of professional Web testing. While it is crucial that a Web application functions correctly, resource usage and end-to-end response are extremely important. When there are problems, you need information about resource usage across the entire infrastructure of your Web application.

Resource information can be captured from any available resource in the Web application infrastructure—not just the Web server hosting the Web application. SOASTA CloudTest can monitor all three tiers of your Web application—the Web server, the application server, and the database server. It can also capture valuable information about other components in your network architecture—load balancers, for example.

Soasta says load testing is the killer app for cloud computing.  Didn’t think about it until now, but makes a lot of sense.

"Load and performance testing is the ‘killer application’ for cloud computing," said Tom Lounibos, CEO of SOASTA. "Companies can very easily create a real-world test environment without having to invest in it. Developers have virtually unlimited and affordable access to thousands of servers, memory, storage, etc. and can, essentially on demand, simulate load and performance tests for tens of thousands of users without having to purchase the hardware."

Why SOASTA CloudTest Lab is uniquely different:

• It’s Real World: Load and performance testing in cloud computing environments is the closest thing to running an application in production minus the worry of negatively impacting your customers. SOASTA CloudTest Lab provides you with a controlled environment to thoroughly simulate and stage a real world scenario before it goes into production.
• It’s On Demand: No more costly investment in new hardware or worries about staffing up for support and management. Ready when you are, SOASTA CloudTest Lab serves as a virtual test lab at your service 24x7x365 and has you testing in a matter of minutes.
• It’s Scalable: An overloaded Web site is a major problem. Being prepared and understanding the limits of your application is crucial to maintaining availability and a quality customer experience. SOASTA CloudTest Lab allows you instant access to up to 1,000 available test servers when needed, and the ability to shut them down when unused to reduce costs. In short, you pay only for what you need. Find out, without the cost and time of setting up and tearing down hundreds if not thousands of servers, whether or not your Web application can scale exponentially at a moment’s notice.
• It’s Affordable: Load and performance testing is no longer cost prohibitive. A capability that would typically cost hundreds of thousands of dollars is now available to companies of all sizes in a matter of minutes for only a few hundred dollars.

Data Center Best Practices – Microsoft, Google, or Uptime Institute, Don’t Forget Sun, HP, Dell, and IBM

TechHermit has a post base on Uptime’s latest conference.

Uptime Institute to IT, Microsoft and Google are your enemy

October 31, 2008

At the Uptime conference this last week in Dallas, Ken Brill shocked many of the attendees by throwing out some amazing statements that can only be classified as fear mongering. Perhaps the rhetoric was a first salvo in an inflammatory exchange between industry giants.

The comments in question came up in a meeting of Uptime members assembled for the event.  His comments to the audience centered around how Microsoft and Google were a direct threat to everyone in attendance.  Perhaps this was a result of the Microsoft Azure release last week and the existence of Google Apps. In addition many of the comments were directed towards the PUE metric.  Specifically that Microsoft and Google were highlighting low PUE results of 1.2x which were unrealistic and flawed.

Now I admit to being biased as I have many more conversations with Microsoft and Google Engineers than Uptime Institute Engineers.  Oh wait,  I’ve talked to Uptime Sales people though.

Microsoft has its Power of Software blog, the Microsoft data center team presenting at conferences and even Microsoft Research giving away data center knowledge for no charge.

Google has its data center site.

Sun has both a Blog and data center site.

HP, Dell, and IBM all have data center offerings as well.

The one guy who doesn’t say anything about its data centers is Amazon, but they have a huge presence with Amazon Web Services.  So, don’t expect Ken Brill/Uptime throwing any criticism at Amazon.

Unfortunately, for Ken Brill and Uptime Institute they are making enemies faster than friends.

How long can Uptime Institute survive?

I know most of your out there don’t care, but a few of you do, and this is an interesting study of sharing information vs. charging for information to improve the efficiency of your data center.

Right now market perception is Microsoft and Google are the leaders in data center innovation and all of these companies are battling to be at the top which is good for the whole industry.