Green Data Center Blog

Microsoft Internal Field Presentation on Green/Sustainability Architecture

Michael Manos, GM of Microsoft's Data Center Construction and Operation, and Lewis Curtis, Infrastructure Architect will be presenting for the 3rd time at Microsoft's TechReady event, an internal presentation for Technical Field staff.

TechReady is a bi-annual, 5-day internal technical conference for Microsoft employees. Microsoft employees from all of the international subsidiaries, as well as American employees are presented to by the product groups about all the upcoming releases for the year, and receive training on all of the currently released Microsoft products^[1]. The senior executives will also often present at TechReady, outlining their visions for the future, and taking questions from employees.

TechReady is made up of many separate 'sessions' - from general briefings suitable for all audiences to in-depth 'breakout sessions' for a deep look at a specific product, or part thereof.

For those Microsoft people coming to town for the event, the session is ARC204, Thurs, July 31, 4:15-5:30, Environmental Sustainability Architecture Update.

· Sustainable Intelligence:  example is SCRY

· Sustainable Online Services: example is Live / GFS 

· Sustainable Optimization: example is products like Windows 2008 power mgmt/.Net optimization as well as datacenter optimization best practices from GFS, etc…

· Sustainable Consolidation: example is Windows Server 2008 Virtualization, IIS7, hardware design consolidation, SQL Server 2008, etc…

Maybe I'll be able to catch Lewis and Mike after the event for some comments.

A Look Inside Microsoft’s Green Data Center Efforts, Educating Enterprise Developers and Sys Admins

Microsoft hosted a panel discussion at TechEd 2008 in Orlando.  The panel was moderated by George Cerbone with Mike Manos, Lewis Curtis, Beth Humphreys from Microsoft. Additional panel members are Kathy Malone and David Platt.

The video is here. Don’t hesitate to just hit play, and listen to the audio while you are surfing the web.  It has some good perspectives.

Lewis Curtis presented first his past experiences over the last year working on Green IT at Microsoft.

Mike Manos presented why going Green is a business responsibility to save energy and has monetary savings.

Beth Humphreys shares her experiences in customer engagements discussing Green IT.

Kathy Malone talks about her efforts to build compliance systems and a green IT developer community.

David Platt, a recognized .NET developer expert, explains why developers are not creating green applications. His view is green should be an OS function, and developers don’t have enough time for this.

Mike Manos brings up a good rebuttal that once Microsoft changed to an energy based chargeback system, the business units started to understand the impact of their SW on their energy bill, and have chosen energy efficiency vs. a focus on only performance.

David Platt continues to make the point developer costs and power is negligible, repeating his point that the energy efficiency is the responsibility of the OS, and none of the effort should be taken on by the developer.

The good thing is the rest of the panel think David Platt is wrong.

Kathy Malone makes excellent points on compliance systems.

This video gives you an idea of some of the people in Microsoft who are driving the Green Data Center efforts to the software developer community. 

I am glad I caught Lewis’s post.

An Application Architect who Cares about Energy Efficiency

Pat Helland put up his post about a week at Foo Camp. 

What is Foo Camp? Pat gives an entertaining summary.

A Weekend at Foo Camp

Well...  I was lucky enough to get invited to Foo Camp (which was last weekend) and I figure "What the heck!  Let's do it!".  

Foo Camp is held at the O'Reilly headquarters in Sebastopol, CA which is 1 to 2 hours (or more depending on traffic) north of San Francisco.  It is an invitation only event whose name stands for "Friends Of O'Reilly" and involves about 300 diverse and interesting individuals from different walks of the computer industry (and related industries).  Tim O'Reilly host it at their headquarters and supplies very nice buffet food, showers, rest rooms, and meeting rooms.  I was informed that the best way to enjoy the event is to camp there which means either pitching a tent on their lawn or finding an available space in a meeting room or hallway to throw a sleeping bag.  It was an option to get a hotel room in town.  Now... I haven't camped in about 20 years so this required some thinking... 

I concluded I had two options:

1. Stay in a hotel room and ensure I remained sober enough to drive at the end of the evening, OR
2. Buy a tent at REI in Santa Rosa and then hit the high-end liquor store in Santa Rosa to buy enough whiskey to lubricate a serious subset of the 300 attendees.

Naturally, I chose the second option and bought six bottles (some of my favorites and others I hadn't tried). 

It turns out James Hamilton and Pat Helland are friends and are putting their heads together on Energy Efficiency software. James is left, Pat is center, and Jesse Robbins is right.

 

But, back to the Energy Efficiency software.  Here are Pat's comments.

Sometimes, it was hard to select between the various cool sessions.   I remember the following ones:

• Data center power -- James Hamilton (my friend from Microsoft) and Jeff Hammerbacher who leads the Facebook data team (but will be leaving soon).  Both James and Jeff were filled with information about running large and dynamic data centers.   The power issues for data centers have been on my mind the last few years and I find that James is a wonderful font of knowledge.   I most definitely love that he is at Microsoft and my friend... I plan to come pepper him with additional questions in the months to com.   Jeff, also, has tons of knowledge from supporting the data needs of Facebook as it has undergone its explosive growth.  This was a fun and invigorating discussion in which I met an attendee, Roger Magoulas who is a research director at O'Reilly.   I have a feeling that there will be opportunities for me to work with Roger, too.
• Parallel Programming -- Kerry Hammil of Microsoft Research.   We had a fun discussion of the difficulties of getting applications (and, indeed, their libraries and OSes) to be parallel.  There were about 25 great and interesting people participating in this group and, not surprisingly, I participated, too.     This was such a lively discussion for me that it ended up in the hallway and we skipped the next session.

Here is an application architect who cares about data center power and parallel (multicore) programming. I hope we see more people like Pat show up.

Jesse Robbins in the above picture has an interesting background.

• Technical Program Manager at Amazon.com
• Manager - IT Operations at Amazon.com
• Task Force Leader at World Shelters Task Force 1

• Systems Engineer at Amazon.com
• Firefighter/EMT (Intern) at Palo Alto Fire Department

Microsoft's James Hamilton Presents "Where Does Power go in Data Centers and How to get it Back?"

James Hamilton has a blog post about his attending O'Reilly's Foo Camp and his presentation on "Where does Power go in Data Centers and How to get it Back?"

The title for my session was Where Does the Power go in Data Centers and How to get it Back?  I didn’t show slides but much of what we covered is posted at: http://mvdirona.com/jrh/TalksAndPapers/JamesRH_DCPowerSavingsFooCamp08.ppt.  In the session, we talked through how contemporary large data centers work first looking at power distribution. We tracked the power from the feed to the substation at 115,000 volts through numerous conversions before arriving at the CPU at 1.2 volts. We then talked about power saving server design techniques.  And then the mechanical systems used to get the heat back out.  In each section we discussed what could be done to improve the design and how much could be saved.

Our conclusion from the session was that power savings of nearly 4x where both possible and affordable using only current technology.  For those participated in the session, thanks for your contribution and  for your help. It was fun.

James comes to the conclusion there is a power savings of 4x. If  you are curious as to how he comes to this look at his slides.  One of his ideas that flies in the face of high density computing is Thin Slice Computing.

Microsoft Research Paper on SSD Performance and Design Tradeoffs

StorageMojo has a post about Microsoft Research’s paper on Design TradeOffs of SSD.

Design Tradeoffs for SSD Performance

July 15th, 2008 by Robin Harris in Architecture, Future Tech, SSD/Flash Disk

A new Usenix paper looks at NAND flash SSD performance. From a team at Microsoft Research and the University of Wisconsin, including Ted Wobber who worked on last year’s A Design for High-Performance Flash Disks [see Flash chance for the StorageMojo take on that excellent paper - a post Ted was kind enough to review and comment on].

Design Tradeoffs for SSD Performance (by Nitin Agrawal, Vijayan Prabhakaran, Ted Wobber, John D. Davis, Mark Manasse and Rina Panigrahy) makes a deep dive in flash translation layer (FTL) issues. As the authors note, flash vendors keep their FTL designs secret, so the team developed a NAND flash simulator to look at how design choices affected performance.

What they found
They ran several workloads on their trace-based simulator, including TPC-C, Exchange and some file system benchmarks. They found several critical issues in SSD design.

• Data placement Needed for wear leveling and load balancing.
• Parallelism Single flash chips aren’t very fast so they need to work together.
• Write ordering Small random writes are a killer.
• Workload management You can optimize for sequential or random workloads, but managing both well is hard.

and as StorageMojo closes you need to read the Microsoft Research paper to get a full understanding.

The StorageMojo take
This paper is too rich in detail to summarize well. If understanding SSD controller design is important there is no substitute for a careful read.

The net is that engineers have many options in configuring and managing flash devices inside a solid state disk. The interaction of these design choices with applications is likely to remain a fruitful area of study for years to come.

Expect to see many performance oddities as new solid state disk designs are released. This is a different world than disk drives. There is much innovation and much to learn.

A macro longer-term trade-off is the extent to which SSD vendors should attempt to alter operating system behavior to better match SSDs. In the short term designers must conform to today’s disk I/O oriented operating systems. In the long term however, there must be major opportunities to tweak operating systems to enhance solid-state disk performance.

For this reason SSDs is may find their best short term market to be inside storage arrays where array vendors have complete control over the interface to the array software. This will be no small advantage as array vendors struggle to remain relevant in a world where high performance solid state disks have the potential to replace midsize arrays.

James Hamilton also has a post referring to Spansion’s Flash Memory announcement.

Microsoft Grabs Leadership Positioning in NetworkWorld Article

After 2 days at the National Data Center Energy Efficiency Workshop and Energy Star, I was looking for a way to summarize some of the issues covered for the 150 attendees.  NetworkWorld reported on the workshop, and did the work for me.  So, let me highlight some parts.

Good incentives boost data-center energy efficiency

By Nancy Gohring , IDG News Service , 07/09/2008

A Microsoft executive shared techniques the company has used, including new kinds of employee incentive programs and internally created automation tools, to reduce the energy consumption of its growing data centers.

The methods he described could help other companies that use or operate data centers reduce costs, said experts who also spoke at the data-center efficiency strategy conference put on by the U.S. Department of Energy and the Environmental Protection Agency in Redmond, Washington, on Tuesday.

While there are plenty of technology solutions for improving data-center energy efficiency, not many companies are using them, said Christian Belady, principal power and cooling architect at Microsoft. "It boils down to a behavioral problem, not necessarily a technology problem," he said.

Microsoft decided to change the incentives for workers as a way to encourage them to use the most energy-efficient techniques. Traditionally, the various business groups within the company were charged for using the company's data centers based on the amount of floor space required to stack the servers that their services used. That spurred a drive within the business units to minimize the space they used, often through the use of extremely dense servers. Those servers, however, sucked power and required more cooling, Belady said.

Now, Microsoft charges business units based on the amount of energy consumed by the servers that host their services. "We moved from cost as a function of space to cost being a function of power," he said.

That shift made individual business units conscious of the number of DIMMs (dual in-line memory modules) they had at their disposal, for example. "Now those DIMMs are costing you power, and you're getting a year-over-year chargeback for those DIMMs," he said. Such charges make the business units less likely to require more memory then their services actually need, he said.

Other industry speakers are quoted.

Incentives are also changing at utility companies in ways that can benefit enterprises. "As a facility manager my incentive isn't to sell you more electricity, but to give you the tools to be more efficient," said Francois Rongere, segment supervisor with PG&E's high technology energy-efficiency team. "My bonus is based on how much savings you have done in my territory."

Those energy savings often translate into real money from the utility. Ray Pfeifer, who works with the Silicon Valley Leadership Group, was recently involved in a series of experiments with companies to try to quantify how certain changes to their data centers affected energy usage. He said that many of the implementations were 100 percent funded by utilities, which often offer incentives to companies for investments that can cut their energy usage.

While that may be good news to the enterprise, the utility incentives only show how behind the curve businesses are in general, said Brill. "It's appalling to me that we have to have utilities offering incentives to do what's good business sense," he said.

Thanks to Nancy Gohring for writing a good article.

Working at Google vs. Microsoft

Here is an article about developers who have chosen Microsoft over Google, and even some developers who worked at Microsoft went to Google, and went back to Microsoft. I looked at this for maybe some hints to why Google needs 200 employees for data centers vs. Microsoft’s 50. I would bet there are ex-Google data center employees working at Microsoft, but it is almost impossible to find them, let alone get them to share their insights.

Working at Google vs. Working at Microsoft

I have theory that Google's big problem is that the company hasn't realized that it isn't a startup anymore

By: Dare Obasanjo

Jul. 2, 2008 09:15 AM

Dare Obasanjo's Blog

Recently I've been bumping into more and more people who've either left Google to come to Microsoft or got offers from both companies and picked Microsoft over Google. I believe this is part of a larger trend especially since I've seen lots of people who left the company for "greener pastures" return in the past year (at least 8 people I know personally have rejoined). However in this blog post I'll stick to talking about people who've chosen Microsoft over Google.

Interesting parts.

Google software business is divided between producing the "eye candy" - web properties that are designed to amuse and attract people - and the infrastructure required to support them. Some of the web properties are useful (some extremely useful - search), but most of them primarily help people waste time online (blogger, youtube, orkut, etc)

This orientation towards cool, but not necessarilly useful or essential software really affects the way the software engineering is done. Everything is pretty much run by the engineering - PMs and testers are conspicuously absent from the process. While they do exist in theory, there are too few of them to matter.

On one hand, there are beneficial effects - it is easy to ship software quickly…On the other hand, I was using Google software - a lot of it - in the last year, and slick as it is, there's just too much of it that is regularly broken. It seems like every week 10% of all the features are broken in one or the other browser. And it's a different 10% every week - the old bugs are getting fixed, the new ones introduced. This across Blogger, Gmail, Google Docs, Maps, and more
…
The culture part is very important here - you can spend more time fixing bugs, you can introduce processes to improve things, but it is very, very hard to change the culture. And the culture at Google values "coolness" tremendously, and the quality of service not as much. At least in the places where I worked.

After reading this, maybe Google really does need 200 people per data center to fix all the problems.

And, the article closes with

The fact that Google is having problems retaining employees isn't news, Fortune wrote an article about it just a few months ago. The technology press makes it seem like people are ditching Google for hot startups like FriendFeed and Facebook. However the truth is more nuanced than that. Now that Google is just another big software company, lots of people are comparing it to other big software companies like Microsoft and finding it lacking.

Information Week tours Microsoft’s San Antonio Data Center under construction

Information Week was given a tour of Microsoft’s San Antonio data center under construction.

Seeding The Cloud

Posted by J. Nicholas Hoover, Jun 20, 2008 02:39 PM

Recently, I got the chance to visit one of Microsoft (NSDQ: MSFT)'s mega data centers for a tour while it was under construction. I'll just say this: creating the infrastructure needed for cloud computing isn't child's play.

Microsoft rolls out data centers as if they were aircraft carriers in the United States Navy. There's the Quincy-class, there's the Dublin-class, and there's the Chicago-class. I myself was at the Quincy-class San Antonio location, a monstrous ship of the online if there ever was one at 475,000 square feet.

And that's only one of the numerous (Microsoft won't give a number) $500-million-or-so data centers the company's got under construction or up and running around the world. With all that going on, Microsoft's got to have a plan in place to figure out when and where to build.

The company uses what Microsoft data center services general manager Mike Manos calls a "continuum strategy." It starts with Microsoft mapping out the world to identify the company's important data center markets. From the maps, Microsoft (duh) determines where new data centers are needed.

There isn’t anything new in this article that hasn’t been reported on previously. But, Microsoft gained more eyes and the writer speculates on the software + services strategy.

P.S. For all of those who read this and say, OK, but what is Microsoft going to use this ridiculous amount of server energy for, check here for more, starting on Saturday. Remember that top Microsoft execs have said that almost every Microsoft product will have a services component or a version delivered as a service going forward, and that Microsoft execs also have hinted at the possibility of utility computing services a la Amazon Web Services. That's a lot of Windows Servers, SQL Servers, and so on. You can be sure that come the end of October, you'll know a lot more.

Microsoft Reaches a Sustainability Milestone, Life after Bill Gates

When I went to Microsoft in 1992, I told many stories of what the difference is between working at Apple and Microsoft. I’ve had the interesting experience of twice being escorted out of my employer as I went to a competing company.  Yes, when I left HP to go to Apple, within hours of telling them I was going to Apple, security was supervising the packing of my desk. I didn’t realize I was working on something so competitive. When I made my announcement to go from Apple to Microsoft, I was ready to be escorted out the door.

Working at great companies like HP, Apple, and Microsoft, there is an interesting transition they have.  How do they survive after their founders move on? HP made the transition beyond David Packard and Bill Hewlett. Apple’s John Sculley kicked Steve Jobs out, but Steve is back. And, whether Apple will survive beyond Steve Jobs is highly debated. A Barron’s article puts things in perspective.

But what really worries me is the factor that pressured the stock last week: concern about the health of Steve Jobs. More than any other company, Apple is viewed as a creation of its CEO; it's a cult of personality. And, as I noted on my blog last week, many attendees at the developer conference's keynote-address session were taken aback by Jobs' gaunt appearance.

Remember, Jobs just a few years ago received surgical treatment for a rare form of pancreatic cancer. And Apple didn't disclose the situation until months after he had become ill. Last week, in response to press inquiries, it said that its leader recently had a "common bug," which was treated with antibiotics. It wouldn't comment on speculation that he's had a relapse.

While I understand the company's reluctance to respond to rumors -- once you respond to one, it is hard to stop responding -- there's nothing more material to Apple shareholders than the status of its legendary chief's health. I have no idea if the aforementioned bug was the only reason for his ultra-thin appearance, and I hope there's nothing to the rumors.

Nonetheless, even if based on untrue information, last week's selloff was a dry run for the selling that would be triggered if Jobs actually did suffer a relapse. So while I think the iPhone 3G is going to be a big hit, and should boost corporate profits, just realize that there's at least one factor that not even Steve can control.

Bill Gates has reached the point where he is saying good bye to his job at Microsoft.  And, it is news coverage people are competing for to cover. It’s probably bigger news than any Window launch except Win95. Newsweek’s Steven Levy has an article.

In some respects, this week won't be terribly different for Bill Gates than the previous 1,712 weeks he has spent working full-time at Microsoft, the company he co-founded as a teenager. The 52-year-old icon has some one-on-one meetings scheduled with a few of his top technical executives. He has some customer meetings. And, as often happens, he'll go to the television studio on Microsoft's Redmond, Wash., campus to tape a few messages for events he won't be able to attend. In addition, he says, "I hope to write a few memos."

But normalcy will be an illusion. Everybody knows that when the week ends, Bill Gates will walk out of his office for the last time as someone on the clock for Microsoft. (On that final day, the routine will be shredded, and the staff has planned some internal commemorative events.) He'll take a break this summer (including a sojourn to the Beijing Olympics), and beginning in September the new focus of his work life will be the Bill & Melinda Gates Foundation, the organization he began with his wife in 2000. With a current $37.3 billion endowment, it's the world's richest philanthropic institution.

Paul Allen gives words of advice in the Newsweek article.

Treading on uncertain ground like that underlines the difficulties Gates may face in leaving the job he has loved so much. "It may be more of a change than he thinks," says Paul Allen, recalling his own departure from Microsoft in 1983. "You don't always realize how dramatic that transition is going to be when people aren't depending on your decisions day by day."

Microsoft’s transition to life without Bill is a new test in sustainability for the company. Can Microsoft make the transition successfully to life beyond its founder? 

Apple’s transition to life after Steve Jobs has massive risks. When Apple tried to survive after Steve Jobs, Apple needed Steve to come back and rescue the company.  Would Bill come back to Microsoft if it meant saving the company? Without a doubt, but that’s not a sustainable strategy.

Microsoft Launches New Blog for Energy Efficiency Best Practices

Microsoft launched a new blog called “The Power of Software.” This looks like a site you should add to your rss feeds. I am.

Welcome to The Power of Software blog, a new undertaking by the patterns & practices team. As you may know, our traditional focus has been on building guidance that helps software architects and developers successfully design and build applications.

This blog is a slight departure from that. We’re exploring ideas relating to Green IT and the ways we, as a company, can use energy more efficiently. Some currently planned subjects include ways to save energy through the use of software and ways to optimize datacenters. All posts will be written or reviewed by subject matter experts, just like other patterns & practices projects.

We hope this starts a dialog with the community—please let us know the topics that interest you.

RoAnn Corbisier
Editor

The first entry is by Christian Belady and Mike Manos on Microsoft’s experience using PUE in their data centers.

Microsoft’s PUE Experience—Years of Experience, Reams of Data

This short series of articles describes how Microsoft uses Power Usage Effectiveness (PUE), an industry standard metric for the efficiency of a datacenter. Being able to measure and monitor the effective power consumption of a datacenter in terms of the computing power it contains provides a way to ensure that you make best use of resources while minimizing your environmental footprint. This first article introduces PUE and looks at the issues that it can help you to resolve.

Part 1—"What Color is your Datacenter?"

Imagine if a child were to draw a picture of your datacenter. Does it look green, or is it a glowing orange or even as black as night? Look at the individual pieces of equipment in your datacenter—are any of them green?

If you want the picture of your datacenter to look greener (more energy efficient), you could try upgrading items to more energy-efficient equivalents, as if they were pieces of a puzzle that can simply be replaced. This upgrade method is what many companies are using as a way to convince themselves that they are reducing energy costs. The problem is that, unless you look at the big picture and understand how the pieces fit together, you could end up being disappointed with the outcome.

Figure 1 shows an example of how the painter Seurat demonstrated a scientific approach to painting called pointillism, where the artist uses combination of color dots to create an image that is harmonious and effective, while minimizing the number of colors used. This approach is analogous to management telling their datacenter team, “I want a good looking picture where everything works together and uses as few resources as possible.”

A simple idea needs a simple metric to work. In Seurat's paintings, it is a visual test. For a datacenter, it is an efficiency value—"Tell me what the energy overhead is to run the IT equipment". Microsoft has been using this approach as long as anyone can remember, and when industry groups like The Green Grid started promoting a metric for datacenter efficiency, Microsoft was an early supporter and contributor to the standard as they had years of experience with their own datacenter efficiency metrics.