Microsoft Shares Insight on its Data Center Chargeback System

August 28, 2008 Dave Ohara

The Power of Software blog has a new post by Amaya Souarez. There is so much information in this post, it is hard to summarize, so let’s start with a list, extracting words from the post.

Green is a priority.

I believe that implementing chargeback models based on power usage will encourage customers to consider power efficiency more seriously and reduce our overall impact on the environment.

Interesting analogy

I would like to illustrate this point by using the analogy of getting a first car for a teenager. By the way, if you currently have one of these creatures at home, then you have my commiserations.

Comparing Chargeback Models

Let’s say you are investigating the conditions under which you will allow your teenager to have a car. Consider the following two options:

The teenager buys the car, but you pay for the gas.

You buy the car and the teenager pays for the gas.

For each option, what sort of car do you think your teenager will want?

With option 1, you may find that they come back with a clapped out 351 cubic inch monster with the thirst of a Jentil, maybe something like the “striped tomato” from Starsky and Hutch. And who cares? You’re paying for the gas, right?

But there’s no point in having the coolest car in the neighborhood if you can’t even afford to take a date to a drive-in movie. If you select option 2, you may find your teenager develops a more healthy interest in how many miles per gallon (MPG) something more sensible can manage (like the new Prius hybrids at the Redmond Campus), rather than whether they have enough torque to leave tire slicks longer than a 747 landing at Princess Juliana International airport.

Statement of the problem.

With servers, the situation is more complex; the main issue that we struggle with is the fact that there is no direct equivalent to miles per gallon. How do you measure application output? It isn’t just about CPU utilization. What about an application that makes repeated calls to hard disk but doesn’t use much in the way of processor resources? If you fit more memory to the server, it can may be able to cache the disk access calls, but you’re then consuming more power in the additional memory module.

All the standard performance monitoring areas, such as processor, memory, hard disk, network, and cache make varying contributions to power consumption. What we needed at Microsoft was a chargeback model that is easy to understand, straightforward to administer, and allocates data center costs to customers proportionately.

Confession of the issues.

For the Microsoft data centers, the effort to change our chargeback model was not a simple conversion, as it took us one and a half years to move from our previous model of charging for floor space based upon rack utilization to the new model. Not surprisingly, it was not the tooling or process modifications that posed the biggest hurdle but the cultural and political changes that were required. Even today, I frequently have to remind customers that ‘DC space is power.’

Final action.

The final model that we now use has two basic components:

Floor Space. This component is billed per kilowatt (kW) of usage and includes all the floor space costs.

Power and Cooling. This component is billed per kilowatt hour (kWh) of usage and includes the cost of electricity as billed by our energy suppliers.

I’m not using real figures here, but you should be able to see the basis of how we implemented charging based on power consumption.

And there is plenty more in the blog entry.

Amaya Rocks!

Here are her final words.

Summary

Changing our chargeback model to one that uses power as the basis for floor space makes sense, both for us and for our customers. As older equipment is retired and replaced, we expect to see greater emphasis on power efficiency rather than raw output. Reducing power consumption on individual servers results in a reduction in the total power consumption for the data center, helping to conserve our power bandwidth and minimize our impact on the environment.

Avanade's Anti-Greenwashing Initiative, Part 1

August 27, 2008 Dave Ohara

Avanade's Toby Velte is about to have a book published on Green IT. Here is a link to the book on amazon.com.

"Green IT is a social imperative that meets the needs of business. This book guides organizations in applying environmentally sound practices to business and technology decisions--helping them reduce their consumption of resources, energy dependencies, and costs. These issues are real and the time to act is now." --Adam Warby, CEO, Avanade

After talking to Toby to understand his book and Avanade Tools.

Avanade has created a tool that uses more than 120 variables and real-world customer data to measure the concrete impact of Green IT projects for a customer. This deep ROI modeling tool measures the true impact that virtualization and data center optimization projects can have on costs, energy use, carbon footprint, etc.

I was thinking great I will write a blog entry. Where is your url for your Green IT initiative? "Well, we don't have a place on our web site." So, I can't actually link to anything on Avanade's web site.

Wait, you have a book, you have Green IT analysis tools, and you don't have a marketing site. How can anyone claim what you are doing is greenwashing when you don't have a marketing initiative? You are truly working on filtering the greenwash: anti-greenwashing.

Here is a sample screen shot from Toby's tool.

This screen is a sample of a virtualization calculation taking into account typical TCO costs, but adds specifics for kW/Hr and carbon footprint. Staring at these numbers Toby calculates data center facilities cost per sq ft. It would be more accurate to calculate the data facilities cost as a function of kW capacity.

It turns out Toby is going to be coming to meet with a software company in Redmond in two weeks, and I can meet with him face to face as my office is close to a Redmond software company too.

This is Part 1 of what I am sure will be another post after I meet with Toby and get a copy of his book.

Reduce the environmental and budgetary impact of your IT infrastructure

This groundbreaking work offers a complete roadmap for integrating environmentally sound techniques and technologies into your Information Systems architecture. Green IT explains how to adopt a business-driven green initiative and provides a detailed implementation plan. You will find strategies for reducing power needs, procuring energy from alternative sources, utilizing virtualization technologies, and managing sustainable development. Case studies highlighting successful green IT projects at major organizations are included. Keep your IT department and your organization in the green--both environmentally and financially--with help from this comprehensive guide.

Work within current global initiatives and standards for e-waste

Minimize power usage and use alternative cooling methods in your datacenter

Transition your office into a paperless environment

Equip your organization with green hardware, including EPEAT-, RoHS-, and ENERGY STAR-certified machines

Implement efficient datacenter design in terms of energy consumption, cooling, server configuration, consolidation, cabling, redundancy, and more

Virtualize servers and storage using the latest technologies from VMware, Microsoft, Compellent, Incipient, and others

Measure existing datacenter efficiency using current metrics, and track progress with Business Intelligence tools

Establish a green supply chain

Explore the Software as a Service (SaaS) model

Manage ongoing compliance and sustainable growth

I have a copy of the virtualization chapter, but I want to see what Toby has for his data center optimization advice.

4 Customers, 4 Points - Trendpoint Launches Green Data Center plan, Sharing Best Practices

August 25, 2008 Dave Ohara

Trendpoint Systems has a press release announcing their new Four Point Plan for Green Data Centers.

DATA CENTER ENERGY MANAGEMENT LEADER, TRENDPOINT, LAUNCHES FOUR POINT PLAN FOR “GREEN DATA CENTERS” AIMED AT REDUCING ENERGY, CARBON EMMISSIONS

Data centers, projected to be world’s biggest source of greenhouse gas pollution, face tough regulations and carbon taxes for “energy offenders”

San Ramon, CA, August , 2008 –Trendpoint Systems, a leader in data center energy management solutions, today announced a four-point plan aimed at dramatically reducing energy costs and carbon emissions at data centers. A recent study by McKinsey & Co. predicted that data centers will surpass the airline industry as the largest source of carbon dioxide emissions by 2020 and called for data centers to double their energy efficiency by 2012. Trendpoint projects that by implementing only a part of its plan -- focused on managing cooling costs -- data centers can reduce energy costs by a third or more.
The four point plan from Trendpoint provides a comprehensive approach for measuring, monitoring and managing energy use at data centers in order to help companies tackle this urgent problem. The plan will also enable companies to set and manage an energy budget that will allow them to comply with proposed regulations on energy use and emissions, such as those slated for enforcement in California by 2010. The state will be the first in the U.S. to follow the lead of the U.K. and other European countries in establishing “cap and trade” regulations that will impose limits on energy use at data centers and other businesses, imposing a hefty “carbon tax” for those that do not comply. Businesses that do comply will be able to trade their energy savings on the open market in an “energy exchange.”

I had a chance to interview CEO Bob Hunter to understand his product and get a better background on how Trendpoint developed its four point plan. What I found as a basis for their ideas were four showcase customers who were key in developing the plan. The four customers are VMware (NYSE: VMW), Facebook, Carillion IT Services (EDS), and Rosendin Electric.

Proof of how well the Four Point Plan works references an EDS Data center implementation.

EDS saves 25% of cooling energy costs, ROI in less than one year

Trendpoint is already proving the effectiveness of its plan and energy management products at some of the world’s largest data center “brands.” In a study done at an EDS data center in Newcastle, U.K. (conducted by site management firm, Carillion, PLC) the facility was able to cut cooling energy costs by 25% simply by using the Trendpoint energy management solution to target cooling resources to match the heat generated by servers. Hunter believes that even more savings could be realized by implementing the complete Trendpoint Four Point Plan. The Carillion study also reported an ROI of less than one year for the Trendpoint solution at the EDS facility. According to Carillion, the Trendpoint solution will be extended to a second site in the U.K, and to sites throughout the U.K. and Western Europe over the next 2-3 years.

Here are the Four Points. Keep this in mind when you think about your Green IT projects.

Set an energy budget – In the same way that companies set and manage travel budgets, data centers need to be able to set energy and carbon budgets that can be broken down along established parameters such as department, site, etc., all the way down to the circuit level. Employees accountable for those budgets need the ability to view and manage their individual budget segments securely, against assigned metrics. Co-location facilities, in particular, need to be able to access and manage their own energy and carbon budgets so that they can bill back to customers appropriately.
Virtualize servers -- Companies can reap instant savings by consolidating underutilized physical data servers onto “virtual machines” that act like physical computers, but don’t require the space, management time or energy of individual servers. But Trendpoint cautions against virtualizing without proper cooling management since virtualized servers generate enormous heat. Without proper cooling, they will develop “server thermal inversions.” Just as smog recycles and builds up in an atmospheric thermal inversion, a server can develop an inversion that causes cool air to be trapped and recycled, wasting cooling resources. “Server thermal inversion” will not only eat up all of the energy savings in increased cooling costs, but is a leading cause of costly downtime in a data center.
Equalize heat and cooling balance – Data centers waste enormous amounts of energy by overcooling the majority of their data cabinets. This is because they make macro-cooling decisions based on the heat generated by their hottest cabinets. By matching cooling resources (from floor vents or liquid cooling units) to the actual needs of each individual cabinet, data centers can realize significant savings on energy use. In addition, further savings can be achieved by balancing heat loads on an intra-cabinet basis. By grouping servers in a minimum of 2 and preferably 3 zones within a cabinet and moving towards equalized heat loads between the zones -- thereby avoiding “server thermal inversion” -- users will reap additional cooling savings of 10-30%.
Manage to the metrics – As data centers add, move, and change servers, many on a daily basis, they need to continue to monitor and manage heat generation and cooling requirements. Trendpoint provides the only solution for managing energy usage of data centers over the long term to achieve maximum energy efficiency. The Trendpoint solution also enables companies to manage their carbon and energy budgets in order to comply with corporate mandates for energy conservation and fast-approaching regulations on energy use and emissions.

Trendpoint is one of the few products that ships a hardware monitoring solution along with a monitoring server. They have found most customers prefer to have a turnkey solution with complete hardware and software.

For a more technical discussion on the merits of Trendpoints monitoring technology here is their white paper.

Summary
The measurement of true heat and energy values in data centers is vital to improving energy efficiency. True heat values are only expressed in RMS watts. Its accumulated value, kWh forms the basis for all energy billing as well as carbon to kWh comparisons. First-generation monitoring devices employed average amperage and/or average wattage data. Average wattage is a highly inaccurate measurement and cannot be relied on to provide actual heat values or billing kWh. True RMS wattage and kWh units that measure directly at the circuit-breaker have become affordable and form the basis for quality information gathering in data centers.

Tip for Building Green Applications

August 25, 2008 Dave Ohara

I just had a conversation with an industry magazine editor and had a good time discussing green IT. One of the topics we talked about was green applications, and how difficult it is to get developers to write greener code. There are no hard and fast rules as to what code is greener, but after the discussion it hit me there is one way to make a greener app.

Problem: Developers are removed from the production environment and the resources used. As CapEx and OpEx increase for power and cooling infrastructure developers are unaware that their energy use is a dominant cost beyond the cost of the hardware.

Opportunity: How do you get software developers to change their behavior?

Solution: If you use cloud computing services like Amazon Web Services or Google Application Engine, then the costs for compute, storage, and network use are easy to calculate as this is their chargeback model. As the application is developed, tested and put into production, the costs for running the solution is a metered number that has a direct relationship to the code.

Give your developers a cost per unit of work budget. If you are a startup you have limited budget and if you are trying to run a site where advertising is one of your main revenue streams, your profit will be determined by the difference between your revenue and your costs. The last thing you want is a site that costs more to run per click event than an advertising revenue click brings in.

This applies to few of the enterprise developers, but more and more companies are starting up using AWS and GAE, training a set of developers who are learning to be efficient with compute, storage, and network. Do you think these developers waste storage and network bandwidth needlessly moving data around?

If you could build an internal chargeback system for compute, storage, and network, charging the departments direct costs, then you too can motivate your developers.

If you think it is too much work for you to build your own system, here is a more radical idea, use AWS or GAE as your development environment, and you can roll out the production system in your own environment, but you used AWS/GAE to measure your solution and you outsourced the development environment. The guys at Skytap could support this model if they make changes to their chargeback model.

This is a good opportunity for the management tools companies like CA, HP, and IBM to support as well.

Can You Detect Internal Greenwashers? WSJ Discusses Made Up Problems

August 25, 2008 Dave Ohara

WSJ has an article titled Munchausen at Work.

'Munchausen at Work'

Employees Advance
By Fixing Problems
They Had Created

By PHRED DVORAK

In late 2005, the night manager of a suburban Atlanta restaurant called owner J.D. Clockdale to boast about how well she had handled an irate female customer. The customer "ranted and raved" about a botched order, but calmed down after the manager gave her a free meal, Mr. Clockdale recalls being told.

INVENTING TROUBLE

Experts say a good work environment can dissuade employees from creating "problems" that need to be solved. A few tips:

• Stress teamwork over individual problem-solving

• Be wary of creating office "heroes"

• Watch out for information hoarding

• Ensure bosses are attentive to employees' needs

Source: WSJ Research

The problem: the tale was untrue -- as Mr. Clockdale discovered by reviewing surveillance footage and phoning the customer, who was an acquaintance. He concluded the order mistake was minor and remedied without histrionics.

Mr. Clockdale confronted the night manager, who confessed that she invented the altercation to look good. "She wanted more responsibility," he says.

With the popularity of Green and problems of Greenwashing coming from suppliers, it is quite possible you have internal greenwashers who are either making up problems that they say they solve or even worse they create the inefficient operation and fix it, claiming success. This is scary to think about, but I am sure for people have been in data center industry for a while you have found problems under the category of Munchausen Syndrome.

In Munchausen syndrome, the affected person exaggerates or creates symptoms of illnesses in themselves in order to gain investigation, treatment, attention, sympathy, and comfort from medical personnel. In some extremes, people suffering from Munchausen's Syndrome are highly knowledgeable about the practice of medicine, and are able to produce symptoms that result in multiple unnecessary operations.

WSJ continues with how Munchausen syndrome manifests at work.

The story illustrates a troublesome workplace phenomenon that's now attracting attention: employees who quietly cause problems so they can later take credit for fixing them. Georgia Institute of Technology business professor Nathan Bennett dubs the behavior "Munchausen at work," because it resembles a rare psychological disorder in which sufferers seek attention by making up an illness or inducing sickness in others.

Mr. Bennett says most experienced managers he has interviewed have encountered Munchausen at work and consider it disruptive. Such actions can be hard to detect and eradicate; perpetrators may gain promotions or recognition, encouraging additional attempts. "You get the kind of behavior you reward," Mr. Bennett says.

Solving the problem like any problem first involves detection, and then monitoring the issues.

Spotting the deception is the first step toward stopping it. Health-care administrator Gary Barnes suspected an office manager was causing the problems she took credit for solving at a Pittsburgh clinic where he worked during the 1990s. The manager blamed computer glitches for a delay in depositing insurance checks, then claimed to have fixed the problem. Mr. Barnes searched the manager's desk and found she had kept the checks in her drawer; he fired her.

Keep this is mind when you architect the design of your energy monitoring systems. The problems you may have can be personnel behavior in addition to your equipment. Few people think of tracking who is doing what in the data center.

Can you track efficiency improvements correlations to a person/group?

As mentioned in the WSJ article here is what else you can do.

INVENTING TROUBLE

Experts say a good work environment can dissuade employees from creating "problems" that need to be solved. A few tips:

• Stress teamwork over individual problem-solving

• Be wary of creating office "heroes"

• Watch out for information hoarding

• Ensure bosses are attentive to employees' needs

Source: WSJ Research