Don't Blame the Geeks, Management Deserves the Heat for Failure to Identify, says WSJ

The WSJ had an article about the failure of Wall St's Credit risks, and the blame does not fall on the geeks, but management. I liked the article because you can apply most of what is said to the issues of energy efficiency in the data center. Don't blame your energy geeks, its management that causes the energy inefficiency behavior.

The stereotype of a bank risk manager is a geek scrawling Greek letters on a whiteboard. But mathematical errors by the pocket-protector crowd aren't to blame for Wall Street's woes. Regulators from five countries just published a report analyzing 11 banks' risk management practices. From their conclusions, it appears the losses were due to amateurish management blunders.

[Views]

First, the big losers didn't have effective firm-wide systems for collecting data on, and evaluating, their risks. They allowed business heads too much leeway in setting and enforcing risk limits, and didn't work to break down bureaucratic barriers that kept bad news from flowing upwards. The result was a profusion of disparate businesses indulging their own short-term appetites for profits, largely immune from having their performance evaluated on a risk-adjusted basis.

Those that dodged bullets were constantly updating and tweaking their models and used them to supplement, rather than replace, their market judgment.

It might have been more comforting if the regulators had been able to blame the mess on some poor quant's slide-rule mishap. Such a mistake could be easily corrected. The periodic recurrence of banker stupidity is a less tractable problem.

Read more

Will Higher Energy Efficiency Increase Consumption?

SearchDataCenter.com's Mark Fontecchio writes Has green computing increased data center energy consumption?

Some skeptics of the green computing wave say that more energy-efficient data centers won't necessarily minimize overall energy consumption. In fact, they say it will lead to greater energy use. There's an important factor missing from the equation, say green-computing detractors: increasing consumer demand.

Mark cites Christian Belady and Lewis Curtis in his article. On first read I've accepted Christian and Lewis's ideas when I read them a couple of months ago.

Their argument goes like this: The more efficient a product, the more you use it, and it ends up consuming more resources overall than it did when it was less efficient. Christian Belady, a former technologist at Hewlett-Packard Co. who now works at Microsoft, compared data center energy consumption to the cost of gasoline. If the price of gas were to decline as significantly as the price of compute performance per watt has for IT equipment, you'd probably put a generator in your house and run everything on gasoline instead of electricity.

Coincidentally, I found articles about Boeing's planes

One of the concerns voiced about the advent of the new, more fuel-efficient aircraft (as well as its apparent popularity, as orders are outpacing analysts’ earlier projections) is the potential for a “rebound effect” similar to that seen with increases in fuel efficiency in light-duty vehicles—i.e., the decrease in fuel consumption (and thus, operating costs) leads to an increase in vehicle miles travelled.

and Cars on the same subject of increased efficiency will increase consumption.

Meanwhile, the bulk of the money spent on transportation infrastructure was directed to building more and bigger highways. We could have subsidized bullet trains and more light rail systems, but we didn't.

Now, many of the environmentalists, politicians and scientists who made the case for boosting vehicle fuel efficiency are turning their attention to the problem of how much we drive -- and the legacy of 20th century land use and transportation choices.

So, do I agree with Christian and Lewis. I think the issue can be looked at another way like the Rashomon Effect. There are multiple forces driving increases in data center services with many of the services free.  So, even though the data center operators decrease their costs by having higher efficiency, the cost impact is not visible to the user. Given most data centers don't monitor their power bills, I don't think they even know they are more efficient, so the analogy of gas efficiency doesn't necessarily apply.

Let me propose another question

Is being Green (traveling less, using the web more) driving up data center consumption?

To stir things up I'll send this post to Lewis and Christian. Plus, I am going to have dinner with Lewis Curtis this week, and we can this subject. Maybe, I can find out how his latest Green Data Center presentation went at TechReady 6. 

On a side, it would be interesting to know if Prius drivers do drive more. I found this poll. And, I may ask this question given I am writing this blog from a Toyota Dealer.  :-)

View Poll Results: How many miles do you drive your Prius

<5,000 miles/year

4
2.72%

5,001 - 7,500

13
8.84%

7,500 - 10,000

17
11.56%

10,000 - 15,000

36
24.49%

15,000 - 20,000

26
17.69%

20,000 - 30,000

34
23.13%

30,000 - 40,000

15
10.20%

>40,000

2
1.36%

Voters: 147. You may not vote on this poll

Read more

Can you Feel the Energy Efficiency?

I was in a late night IM session with Microsoft's Lewis Curtis and we were discussing Power Management.  In my past life as a Microsoft employee, one of my jobs was program manager and software evangelist for Windows 2000 power management feature so I know the features, and even though Windows Vista and Windows Server 2008 have improved their power management capabilities. I wonder if they're still missing the target.

The problem is if you save let's say 50% of your energy consumption on your laptop because it sleeps/hibernates more aggressively, does the user feel like they are saving energy? The Toyota Prius works, because the Dashboard gives you real time feedback on how efficient you are. As a Prius driver you can make a conscious decision on how energy efficient you are.

So, is what is needed is an energy efficiency user interface?  Addressing the issue of "Can you feel the energy efficiency?"

On the desktop, the one company who I think may do this would be Apple. They have ability to develop the HW, modify the OS, make BIOS changes, and the hardest part make sure the drivers work in the right way. Plus Steve Jobs will do something if he thinks it feels right. It's not logical, but it sells the user experience.

What do you think the UI should be for the Data Center?  Is it Power Usage Effectiveness, PUE? Do the users care about PUE, or do they need something else because PUE works for total data center operations? I want to discuss this with a few who have energy monitoring in data centers to get their opinions.

OK, a few confessions came up when I was writing this.

  1. I worked on the Mac Portable while at Apple. (PC World rated the Macintosh Portable as the seventeenth worst tech product of all time)
  2. I don't own a Prius.  But, I only drive 3,000 miles a year for the past 15 years.  So, my environmental impact would be bigger if I bought a Prius and didn't drive it much.
  3. In hindsight, I could have done a much better job on Win2K power management features.
  4. I need to work on my blog post for greenest thing I've done.
Read more

HP's Consolidation Project runs into Organizational Issues, needs CEO Support

WSJ writes an article about Taming Technology Sprawl and how consolidation is HP's main method to save energy and costs.

Consolidation is a strategic tenet of H-P Chief Executive Mark Hurd, who has trimmed costs and improved operations since he was named to the post in early 2005. Mr. Hurd's goal for the IT project: Cut the percentage of annual revenue spent on IT by more than half. In 2005, Mr. Hurd says, H-P spent $4.2 billion -- about 5% of 2005 revenue -- to maintain its IT systems; he wants that to drop to 2% by the end of this year.

They ran into typical issues like underestimating the # of computer programs and not allocating a large enough budget as a result.  But, the bigger problem they ran into is vice presidents who didn't want to take orders from the CIO.

In H-P's case, obstacles surfaced as early as December 2005. At the time, several vice presidents "really dug in" and resisted, says Mr. Mott. Some units said, "'We're not going to give you a cost-benefit analysis (for why we use the IT we have). We're just going to tell you what product we want,' " the chief information officer says.

In the end HP needed support from their Board and CEO to make the project continue.

To address such issues, Messrs. Hurd and Mott got a mandate supporting the project from H-P's board. Mr. Mott says he also learned the importance of telling employees about the consequences of not cooperating. He told difficult executives that doing a cost-benefit analysis of their IT use "isn't really a choice," and, backed by Mr. Hurd, threatened some with termination. "Saying 'this is a policy and if you don't follow it you'll be in violation' was a powerful thing," Mr. Mott says.

"There are going to be booby traps all along the way if you have a culture like we do at H-P," Mr. Hurd cautioned the chief information officers, adding that the solution is to get management support from the top. "Getting the CEO lined up is hard, and that's the key person," he said.

This is a good lesson to learn for a big Green/Energy saving project in your data center like a consolidation project.  Without support from your executive staff, the VPs and other business unit owners are going to be one of your top organizational issues in going Green.  As soon as you start talking about PUE, and equipment efficiencies executives will wonder why you are bothering them with the details.  At the core of any green program is change.  And change will be resisted by organizations who don't see the downside of not supporting the change. Don't bother the execs with the technical details, focus on the issues of instituting changes required to support a Green Data Center.  List those changes that need to be supported, and get the support from your management.

Note: part of the inspiration for this post is a question from a friend who wants to discuss Green Data Centers at an executive conference.  We're stuck in that the details appeal to CIO and his staff, but without the buy-in from the CEO too many projects will fail as business units resist the changes required to go Green.

Read more

Dynamic PUE real world use

I've been meaning to write about PUE, and have been stumped in that It is defined as a metric, and in the Green Grid document referenced it makes no reference that is dynamic. In reality PUE will be a dynamic # that changes as the load changes in a room. How ironic would it be that your best PUE # is when all the servers are running at near capacity, and shutting down servers to save power will increase your PUE? Or your energy efficient cooling system uses large amounts of water in Southern California where it is just a matter of time before water shortages will cause more environmental issues?

What helped me to think of PUE as a dynamic # is to think of it as quality control metric. The quality of the electrical and mechanical systems and their operations over time are inputs into PUE.  As load changes and servers will be turned off the variability of the power and cooling systems influence you PUE.  So, PUE can now have a statistical range of operation given the conditions.  This sounds familiar.  It's statistical process control.

Statistical Process Control (SPC) is an effective method of monitoring a process through the use of control charts. Much of its power lies in the ability to monitor both process centre and its variation about that centre. By collecting data from samples at various points within the process, variations in the process that may affect the quality of the end product or service can be detected and corrected, thus reducing waste and as well as the likelihood that problems will be passed on to the customer. With its emphasis on early detection and prevention of problems, SPC has a distinct advantage over quality methods, such as inspection, that apply resources to detecting and correcting problems in the end product or service.

For example, a breakfast cereal packaging line may be designed to fill each cereal box with 500 grams of product, but some boxes will have slightly more than 500 grams, and some will have slightly less, in accordance with a distribution of net weights. If the production process, its inputs, or its environment changes (for example, the machines doing the manufacture begin to wear) this distribution can change. For example, as its cams and pulleys wear out, the cereal filling machine may start putting more cereal into each box than specified. If this change is allowed to continue unchecked, more and more product will be produced that fall outside the tolerances of the manufacturer or consumer, resulting in waste. While in this case, the waste is in the form of "free" product for the consumer, typically waste consists of rework or scrap.

By observing at the right time what happened in the process that led to a change, the quality engineer or any member of the team responsible for the production line can troubleshoot the root cause of the variation that has crept in to the process and correct the problem.

This last point of observing at the right time what happened in the process that led to a change ultimately what needs to be achieved with a dynamic PUE system.  Without a system like this and mindset, you wouldn't know how to fix PUE problems. Which is what I think is wrong with a static PUE mindset.  You need a closed loop feedback to monitor the PUE and see if it is performing as expected given the operating conditions and load.

Note: the point about breakfast cereal reminds of Microsoft's Mike Manos, Sr. Director Data Center Services, and his first job working in Rice a Roni operations, learning process control, which is probably why he has invested in software from OSIsoft to help monitor PUE.  Cornell uses the same SW as well.  For more details see Microsoft's Jeff O'Reilly presentation or Cornell's Jason Banfelder presentation.

Read more