Canadians Pitch Data Center Location to Reduce Costs

Canada’s “The Globe and Mail” has an article on data centres.

Technology for Tomorrow
Business seek ways to reduce data centre costs

As software-as-service, cloud computing and streaming Internet services grow, companies big and small face exorbitant electricity costs

Ian Harvey

Globe and Mail updateTuesday, Sep. 01, 2009 08:39AM EDT

When crude oil goes up, drivers can shop around for gas but when it comes to electricity, businesses are subject to paying high rates because most operate during daylight hours when prices peak. They can't simply up and move our homes or businesses to the cheapest energy location.

One major headache for business are data centres, those special secured and cooled rooms with rack upon rack of monolithic black towers humming away amidst a tangle of cables that consume vast amounts of electricity.

“Up to 30 per cent of energy costs in a business can come just from the servers,” says Bill St. Arnaud, Chief Research Officer at CANARIE (Canada's Advanced Internet Development Organization). “And if that's dirty power – from coal or oil – under Carbon Tax legislation proposed in the U.S. it could be triple the cost it is now.”

The article sites a variety of industry experts and facts to educate the reader on data centre issues. Then closes with a pitch for Canada vs. NY’s Buffalo win of Yahoo.

Similarly, Yahoo is building a 200,000 square foot datacentre outside Buffalo because it will save $100-million over 15 years by accessing Niagara Falls' sustainable and affordable hydro. They're also planning to offset cooling costs by using the frigid Buffalo winter air.

Canada has many suitable locations, says Mr. St. Arnaud, pointing to our abundance of hydro, cold climate, deep lakes and river, political and geographic stability.

Read more

Mobile Network Data Centers can save 42% Energy

GigaOm Pro references a Pike Research report that says 42% energy can be saved in the mobile network data centers.

Report: How Mobile Networks Can Cut Carbon This content requires a paid GigaOM Pro subscription

Summary:

This Pike Research report focuses on the direct impact of green technologies and practices as applied to mobile telecommunications networks, with an emphasis on the opportunity to reduce carbon emissions from network operations. Within mobile networks, base stations and switching centers consume 70-80 percent of an operator’s network energy usage, so improvements here are critical.

The summary continues.

As operators concentrate on improvements in radio frequency (RF) amplifiers, new network architectures and topologies, fresh air cooling solutions and the use of sustainable energy solutions for off-grid locations, Pike Research believes that a significant opportunity exists to dramatically improve the efficiency and environmental impact of mobile networks. Our analysis indicates that there are sufficient technology and process improvements that could reduce 2013 infrastructure emissions by at least 101 million tons of carbon dioxide equivalent (MtCO2e), a decrease of 42 percent from business-as-usual (BAU) trendlines. Other key factors supporting this trend include government emissions mandates in most parts of the world, along with operators’ increasing shift away from capex-only business case analysis to a total cost of ownership (TCO) approach for purposes of calculating return on investment (ROI) for major infrastructure upgrades. This report examines some of the key opportunities and business case scenarios for achieving these reductions.

The Pike Research report web site has more details.

Green Telecom Networks

Energy Efficiency, Renewable Power, and Carbon Emissions
Reductions for Fixed and Mobile Telecommunications Networks

Green TelecomEnergy consumption is one of the leading drivers of operating expenses for both fixed and mobile network operators.  Reliable access to electricity is limited in many developing countries that are currently the high-growth markets for telecommunications.  At the same time, many operators have adopted corporate social responsibility initiatives with a goal of reducing their networks’ carbon footprints, and network infrastructure vendors are striving to gain competitive advantage by reducing the power requirements of their equipment.  All of these factors will continue to converge over the next several years, creating significant market potential for greener telecom networks.

Read more

Open Source Storage Hardware

Storage is typically 10% of IT load, but storage can be quite expensive from a cost and power & cooling requirement if you use an EMC or NetApp storage appliance. A change though is the Open Sourced model coming to storage appliances as GigaOm reports.

Are You Ready for Open Source Hardware?

By Om Malik | Tuesday, September 1, 2009 | 6:00 AM PT | 0 comments

According to the Chaos Theory, in a giant system that lots of interconnections, even the smallest effect can cause a massive impact. It is more simply described by the Butterfly Effect. This theory has taken its toll on the software business, thanks to the rise of the open source software platforms. Today, I learnt about a move made by Backblaze, a small San Francisco-based online back-up service that can cause a similar disruption in the storage industry.

The company, whose primary business is selling online storage to consumers for a small monthly fee today announced that it is giving away the design of its storage cluster for anyone to use, modify and build upon. The design allows anyone to build large storage clusters – from a few terabytes to over a petabytes. What’s so disruptive about this. What if I told you that you could build a petabyte sized cluster for around $120,000.

Now compare that to a couple of million dollars to a storage company like EMC Corp. or a server maker such as Sun Microsystems. The image below actually does a much better job of making a comparison between the Backblaze-solution and other commercial storage options.

costofapetabyte.gif Actually if this works, companies like NetApp and EMC could be in trouble. Just like Linux slowly eroded away the premiums charged by the likes of Sun, these storage giants could see their business get negatively impacted. As the IT world transitions to a cloud-based computing, the need for web-scale storage systems is going to increase. Google, for instance has shown that you can build gigantic storage systems out of commodity parts and smart software.

A more critical view comes from StorageMojo.

Cloud storage for $100 a terabyte

by Robin Harris on Tuesday, 1 September, 2009

Imagine cloud storage that didn’t cost much more than bare drives. High density storage with RAID 6 protection, reasonable bandwidth and web-friendly HTTPS access.

And really, really cheap.

Raw disk cost is only 5-10% of a RAID systems cost. The rest goes for corporate jets, sales commissions, 3 martini lunches, tradeshows, sheetmetal, 2 Intel x86 mobos, obscene profits and a few pale and blinking engineers in a windowless lab who make the whole thing work.

Storage for ascetics
But let’s say you didn’t want the 3 martini lunch or the barely-clad booth babes. All you want is really cheap economical, reasonably reliable storage.

You aren’t running the global financial system – what’s left of it anyway – and you don’t have a 2500 person call center hammering on a few dozen Oracle databases 7 x 24. No, you’re thinking a quiet cloud storage business for SMB’s, maybe backup and some light file sharing, that will give you a nifty little revenue stream with annual renewals so you can see trouble coming 12 months in advance.

Enough redundancy so when something breaks you can wait until morning to fix it instead of an 0300 pajama run to the data center. Easy connectivity so you aren’t blowing the savings on Cisco switches.

Part of being open is the backblaze blog entry.

What Makes a Backblaze Storage Pod

A Backblaze Storage Pod is a self-contained unit that puts storage online. It’s made up of a custom metal case with commodity hardware inside. Specifically, one pod contains one Intel Motherboard with four SATA cards plugged into it. The nine SATA cables run from the cards to nine port multiplier backplanes that each have five hard drives plugged directly into them (45 hard drives in total).
Backblaze Pod Items

Above is an exploded diagram, and you can see a detailed parts list in Appendix A at the bottom of this post. The two most important factors to note are that the cost of the hard drives dominates the price of the overall pod and that the rest of the system is made entirely of commodity parts.

Wiring It Up: How to Assemble a Backblaze Storage Pod

The power wiring diagram of a Backblaze Storage Pod is seen below. Power supply units (PSUs) provide most of their power on two different voltages: 5V and 12V. We use two power supplies in the pod because 45 drives draw a lot of 5V power, yet high wattage ATX PSUs provide most of their power on 12V. This is not an accident: 1,500 watt and larger ATX power supplies are designed for powerful 3-D graphics cards that need the extra power on the 12V rail. We could have switched to a power supply designed for servers, but two ATX PSUs are cheaper.
Server Power Wiring Diagram

PSU1 powers the front three fans and port multiplier backplanes 1,2,3,4, and 7. PSU2 powers everything else. (See Appendix A for a detailed list of the custom connectors on each PSU.) To power the port multiplier backplanes, the power cables run from the PSUs through four holes in the divider metal plate that holds the fans at the center of the case (near the base of the fans) and then continue to the underside of the nine backplanes. Each port multiplier backplane has two molex male connectors on the underside. Hard drives draw the most power during initial spin-up, so if you power up both PSUs at the same time, it can draw a large (14 amp) spike of 120V power from the socket. We recommend powering up PSU1 first, waiting until the drives are spun-up (and the power draw decreases to a reasonable level), and then powering up PSU2. Fully booted, the entire pod will draw approximately 4.8 amps idle and up to 5.6 amps under heavy load.

Below is a picture of a partially assembled Backblaze Storage Pod (click on the photo for a larger image). The metal case has screws mounted on the bottom, facing upward, where we attach nylon standoffs (the small white pieces in the picture below). Nylon helps dampen vibration, and this dampening is a critical aspect of server design. The circuit boards shown on top of the nylon standoffs are a few of the nine SATA port multiplier backplanes that take a single SATA connection on their underside and allow five hard drives to be mounted vertically and plugged into the topside of the board. All the power and SATA cables run underneath the port multiplier backplanes. One of the backplanes in the picture below is fully populated with hard drives to show the positioning.

Backblaze Server Partial Assembly

A note about drive vibration: The drives vibrate too much if you leave them sitting as shown in the picture above, so we add an “anti-vibration sleeve” (essentially a rubber band) around the hard drive in between the red metal grid and the drives. This seats the drives tightly in the rubber. We also lay a large (16″ x 17″ x 1/8″) piece of foam along top of the hard drives after all 45 are in the case. The lid then screws down on top of the foam to hold the drives securely. In the future, we will dedicate an entire blog post to vibration.

The SATA wiring diagram is seen below.
SATA Wiring Diagram
The Intel Motherboard has four SATA cards plugged into it: three SYBA two-port SATA cards and one Addonics four-port card. The nine SATA cables connect to the top of the SATA cards and run in tandem with the power cables. All nine SATA cables measure 36 inches and use locking 90-degree connectors on the backplane end and non-locking straight connectors into the SATA cards.

A note about SATA chipsets: Each of the port multiplier backplanes has a Silicon Image SiI3726 chip so that five drives can be attached to one SATA port. Each of the SYBA two-port PCIe SATA cards has a Silicon Image SiI3132, and the four-port PCI Addonics card has a Silicon Image SiI3124 chip. We use only three of the four available ports on the Addonics card because we have only nine backplanes. We don’t use the SATA ports on the motherboard because, despite Intel’s claims of port multiplier support in their ICH10 south bridge, we noticed strange results in our performance tests. Silicon Image pioneered port multiplier technology, and their chips work best together.

And the software stack.

A Backblaze Storage Pod Runs Free Software

A Backblaze Storage Pod isn’t a complete building block until it boots and is on the network. The pods boot 64-bit Debian 4 Linux and the JFS file system, and they are self-contained appliances, where all access to and from the pods is through HTTPS. Below is a layer cake diagram.
Software Layering Cake Diagram
Starting at the bottom, there are 45 hard drives exposed through the SATA controllers. We then use the fdisk tool on Linux to create one partition per drive. On top of that, we cluster 15 hard drives into a single RAID6 volume with two parity drives (out of the 15). The RAID6 is created with the mdadm utility. On top of that is the JFS file system, and the only access we then allow to this totally self-contained storage building block is through HTTPS running custom Backblaze application layer logic in Apache Tomcat 5.5. After taking all this into account, the formatted (useable) space is 87 percent of the raw hard drive totals. One of the most important concepts here is that to store or retrieve data with a Backblaze Storage Pod, it is always through HTTPS. There is no iSCSI, no NFS, no SQL, no Fibre Channel. None of those technologies scales as cheaply, reliably, goes as big, nor can be managed as easily as stand-alone pods with their own IP address waiting for requests on HTTPS.

Read more

Data Center Job Insecurity, Risk to health bigger than losing the job

Data center uptime is an obsession by any enterprise who has measured the revenue loss when their data center is down. Unfortunately, this can turn into turning up the pressure and stress on the data center staff as they could lose their job for a mistake.

MSNBC and LiveScience have an interesting study that discusses job stress is worse for your health than no job.

Worry over job is worse for health than no job

Uncertainty can lead to health woes, depression, study finds

By Robert Roy Britt

updated 1:53 p.m. PT, Fri., Aug 28, 2009

Simply worrying about losing your job can cost you your health, a new investigation of data from two long-term studies finds.

Surprisingly, the effect is worse than actually losing your job, the research suggests.

"Based on how participants rated their own physical and mental health, we found that people who were persistently concerned about losing their jobs reported significantly worse overall health in both studies and were more depressed in one of the studies than those who had actually lost and regained their jobs recently," said Sarah Burgard, a sociologist at the University of Michigan.

What I challenged any data center operator to do is to measure the mental health of its staff as an indicator of the potential risk to the site.  Human error is still the largest cause of data center outages, and job stress is a leading contributor to problems.

The article continues with another point about a tough job.

If you're feeling good about your job's prospects, here's one more thing to stress about: Other research has shown that the stress of a tough job — long hours and high pressure to perform — can also ruin your health.

This reminds me the good data center managers I’ve met have a genuine concern for their employees well-being.

Read more

Children in your Data Center, Spot the TOT

MSNBC republished a BusinessWeek article that brings up a challenge almost all of us have run into.

How to parent your terrible toddler of a boss

Five signs you have a Terrible Office Tyrant and ways to manage up

By Lynn Taylor

updated 4:47 a.m. PT, Fri., Aug 28, 2009

If you've been hitting the snooze button lately on weekday mornings instead of hitting the shower — or find yourself taking the long way around to avoid passing by the corner office, you may just be working for a TOT, that is, a "Terrible Office Tyrant."

TOTs are bosses who act strikingly similar to children, oftentimes toddlers in their Terrible Twos. Why does this happen? Because we're all human, and behind the professional facade are grown kids who act out and can't moderate their power. Unfortunately, at some point from 9 to 5, they just cannot allow the child within to stay there. So in the workplace, these tykes ruin your day and wreak havoc on office productivity.

And, it is not just you who don’t like the TOTs. CEOs as well see the problems.

CEOs are not particularly fond of TOTs either. Childish managers sap productivity and hurt the bottom line. I advise CEOs to TOT-proof their company by making it safe for success — so that employees can make mistakes, communicate, and innovate. Where TOTs lurk, so does turnover, absenteeism, loss of customers, poor employee recruitment and retention, and profit erosion.

The first step is spot the TOT, and commit to being TOT-free.

The first step is spotting a TOT. At first glance, your boss's childish behaviors can be mistaken for a sporadic outburst. But after a while, you'll observe a pattern. Fortunately, by recognizing the parallel between out-of-control kids and bosses, you'll discover that the same basic techniques often work effectively for both.

Conversely, a firm dedicated to being "TOT-free" is successful, progressive, worth investing in and working for, as evidenced by countless lists of desirable, profitable places to work and most admired companies.

Next is what do you do?  There are five different scenarios –Tantrums, Demanding, Needy, Stubborn, and Distracted. The last one can be the most frustrating.

Distracted
You steer clear of the boss's door because she requested privacy as she puts the finishing touches on her report. Three hours later you enter her office and not only has she neglected her work, but she is crafting the world's longest paper clip chain. TOTs like this suffer from BADD — Boss Attention Deficit Disorder. They're only interested in what seems important at any given moment in time and have trouble paying attention to you.

The author summarizes

There's a good chance you've encountered more than one TOT in you career. Who knows, maybe you even act like one yourself on occasion. Just remember, to "TOT is human," and anyone can fall prey to it. Everyone can play a role in humanizing the workplace with greater sensitivity to what's really behind that facade.

Can you spot the TOT in your data center project?  I bet you found multiple TOTs, and you sit with your peers wondering why these people can’t act like adults.

The best data centers I’ve seen have commitments to be TOT-free. 

Read more