Illogic or Logic of building your own servers

Facebook is creating more visibility for build your own servers.

The logic of the approach is in this Bloomberg article.

Dell Loses Orders as Facebook Do-It-Yourself Servers Gain: Tech

By Ian King and Dina Bass - Sep 11, 2011 9:01 PM PTMon Sep 12 04:01:00 GMT 2011

When Facebook Inc. set out to build two new data centers, engineers couldn’t find the server computers they wanted from Dell Inc. (DELL) or Hewlett-Packard Co. (HPQ) They decided to build their own.

“We weren’t able to get exactly what we wanted,” Frank Frankovsky, Facebook’s director of hardware design, said at a conference on data-center technology last month.

Will this logic be successful is countered by ZDNET “between the lines”

Facebook DIY servers really poaching from Dell, HP, IBM? It's too early to tell

By Larry Dignan | September 12, 2011, 3:50am PDT

Summary: Do-it-yourself servers designed by Facebook are allegedly poaching server sales from HP, IBM and Dell, but the data is inconclusive at best.

And Larry makes an excellent point which I totally agree on.

Bloomberg reports that do-it-yourself servers used by the likes of Facebook, Google and Microsoft in data centers threaten Dell, HP and IBM. When I saw the headline, I got excited. Why? I thought there would be some quantification in it. Aside from the fact that 20 percent of the server market is customized, it’s unclear how many orders Dell, HP and IBM were really losing. There aren’t any concrete examples or figures to back up the premise.

You can argue the Logic of build your own servers or the Illogic of build your servers.  But from what I am hearing and seeing looking at other indicators of what is going on.  The momentum to build your own servers is growing.  Companies are putting the infrastructure in place to get it their way.

Hold the ‘Pickles’

“People want to be able to build it their way,”Frankovsky said at the Dell-Samsung Chief Information Officer Forum in Half Moon Bay, California. “They kind of want a Burger King: ‘I don’t like pickles -- why do I have to have pickles?’”

Building your own servers is a niche that is addressing a problem the Server industry has seen in the lack of R&D which results in a commoditization of the servers and lack of innovation.

Do you see Google going back to buying servers from OEMs?

For the same reason Apple has changed smartphones, tablets, and computers by integrating SW and HW, isn’t it logical to integrate SW and HW design in the data center?

Think Carnegie and Vertical Integration.

One of the earliest, largest and most famous examples of vertical integration was the Carnegie Steel company. The company controlled not only the mills where the steel was made, but also the mines where the iron ore was extracted, the coal mines that supplied the coal, the ships that transported the iron ore and the railroads that transported the coal to the factory, the coke ovens where the coal was cooked, etc. The company also focused heavily on developing talent internally from the bottom up, rather than importing it from other companies.^[1] Later on, Carnegie even established an institute of higher learning to teach the steel processes to the next generation.

Many of the companies looking to build their own servers use open source so they have control of the supply chain of SW.  They want control of hardware.

Intelligence community transformed from "need to know" to "responsibility to share," shouldn't data centers follow?

9/11 had many affects on people and organizations.  One of those who has had to change the most is the Intelligence Community as they bear blame for not stopping the 9/11 terrorist attacks. WSJ has an article written by James R. Clapper, Director of National Intelligence.  The part that caught my attention was:

How 9/11 Transformed the Intelligence Community

It's no longer about 'need to know.'

Our guiding principle is 'responsibility to share.'

...

We no longer operate largely on the principle of compartmentalization, that is, sharing information based on "need to know." We now start from the imperative of "responsibility to share," in order to collaborate with and better support our intelligence consumers—from the White House to the foxhole.

How many problems are caused in data centers where the standard operating procedure is "need to know."  If the US intelligence community has shifted to "responsibility to share" to solve their integration problems, maybe others should try the same approach.

Prior to the 9/11 attacks, the community had recognized that reorganization, integration of intelligence activities, and a shift in intelligence culture was necessary to adapt to evolving threats. But progress on these initiatives came slowly—too slowly to impact the events of 9/11.

The intelligence community got the message.

...

We can't know with absolute certainty if any of these changes would have led to a different outcome on 9/11, but the tangible benefits of vertical and horizontal integration are indisputable. Today we are unquestionably better positioned to provide the kind of full-scope information that leaders need to make informed decisions about how to protect our nation.

How big is a problem of theft in the data center?

I have been having a variety of discussions with people who have insight into how bad a problem inventory shrinkage is in the data center.  Inventory shrinkage is a term used in retail to explain inventory that disappears, hence “shrinks.”  Given retails thin margins shrinkage is a big deal.

Security and Crime News

Return to news menu

Retail Theft and Inventory Shrinkage

2002 Retail Security Survey Shows U.S. Retailers Losing $31 Billion to Theft 

Inventory shrinkage, a combination of employee theft, shoplifting, vendor fraud and administrative error, cost United States retailers over $31 billion last year according to the latest National Retail Security Survey report on retail theft, which analyzed theft incidents from 118 of the largest U.S. retail chains.

According to University of Florida criminologist Richard C. Hollinger, Ph.D., who directs the National Retail Security Survey, retailers lost 1.7 percent of their total annual sales to inventory shrinkage last year. The surveyed portion of the retail economy transacts over $1.845 trillion dollars annually, making the loss worth over $31.3 billion. Total inventory shrinkage was down slightly from $32.3 billion in 2000.

Data Centers tend to feel secure because they think their security systems eliminate theft.

Where Inventory Shrinkage Happens

Source of Inventory Shrinkage	% of Loss*	$ Lost
Employee Theft	48.5%	$15.1 billion
Shoplifting	31.7%	$9.7 billion
Administrative Error	15.3%	$4.8 billion
Vendor Fraud	5.4%	$1.7 billion
Total Inventory Shrinkage		$31.3 billion

But, even if you eliminate all theft, 20% of inventory shrinkage comes from administrative error and vendor fraud.

To give you an idea of data center theft, Seagate has a paper on stolen drives and servers.

Stolen Drives and Servers Don’t Think it Can’t Happen in Your Data Center  September 2007

Executive Summary

Almost every organization is well aware of the risk to confidential data stored on mobile devices such as notebook PCs that can be lost or stolen.  But few organizations realize that drives or even entire servers are vulnerable to theft, loss, or maintenance mix-ups despite the “protection” of residing in the organization data center.  Of course, that means that the confidential data stored on those devices is subject to unauthorized use by the growing army of cyber criminals.  Because data centers contain the most concentrated data in the organization, such thefts can be catastrophic in terms of financial, regulatory and legal consequences.  Even small incidents can necessitate high costs of remediation because when such thefts occur it is extremely difficult to determine what was compromised, so the “worst case” scenario must be assumed.

The paper goes on to point out the risk in data centers.

Servers can be at risk even inside large, well-managed facilities Despite standard physical security measures employed at data centers, there are still many opportunities for insiders or skilled thieves to steal important servers and drives, even during normal hours of operation.”  For example, when systems are being expanded or modified, there are frequently large numbers of contractor technicians who carry equipment in and out and have the opportunity to remove drives or servers with few questions asked.

How bad of a problem do you have?  When asking companies who handle decommissioning of equipment it is common for them to find between 20 - 40% of the servers are not what users think they are.  What types of problems?  Missing RAM, HD, and processors.  Incorrect replacement parts - power supplies, cables.  How many of your inspect warranty service?

PG&E fires four inspectors - just another example of why you need to audit work in the data center

I am constantly amazed when critical infrastructure in the data center work is not audited.  Domenic Alcaro's talk at 7x24 about submarine maintenance and ideas that apply to the data center, the inspection and auditing of work is standard procedure.

SJ Mercury news discusses the situation PG&E has found when they audited their inspectors.

PG&E fires four inspectors

By Steve Johnson

sjohnson@mercurynews.com

Posted: 09/01/2011 02:07:48 PM PDT

Updated: 09/01/2011 02:54:21 PM PDT

PG&E has fired four inspectors after an internal investigation determined they falsely claimed to have inspected the company's underground electrical gear, and four others were disciplined.

The probe, which PG&E launched in November after receiving an employee's tip about the phony inspections, was first disclosed publicly in June. Over the past two months, the company reinspected nearly 25,000 underground enclosures containing various electrical equipment and found 25 where inspectors had falsely claimed to have inspected them, according to PG&E spokesman Andrew Souvall.

...

"In the four instances where employees were terminated, we found substantial evidence that they had not properly conducted their inspections and had falsified records," Souvall said.

Some of my friends and I have been thinking about how to solve this type of problem in the data center, and we are circulating the solution to some early adopters.