Calxeda's Tough Battle was not ARM, but Intel's investment in a Fabric to reinvent servers

There is sadness in Calxeda’s lack of funding to continue its efforts on ARM servers.

Calxeda Closes Its Office: The Sad Story Of One Of The Most Innovative Companies In Years

But, over a year ago, I started to doubt whether Calxeda would survive.  I would chat with some server hardware folks and one of the ideas we would discuss is out of the 8+ companies out there with ARM chips there would eventually be only three who survive, and whether Calxeda would be one of them was not clear.  We had more faith in Samsung to be a player than many of the others.

Almost everyone focuses on ARM as if the ARM chip has magical powers that can beat the x86 chip once someone ships an ARM chip for servers.  ARM is just a chip.  x86 is a chip.  I decided to look for a recent paper that compares ARM vs. x86 and found this paper.

A Detailed Analysis of Contemporary ARM and x86 Architectures
Emily Blem, Jaikrishnan Menon, and Karthikeyan Sankaralingam
University of Wisconsin - Madison

The results show. (note: ISA is Instruction Set Architecture)

We find that ARM and x86 processors are simply engineering
design points optimized for different levels of performance, and
there is nothing fundamentally more energy efficient in one ISA
class or the other. The ISA being RISC or CISC seems irrelevant

So if ARM vs. x86 is irrelevant what was the battle to fight?  It is about creating a new Fabric for Servers.  Read the press statement from Calxeda.  Their first two paragraphs say Fabric.

Over the last few years, Calxeda has been a driving force in the industry for low power server processors and fabric-based computing.

The concept of a fabric of ARM-based servers challenging the industry giants was not on anyone¹s radar screen when we started this journey.

In April 2012 Intel announced it had licensed Cray’s supercomputing fabric.

Intel Acquires Industry-Leading, High-Performance Computing Interconnect Technology and Expertise

SANTA CLARA, Calif., April 24, 2012 – Intel Corporation today announced it has entered into a definitive agreement with Cray Inc. to acquire certain assets related to its high-performance computing (HPC) interconnect program. With the agreement, Intel gains access to Cray’s world-class interconnect personnel and intellectual property.

 

The Cray interconnect team is responsible for the award-winning Gemini interconnect as well as the upcoming Aries interconnect, designed to work in Cray’s next-generation  supercomputer, codenamed “Cascade,” which will integrate Intel® Xeon® processors. The transaction is expected to close before the end of the current quarter, subject to customary closing conditions being met.

Intel announced the C2000 Atom to go against ARM.  In this blog post I referenced posts on Barron’s blog by analyst who discussed the ARM vs. Atom.

There are probably many other small things that made it difficult for Calxeda.  It’s hard to be a disruptive force especially in Intel’s turf.

Getting executives to hear the Data Center Issues, Speaking Truth to Power

I had beers for the holidays with some data center friends last night.  We had fun chatting about all kinds of topics.  One guy is lucky to have executives who will listen to the data center issues.  One of the other guys is unlucky with management who doesn’t understand the data center.  The gal is watching craziness at the executive level at another company as they save costs.  My own situation at big companies is long gone.  My past problem at big companies is I was focused on doing the right thing in spite of management.

Just yesterday I was reading this white paper on Speaking Truth to Power and many points are made that resonated.

The truth that makes men free is for the most part 
the truth which men prefer not to hear.
--Herbert Agar, A Time for Greatness (1942)

The author writes about an authority power dominated company (Cowles Media) vs. an open challenge of authority’s decisions (FedEx) and summarizes with...

The lesson I drew from these examples nearly three decades ago was that mangers in companies with healthy cultures were constantly willing to rethink even their most basic assumptions through a process of constructive dissent.

Many companies have the belief the CEO should always be the authority with power.

thanks to the much-publicized behavior of imperial CEOs in the 1990s, a cultural expectation has been created that leaders need to be decisive, tough, take-charge men who quickly fire those who are not "team-players." Imagine the courage it would take to tell a Jack Welch, a Scott McNeely, an Andy Grove, or a Larry Ellison news he didn't want to hear? Even in a book by fawning admirers, Jack Welch came across as a modern-day Attila when GE mangers dared to question him. Dissenters were berated, insulted, and abused: "According to former employees, Welch conducts meetings so aggressively that people tremble. He attacks almost physically with his intellect-criticizing, demeaning, ridiculing, humiliating." One humiliated former GE-executive who had been publicly dressed down by Welch for daring to question his boss admitted to the moderator of an Aspen Institute seminar in the early 1990s that Welch's furious tirade "caused me to soil my pants."

The white paper is a long read, but will get you thinking about the challenges of Speaking Truth to those in Power.

ARM Server Effort runs out of Steam, Calxeda shuts down

I was lucky to get to know the folks at Calxeda early on.  Back when I first talked to them in Apr 2010 they were called Smoothstone.

Are ARM Servers the disruptive change coming to Green the Data Center? Smooth-Stone is trying

At same time i was discussing the idea that Intel Atom chips would be put in Servers.

The efforts by ARM did help to wake up Intel that low power servers were a growing market.  Back in 2010 it was all about the # of cores.  Now Performance per watt is a standard measure. RAM and HD are more energy efficient.  SSD is more common.  PUE’s of 1.10 are no big deal.  Power supplies are 90% energy efficient.  Batteries are put on server boards by Googl and others.

A lot has changed in 3 1/2 years and during this time the demand for ARM based servers is higher, but no big buyers.  Why?  Intel made changes along with the rest of the server ecosystem to use less power.  

Intel Atom’s are used in low power cold storage systems.

And the x86 code base made it so much easier to recompile code for new chip designs to improve performance vs. porting to the ARM instruction set.  64 bit processors are just showing up in ARM.  Who wants to run a server on a 32 bit chip even if it has 4 cores.

GigaOm’s Stacey Higginbotham reports on Calxeda shutting its doors.  It is kind of sad, but most of my friends who think about chip designs agreed that Calxeda had a tough road to success.  Why?  Even with $100 Mil, it doesn’t last long when you are trying to create Silicon.

ARM server pioneer Calxeda shuts its office after running out of cash

 

2 HOURS AGO

1 Comment

SUMMARY:

ARM server maker Calxeda has hit the skids after raising more than $100 million. The company said it will restructure but news reports have the firm shutting down.

Calxeda, a company that was founded in 2008 to build out ARM-based servers for the web world says it is in the process of restructuring its business after apparentlyrunning out of funding. A story by All Things D says that the company is shutting down.

We’ll see where the ARM server does take off.  In the meantime ARM dominates the mobile space.

And, Yahoo Mail story continues, Mail continues to be recovered

I wrote about how the Yahoo Mail problem seems like it could be like the Danger T-Mobile outage.  And a week later it does look more like it has amazing similarities.

Why am I continuing to follow this when almost all the rest of the media has dropped the story?  Because, I think the root cause is operation issues which is interesting to those who run mission critical services.  Think about it a storage system went out that affected 1% of the users.  Yahoo immediately restores from back-up bringing down mail for many users.  Somehow it would seem like if 1% were truly only affected, there could have been a better way to restore Mail Service.

Ironically the impact was much more than 1% loss mail.  Millions of their users had no mail for days.

Here is a replay of the events from their log.  On this thread you can see the history.

A description of the problem on Dec 9.

So, what happened?

On Monday, December 9th at 10:27 p.m. PT, our network operating center alerted the Mail engineering team to a specific hardware outage in one of our storage systems serving 1% of our users. The Mail team immediately started working with the storage engineers to restore access and move to our back-up systems, estimating that full recovery would be complete by 1:30 p.m. PT on Tuesday.

Yahoo Mail said it was up and running with updates from Marissa Mayer and the operations team that 100% restored was successful.

Update 12/14/13 10:40 am PST

Here are this morning’s updates:

  • Account Access: 99.9% of affected users may access their accounts
  • Outage Message Queue: 100% cleared
  • IMAP access: 100% restored

We're making progress on restoring full access to messages for affected customers and will update again with more information. 

+ Update 12/13/13 5:00 pm PST

We have posted an update on the Yahoo blog here:http://yahoo.tumblr.com/post/69929616860/an-update-on-yahoo-mail

Users were still complaining.  Two days later there is another update that explains the problem getting to mail.  So, even though the queues were cleared for mail from Dec 9, the older mail was not restored.

+ Update 12/16/13 9:00 pm PST

We’ve restored access for users and continue to make progress on recovering email messages, folders and inboxes for those users who are still missing messages in their inbox.

As the engineering team continues the restoration process, we wanted to give a couple answers to the top questions we’re seeing:
 

Q:  “I’m missing emails in my inbox from certain dates, but can see everything else.”
A:  There are three periods of time at question when it comes to message restoration. Message restoration for each period can follow a different timeline.

  • Emails from Dec. 9 - now: 100% of emails during this time period have been delivered
  • Emails from Nov. 25 - Dec 9, 2013: 75% of emails from this period have been restored
  • Emails prior to Nov. 25: 90% of emails from this period have been restored

After Dec 9 you have 100% of your mail.  Before that you have between 75% and 90%.  Somehow users don’t think that is mail restored.

And now Yahoo Mail users and Yahoo Customer support is in support hell.

Update 12/18/13 12:30 pm PST

Here’s the latest update from us answering some of your questions:


Q:  I’m on hold for a while when I call Customer Care.  What’s happening?
We’ve heard that some users are experiencing longer wait times than usual. We appreciate your patience while we work through a large volume of calls.  We are adding agents quickly to support this large volume of calls. Alternatively, you can click the link to the right here that says “Contact Customer Care.”  We’ll ask you to provide us with a few more details and then will follow up with you.  


Thank you for your patience.

Q:  I still can't access my account, what can I do?
We believe we've restored access for all users related to the outage. If you're having trouble accessing your account, please reach out to customer service so that we can provide you with 1:1 support.

Previous updates

+ Update 12/17/13 2:45 pm PST

We continue to work on recovering email messages, folders and inboxes for users who are still not seeing some messages in their inbox. In the last 24 hours, we've seen an accelerated rate of message recovery for affected users. Additionally, we are reaching out directly to the impacted users with an update specifically related to their accounts. 
 
We believe that we have restored access for all affected users, but if you are still having trouble accessing your account for any reason, please contact Customer Care at 1-800-318-0612.

 

AWS coming to China in 2014

AWS has a press release detailing the arrival of AWS to China.

PRESS RELEASE

Dec. 18, 2013, 3:22 a.m. EST

Amazon Web Services Announces Upcoming China Region for its Cloud Computing Platform

AWS signs memorandums of understanding with the Beijing and Ningxia governments to develop cloud computing services

 

 

 

 

 

SEATTLE, Dec 18, 2013 (BUSINESS WIRE) -- --Xiaomi, Qihoo 360, TCL, Tiens, NQ Mobile, FunPlus, Kingsoft, Mobotap, and Papaya Mobile among the growing Chinese business community already using AWS

AMZN +1.03% -- Amazon Web Services, Inc. (AWS, Inc.), an Amazon.com company, today announced the upcoming limited preview of its China Region for the AWS cloud computing platform. This will be the fourth AWS Region in Asia Pacific and the tenth Region globally. In the limited preview, which will be deployed in early 2014, a select group of China-based and multinational companies with customers in China will be invited to begin using AWS Cloud services in the AWS China Region to build their businesses and run their applications in the cloud. Businesses and software developers can apply for access to the limited preview in the AWS China Region today athttp://www.amazonaws.cn .

One nugget in the AWS blog is the business is a bit different in China than other sites.

Our business model will be slightly different here than in the other AWS Regions. You will need to create an AWS account that is specific to the Region. We will be posting additional information on AWS China website as it becomes available.

Here is the news China AWS site.  Below is the site with Google Translate to english.

NewImage