Amazon's outages creates opportunity for Cloud Services that are transparent

Greenpeace tried to get information about Amazon Web Services cloud environment to determine an environmental impact.  Greenpeace scored Amazon an F.

Geekwire has an written by a Seattle SW developer Keith Smith discussing the communication problem from Amazon and lack of transparency.

Today that lack of transparency has continued.

As problems continued throughout the day, we experienced the obvious frustration from the system failure. But Amazon’s communication failure was even more alarming.

7:30a on Apr 22.  Keith Smith's company Big Door is still down.

Isn't a bit ironic that the cloud is built on open source software, yet the services delivered look no different than proprietary closed systems when you dig for answers to tough questions like an environmental impact and what caused an outage?

Is cloud transparency a future feature differentiator?  Right now all the clouds are pretty obscure.

oops is Amazon Web Services running too lean? Capacity issues are slowing down AWS recovery

News is spreading on AWS outage.

Major Amazon Outage Ripples Across Web

Amazon cloud outage derails Reddit, Quora

You can see the status on AWS http://status.aws.amazon.com/

Problems happen in every data center and systems are designed to recover.  But, it looks like AWS had a capacity shortage that is causing the problems to be slow in remedying.

A networking event early this morning triggered a large amount of re-mirroring of EBS volumes in US-EAST-1. This re-mirroring created a shortage of capacity in one of the US-EAST-1 Availability Zones, which impacted new EBS volume creation as well as the pace with which we could re-mirror and recover affected EBS volumes. Additionally, one of our internal control planes for EBS has become inundated such that it's difficult to create new EBS volumes and EBS backed instances. We are working as quickly as possible to add capacity to that one Availability Zone to speed up the re-mirroring, and working to restore the control plane issue. We're starting to see progress on these efforts, but are not there yet. We will continue to provide updates when we have them.

The thread started at 1:41a PT.

1:41 AM PDT We are currently investigating latency and error rates with EBS volumes and connectivity issues reaching EC2 instances in the US-EAST-1 region.

2:18 AM PDT We can confirm connectivity errors impacting EC2 instances and increased latencies impacting EBS volumes in multiple availability zones in the US-EAST-1 region. Increased error rates are affecting EBS CreateVolume API calls. We continue to work towards resolution.

2:49 AM PDT We are continuing to see connectivity errors impacting EC2 instances, increased latencies impacting EBS volumes in multiple availability zones in the US-EAST-1 region, and increased error rates affecting EBS CreateVolume API calls. We are also experiencing delayed launches for EBS backed EC2 instances in affected availability zones in the US-EAST-1 region. We continue to work towards resolution.

3:20 AM PDT Delayed EC2 instance launches and EBS API error rates are recovering. We're continuing to work towards full resolution.

...

Amazon Announces Cloud Drive, that explains Amazon’s data center build out in Oregon

Amazon.com, Inc. (NASDAQ:AMZN) just announced Cloud Drive along with Cloud Player.

Introducing Amazon Cloud Drive, Amazon Cloud Player for Web, and Amazon Cloud Player for Android

Buy anywhere, play anywhere and keep all your music in one place
Start with 5 GB of free Cloud Drive storage - upgrade to 20 GB free with purchase of any MP3 album

SEATTLE, Mar 29, 2011 (BUSINESS WIRE) --

Amazon.com, Inc. (NASDAQ:AMZN) today announced the launch of Amazon Cloud Drive (www.amazon.com/clouddrive), Amazon Cloud Player for Web (www.amazon.com/cloudplayer) and Amazon Cloud Player for Android (www.amazon.com/cloudplayerandroid). Together, these services enable customers to securely store music in the cloudand play it on any Android phone, Android tablet, Mac or PC, wherever they are. Customers can easily upload their music library to Amazon Cloud Drive and can save any new Amazon MP3 purchases directly to their Amazon Cloud Drive for free.

"We're excited to take this leap forward in the digital experience," said Bill Carr, vice president of Movies and Music at Amazon. "The launch of Cloud Drive, Cloud Player for Web and Cloud Player for Android eliminates the need for constant software updates as well as the use of thumb drives and cables to move and manage music."

Here is the site for Cloud Drive.

After 5 GB here is the pricing.

The music battle between Apple iTunes and Amazon MP3 Store are turned up a notch.

Here are some technical details on what is behind cloud drive.

A Drive in the Cloud

To build Amazon Cloud Drive the team made use of a number of cloud computing services offered by Amazon Web Services. The scalability, reliability and durability requirements for Cloud Drive are very high which is why they decided to make use of the Amazon Simple Storage Service (S3) as the core component of their service. Amazon S3 is used by enterprises of all sizes and is designed to handle scaling extremely well; it stores hundreds of billions of objects and easily performs several hundreds of thousands of storage transaction a second.

Amazon S3 uses advanced techniques to provide very high durability and reliability; for example it is designed to provide 99.999999999% durability of objects over a given year. Such a high durability level means that if you store 10,000 objects with Amazon S3, you can on average expect to incur a loss of a single object once every 10,000,000 years. Amazon S3 redundantly stores your objects on multiple devices across multiple facilities in an Amazon S3 Region. The service is designed to sustain concurrent device failures by quickly detecting and repairing any lost redundancy, for example there may be a concurrent loss of data in two facilities without the customer ever noticing.

Cloud Drive also makes extensive use of AWS Identity and Access Management (IAM) to help ensure that objects owned by a customer can only be accessed by that customer. IAM is designed to meet the strict security requirements of enterprises and government agencies using cloud services and allows Amazon Cloud Drive to manage access to objects at a very fine grained level.

A key part of the Cloud Drive architecture is a Metadata Service that allows customers to quickly search and organize their digital collections within Cloud Drive. The Cloud Player Web Applications and Cloud Player for Android make extensive use of this Metadata service to ensure a fast and smooth customer experience.

DataCenterKnowledge just posted on expanded data center growth by Amazon in Oregon, speculating on AWS growth, but the big growth is Amazon Cloud Drive & Player which is specifically in Amazon S3.

Amazon’s Cloud Goes Modular in Oregon

March 28th, 2011 : Rich Miller

The data center arm of Amazon.com is building data centers at three sites in Oregon, according to local media, who report that two of the sites are using a modular design. A third site, which has been the focus of on-and-off construction activity for several years, appears to be employing a more traditional design.

With the new projects, Amazon.com joins major cloud builders Google, Microsoft and Yahoo in embracing factory-built components as a strategy to reduce the cost and deployment time for data center capacity. The Oregon construction is part of a larger effort by Amazon to prepare for a significant expansion of its data center capacity to accommodate the growth of its cloud computing business, Amazon Web Services (AWS). Amazon has also been acquiring property near Dublin, Ireland to expand the European data center hub for AWS.

Evidence shows AWS is not perfect, Reddit reports outage due to AWS

Amazon Web Services (AWS) is on fire being a leader in the industry and with companies like Netflix committing to AWS, they look like the perfect cloud computing environment to many.  For over a year I’ve heard of many though who are looking to move out of  AWS as they run into performance problems.

DataCenterKnowledge reports on the problems Reddit has had using AWS.

• Reddit Ties Outage to Amazon Performance

  March 18th, 2011 : Rich Miller

  UPDATE: Reddit has now updated its post from saying that it “been working to completely move Cassandra off EBS and onto local storage” to say that it is moving Cassandra “off of EBS and onto the local storage which is directly attached to the EC2 instances.” We have updated out post to reflect that Reddit has not reduced its use of AWS, but only the way it deploys resources on it.

  The social news siteReddit is revising how it uses Amazon’s cloud computing service following performance problems that contributed to six hours of downtime for the Reddit site this week. The Reddit operations team attributed the outages to problems with Postgres and Cassandra servers deployed on Elastic Block Storage (EBS), a service offered by Amazon Web Services. Reddit said EBS servers in a single U.S. availability zone for AWS experienced performance problems.

It will be interesting to watch as more stories become public of those who are moving out of AWS.  Where are companies moving to?  Many big players are going straight to wholesale space.  Some are going to Softlayer where they can get get dedicated hardware.  Keep in mind the hot start-ups like Reddit most of time have their code written to scale on multi-processor servers and utilize the hardware capabilities without virtualization.

If you don’t need virtualization why go to the cloud?

There is a good reason why Amazon doesn’t rent out dedicated hardware.  Do you know why?

423 job openings in AWS, how many employees does AWS have? 2,000

Seattletimes has an article about Amazon's job growth in the Seattle area.

Amazon.com on a hiring spree

Amazon.com has been expanding its office complex in South Lake Union and has about 1,900 openings in Seattle.

By Amy Martinez

Seattle Times business reporter

GREG GILBERT / THE SEATTLE TIMES

Amazon began moving into its new headquarters complex in South Lake Union last spring and now occupies seven buildings covering 845,000 square feet.

It's tough times all around, just not at Amazon's new headquarters in the South Lake Union area of Seattle.

Young workers dressed in jeans and T-shirts type away on laptops in small conference rooms named after products sold on Amazon's website. Vending machines serve up pricey gluten-free cookies and dry-roasted edamame. And for their pooches, doggy biscuits are handed out at the main reception desk.

Amid a sluggish job market and shaky economic recovery, the world's biggest Internet retailer is hiring like crazy at its headquarters complex. Consider Amazon's online jobs board: It lists about 1,900 openings in Seattle, at least twice as many as a year ago. More than 900 call for techies.

If Amazon is adding 900 technical jobs.  How many are in AWS?  I blogged about 166 job positions open in May 2010.

I was in Home Depot yesterday and saw one of my ex-Microsoft friends who joined Amazon Web Services 4 years ago when people weren't familiar with the term Cloud Computing.  We had a quick catch-up in between Grill Aisle and Lawn Mowers.  I told him how I blogged about AWS having 100 job openings. Curious I checked today and there were 147 US jobs and 19 Int'l jobs for AWS.

So, going up to Amazon career and searching for "AWS" there are 43 pages of 10 jobs a page.  The 43rd page has 3 entries.  42 X 10 + 3 = 423.

Current job postings for Amazon of 1,900 for 33,700 employees equals a 5.6% hiring rate.

Worldwide, Amazon had 33,700 employees at the end of 2010, about 9,400 more than at the end of the previous year.

For a fast growing business if you assume a 20% hiring percentage, multiply 423 X 5 is about 2,000 employees.  Take your own guess on the size of AWS higher or lower, AWS is on fire and hiring.

Don't forget versus all the rest of the cloud companies, Amazon has more customers and more data than anyone else. A huge investment in customer analytics systems.  Analyzing behavior gives Amazon an advantage where their size makes it difficult for competitors.