Why Zynga moved from AWS to its own private cloud - zCloud

Zynga has an engineering post where they introduce their private cloud.  This post is a bit old, but it provides good details on why Zynga chose to move out of AWS for its own private cloud.

Now, Zynga still does use AWS, but they are thinking from a business/financial perspective.  Zynga has a hybrid cloud infrastructure, both public and private.

While our private cloud infrastructure has been growing quickly, Zynga also uses the AWS public cloud to fuel our rapid growth. Our use of AWS, while very important to our business, comes with an operating expense. Essentially, we have been trading monthly operating expenses against longer-term amortized capital expense. Yet, sometimes the pace of our growth forces us to make that tradeoff.

For example, when CityVille rapidly grew to millions of users in just six weeks, we had to grow our server infrastructure at a pace that kept up with and sometimes even outpaced this demand. In a strict capital expense model, we would have exceeded the supply chain of our equipment suppliers – the process of physically getting the number of servers ordered, shipped, delivered and implemented just takes too long. So, we traded the cost of operating expense in AWS for capital expense.

Zynga's cloud is compatible with AWS.

The zCloud is our private cloud that looks, feels, and operates in a similar fashion to the way that we use Amazon Web Services (AWS) Elastic Compute Cloud (EC2), our public cloud provider. As infrastructure that is private to Zynga, the ZCloud physically resides in our current datacenters and will expand as we grow our infrastructure over time.

Zynga has learned from its AWS operations and builds its own cloud with its own hardware.

While similar in functionality to AWS, our private zCloud is designed specifically for social games in terms of availability, network connectivity, server processing power and storage throughput. We have achieved these improvements by providing redundant power to each rack, state-of-the-art servers with high memory capacity, a fully non-blocking network infrastructure, the use of inline hardware-based load balancers and local disk storage.

 

Zynga moves from AWS to its own Data Centers

Zynga is one of AWS largest tenants, supporting Zynga's rapid growth.  I discovered most of this information a year ago interviewing some people, and now that there is a public document, I can blog the following information.

VentureBeat's Dean Takahashi reports on the disclosure in the SEC statement.

Zynga planning to diversify beyond Amazon, build its own data centers

One of the little-known facts about social game giant Zynga is that it’s one of the biggest operators of cloud computing infrastructure, built to support its current customer base of more than 281 million monthly active users.

For much of its four-year history, Zynga has relied on a third-party hosting company, Amazon Web Services, for the hardware infrastructure for its server-based games such as FarmVille on Facebook. But to cut costs and diversify its risks, Zynga is now investing more money in building its own data centers, according to the company’s initial public offering filing with the Securities and Exchange Commission.

Zynga considers the investment in its own infrastructure to be important enough to warrant an investment of $100 – $150 million in the second half of 2011, according to the filing.

Where is Zynga moving to?  DCK reports on some of the sites.

Zynga currently leases data center space from two wholesale data center providers, DuPont Fabros Technology (DFT) and Digital Realty Trust (DLR). In the wholesale data center model, a tenant leases dedicated, fully-built data center space. Thisapproach offers greater control and security than shared colocation space, and is quicker and cheaper than building an entire data center facility. The tenant pays a significant premium over typical leases for office space, but is spared the capital investment to construct the data center.


Several of Zynga’s leased data centers are adjacent to Facebook data center facilities.

How fast can Zynga react in its new infrastructure?  How about 1,000 servers in a day.

Using Amazon EC2 and Leased Data Centers
Zynga has a strong cloud-based infrastructure that balances Amazon cloud instances with its own internal cloud infrastructure.  With the ability to add as many as 1,000 new servers to accomodate a surge in users in a 24 hour period (according to the S-1) a heavy hosting cost is associated with increased user demand.  By building more of its own infrastructure in company-owned data centers, Zynga might be able to reduce that cost.

Zynga has architected its solution for AWS.

Cadir Lee (CTO Zynga) quoted in a VentureBeat post:

It’s not the amount of hardware that matters. It’s the architecture of the application. You have to work at making your app architecture so that it takes advantage of Amazon. You have to have complete fluidity with the storage tier, the web tier. We are running our own data centers. We are looking more at doing our own data centers with more of a private cloud.

Netflix is infamous for being 100% in AWS, and Zynga is going in the opposite direction.

Zynga is going the opposite direction than Netflix. While Netflix is focusing (by using Amazon for most of their infrastructure), Zynga is diversifying (building their own data centers) .

And, what has Zynga learned running in AWS.  Note the yellow below "We have experienced, and may in the future experience, website disruptions, outages and other performance problems due to a variety of factors, including infrastructure changes, human or software errors and capacity constraints."

A significant majority of our game traffic is hosted by a single vendor and any failure or significant interruption in our network could impact our operations and harm our business. Our technology infrastructure is critical to the performance of our games and to player satisfaction. Our games run on a complex distributed system, or what is commonly known as cloud computing. We own, operate and maintain elements of this system, but significant elements of this system are operated by third parties that we do not control and which would require significant time to replace. We expect this dependence on third parties to continue. In particular, a significant majority of our game traffic is hosted by Amazon Web Services, or AWS, which service uses multiple locations. We have experienced, and may in the future experience, website disruptions, outages and other performance problems due to a variety of factors, including infrastructure changes, human or software errors and capacity constraints. For example, the operation of a few of our significant games, including FarmVille and CityVille, was interrupted for several hours in April 2011 due to a network outage. If a particular game is unavailable when players attempt to access it or navigation through a game is slower than they expect, players may stop playing the game and may be less likely to return to the game as often, if at all. A failure or significant interruption in our game service would harm our reputation and operations. We expect to continue to make significant investments to our technology infrastructure to maintain and improve all aspects of player experience and game performance. To the extent that our disaster recovery systems are not adequate, or we do not effectively address capacity constraints, upgrade our systems as needed and continually develop our technology and network architecture to accommodate increasing traffic, our business and operating results may suffer. We do not maintain insurance policies covering losses relating to our systems and we do not have business interruption insurance.

As Netflix Streaming declines, Netflix will save money with less AWS instances, but who cares?

Netflix will announce its Q3 results on Oct 24, and it is hard to say the numbers will look pretty.  Let alone what Q4 will look like.

Netflix to Announce Third-Quarter 2011 Financial Results

 

LOS GATOS, Calif., Oct. 4, 2011 /PRNewswire via COMTEX/ -- Netflix, Inc. NFLX -0.45% today announced it will post its third-quarter 2011 financial results and business outlook on its investor relations website at http://ir.netflix.com on Monday, October 24, 2011, at approximately 1:05 p.m. Pacific Time. At that time the company will issue a brief advisory release via newswire containing a link to the third-quarter 2011 financial results and letter to shareholders on its website.

Here is the one year stock performance.

NewImage

One good thing about Netflix using AWS for its streaming instead of data centers is they can scale to demand which usually implies scaling up.  But with Amazon, Apple, and Google all focusing on video it is hard to imagine Netflix's growth will continue at its past pace, and they may decline.  Which means Netflix can just turn off AWS instances and save money.  But, who cares?  Can you image Netflix in a keynote presentation,  we have reduced our streaming content by 15% over the last year and we have saved 15% in our data center costs by turning off 15% of our AWS instances.

My kids use Netflix streaming constantly, but I also have Amazon Prime and have 2 Amazon Kindle Fires on order.  We'll see how the kids like streaming to their Kindle Fire and whether Netflix shows up on the Kindle Fire.  The Amazon Prime users are just about as loyal if not more than the Netflix user base was.

Don't make the mistake Lotus did, betting against the adoption of new technology, Win3.1

A long time ago, almost 20 years ago, Microsoft was launching Win3.1 while the dominant installed base was MS-DOS.   Win3.0 had done OK, but much of the installed base of apps were DOS apps.  Microsoft was trying to get developers to write Windows apps.  One of those companies who dragged their heals to create Windows app was Lotus 123.  So, when Windows 3.1 shipped you could run Lotus 123 in DOS mode or you could run another spreadsheet not as popular, but had been ported from the Mac to Windows, called Microsoft Excel.  Microsoft Excel became the spreadsheet app to use on Win3.1.  And, Lotus 123 never recovered its #1 spreadsheet market on the PC.

Amazon has launched Kindle Fire and there are companies like Netflix that will drag their heals to port their applications to the Amazon Kindle Fire.  Just because an app is dominant on Android marketplace or Apple app store doesn't mean the Kindle Fire users will wait for the port to eventually show up.

Just like when the PC was having the battle between CP/M, DOS, and Windows operating system frustrating developers.  The battle is between Google, Apple, and Amazon.

We'll see who wins and loses if Amazon Kindle Fire is a winner.

Jeff Bezos shows a Data Center Image just like Steve Jobs did

Publishing data center pictures be on the web Is something that is forbidden.  Unless you are the CEO.

Jeff Bezos shows a picture of Amazon's Web Services data centers to support Amazon Silk.

NewImage

Is Jeff Bezos coping Steve Jobs when he discussed iCloud?  Check out my post on what Steve Jobs showed in his keynote.

"Full of stuff.  expensive stuff."  More laughs.  Who would ever call millions of dollars of IT equipment stuff?  You won't see Jobs calling an iPhone, iPod, or iPad stuff.  Do you think he is making fun of the other stuff he doesn't make?

image

image

image