Twitter's Data Center Migration Story on its 5th Birthday

There is lots of news out there on Twitter's 5th Bday. Twitter launched a new website and video.

Twitter Launches New Website & Video To Celebrate Its 5th Birthday [VIDEO]

In celebration of its fifth birthday, Twitter has launched a new website and a new video featuring some of its most prominent users.

Twitter employees have started tweeting links to discover.twitter.com, a website that features 16 different users from a variety of backgrounds, including entertainment, political, business and even astronautics. The list includes tennis star Serena Williams, Virgin Group founder Richard Branson, rapper Snoop Dogg, U.S. Speaker of the House John Boehner and astronaut Paolo Nespoli, among other recognizable celebrities and institutions.

What will be missed by most is the data center story by the Twitter engineering team.

MONDAY, MARCH 21, 2011

The Great Migration, the Winter of 2011

If you look back at the history of Twitter, our rate of growth has largely outpaced the capacity of our hardware, software, and the company itself. Indeed, in our first five years, Twitter's biggest challenge was coping with our unprecedented growth and sightings of the infamous Fail Whale.
These issues came to a head last June when Twitter experienced more than ten hours of downtime. However, unlike past instances of significant failure, we said at the time that that we had a long-term plan.

Here are a few nuggets that jump out.

Once all the data was in place we began serving live traffic from the second data center for end-to-end testing and to continue to shed load from our primary data center. Confident that our strategy for replicating Twitter was solid, we moved on to the final leg of the migration, building out and moving all of Twitter from the first and second data centers to the final nesting grounds. This essentially required us to move Twitter a second time. Yes, that’s right, we moved all of Twitter twice!

Twitter discusses three data centers.

First, our engineers extended many of Twitter’s core systems to replicate Tweets to multiple data centers. Simultaneously, our operations engineers divided into new teams and built new processes and software to allow us to qualify, burn-in, deploy, tear-down and monitor the thousands of servers, routers, and switches that are required to build out and operate Twitter. With hardware at a second data center in place, we moved some of our non-runtime systems there – giving us headroom to stay ahead of tweet growth. This second data center also served as a staging laboratory for our replication and migration strategies. Simultaneously, we prepped a third larger data center as our final nesting ground.

Another released pieced of information is Twitter's expected employee growth from 350 to 3,000 in 2 years.

Now headquartered in the South of Market, Twitter is eyeing a move to Brisbane. Twitter CFO Ali Rowghani informed city officials last week that if the proposed tax exemption were approved, the company would move into the vacant Furniture Mart on Market Street just east of Ninth Street. It projects growing its current workforce of 350 to 3,000 by 2013.

Web 2.0 vs. Enterprise, know your users and their religious orientation

It is interesting how fiefdoms develop in IT.  One of the latest divisions that occur is the enterprise IT group vs. the Web 2.0 online services group.  I've laughed many times watching data center executives walk into Web 2.0 companies and pitch their wares. 

One mistake made most often is using the same presentation they use for a corporate enterprise IT department as the Web 2.0  department.  See this for a web 2.0 department definition.  Think Twitter, Facebook, Zynga

The term Web 2.0 is associated with web applications that facilitate participatory information sharing, interoperability, user-centered design,^[1] and collaboration on the World Wide Web. A Web 2.0 site allows users to interact and collaborate with each other in a social media dialogue as creators (prosumers) of user-generated content in a virtual community, in contrast to websites where users (consumers) are limited to the passive viewing of content that was created for them. Examples of Web 2.0 include social networking sites, blogs, wikis, video sharing sites, hosted services, web applications, mashups and folksonomies.

For enterprise software, think of something you would sell to a government IT department on a department that has mainframes.

Why is this important because Web 2.0 people look down at the enterprise IT as people who are in the past, and they are better.  Telling a Web 2.0 department of your enterprise sales doesn't impress them, it can handicap you as a product not appropriate for Web 2.0.

Don't expect this to logical, it is kind of religious.

Evidence shows AWS is not perfect, Reddit reports outage due to AWS

Amazon Web Services (AWS) is on fire being a leader in the industry and with companies like Netflix committing to AWS, they look like the perfect cloud computing environment to many.  For over a year I’ve heard of many though who are looking to move out of  AWS as they run into performance problems.

DataCenterKnowledge reports on the problems Reddit has had using AWS.

• Reddit Ties Outage to Amazon Performance

  March 18th, 2011 : Rich Miller

  UPDATE: Reddit has now updated its post from saying that it “been working to completely move Cassandra off EBS and onto local storage” to say that it is moving Cassandra “off of EBS and onto the local storage which is directly attached to the EC2 instances.” We have updated out post to reflect that Reddit has not reduced its use of AWS, but only the way it deploys resources on it.

  The social news siteReddit is revising how it uses Amazon’s cloud computing service following performance problems that contributed to six hours of downtime for the Reddit site this week. The Reddit operations team attributed the outages to problems with Postgres and Cassandra servers deployed on Elastic Block Storage (EBS), a service offered by Amazon Web Services. Reddit said EBS servers in a single U.S. availability zone for AWS experienced performance problems.

It will be interesting to watch as more stories become public of those who are moving out of AWS.  Where are companies moving to?  Many big players are going straight to wholesale space.  Some are going to Softlayer where they can get get dedicated hardware.  Keep in mind the hot start-ups like Reddit most of time have their code written to scale on multi-processor servers and utilize the hardware capabilities without virtualization.

If you don’t need virtualization why go to the cloud?

There is a good reason why Amazon doesn’t rent out dedicated hardware.  Do you know why?

Verizon promotes itself vs. as a Cloud Computing alternative

Verizon's Jeff Deacon gets on a soapbox to promote Verizon as a top choice for cloud computing.

Connected Planet: What differentiates a telecom like Verizon from other cloud providers today?

Jeff Deacon: To start, the end-to-end SLAs on application performance are a key differentiator for telcos that can boast core assets such as hundreds of data centers around the world—necessary as a strong foundation for cloud service infrastructure. Also, owning the global IP networks means telecom operators can help enterprises migrate applications to the cloud with real SLA guarantees. Only a cloud provider that has control of the data center and everything in the data center (as well as the networks underlying those data centers and connecting the enterprises) can offer end-to-end SLA capabilities for mission-critical, heavy-duty applications critical to enterprise businesses.

Additionally, for decades, it is the telcos that do the metering and billing that now enable the usage-based capabilities necessary for measuring, charging and billing for what is actually used in a cloud environment. The back-office systems have to recognize the different types of cloud consumption and move the necessary information into billing and charging systems so customers know exactly what they used and for what they are being charged.

For example, today, we monitor usage on a daily basis when it comes to compute memory and storage, and in a couple of months, we will actually take that down to an hourly level so customers can see what they used on a more granular level.

Verizon closes with points it is a competitor of Google, Microsoft, and Amazon.

Rather than focus on one niche, we want to compete against the platform players like Microsoft and Google, as well as against the unmanaged infrastructure players like Amazon. And, we will layer platform applications on top of what we have so that enterprises have a full range of options, as we have a very diverse base of customers that have different needs at different phases of maturity.

A perspective from Tim Bray on Google Android, and tips for the blogger

Tim Bray made the switch to the Google Android team a year ago and writes two interesting posts.

One is his one year on the Google Android Team and share what is good and a problem.

What’s Good · Android, more than anything. A year spent in intimate contact with its coalface hasn’t shaken my feeling that most things about the system are mostly right. As I’ve already said in this space, I’ve never met a more accomplished engineering group; it’s a privilege to be associated with them. ¶

…

What’s a Problem · Being a remote worker, mostly. It’s just not a part of Google culture, and the Android project is particularly centralized. If you’re not spending a lot of time in that building with the dessert sculptures in front, it’s extra-difficult to be in the loops that matter; they’re not unwelcoming, they’re just super-busy. ¶

What was even more interesting was Tim’s “things about blogging.”  Here are three tips.  Visit his post to see the complete list.  Good stuff to think about.

In preparation for the event, I thought I’d jot down some helpful tips and tricks, and in no time at all I had more than twenty. I ran through them real fast in the hope of provoking some conversation — it worked — and got a laugh by saying “I guess I should write these up in a blog post”. Well, then.

For those who don’t know, I’ve been blogging since February 2003 and have written over a million words in this space, it’s been a boost for my career and my life, and I flatter myself that I’ve been involved in some conversations that mattered.

1. Blogging is Healthy · It’s no longer the white-hot center of controversy it was in 2005; now it’s part of the establishment, and if you look at the numbers from the popular platform providers like WordPress and Blogger, still growing quite nicely thank you. ¶
2. Freshness Matters · When you don’t update a blog, it gets stale fast. The natural tendency of the human mind to favor what’s fresh is reinforced by search engines leaning the same way. ¶
3. Write For Yourself · Don’t try to guess what people want to read; you’re the only person whose interests you really understand. In particular, don’t thrash around trying to appeal to a larger audience; the only surefire way is pictures of celebrity breasts, and the world already has enough. ¶