Twitter's Data Center Migration Story on its 5th Birthday

There is lots of news out there on Twitter's 5th Bday. Twitter launched a new website and video.

Twitter Launches New Website & Video To Celebrate Its 5th Birthday [VIDEO]

In celebration of its fifth birthday, Twitter has launched a new website and a new video featuring some of its most prominent users.

Twitter employees have started tweeting links to discover.twitter.com, a website that features 16 different users from a variety of backgrounds, including entertainment, political, business and even astronautics. The list includes tennis star Serena Williams, Virgin Group founder Richard Branson, rapper Snoop Dogg, U.S. Speaker of the House John Boehner and astronaut Paolo Nespoli, among other recognizable celebrities and institutions.

What will be missed by most is the data center story by the Twitter engineering team.

Twitter Engineering

MONDAY, MARCH 21, 2011

The Great Migration, the Winter of 2011

If you look back at the history of Twitter, our rate of growth has largely outpaced the capacity of our hardware, software, and the company itself. Indeed, in our first five years, Twitter's biggest challenge was coping with our unprecedented growth and sightings of the infamous Fail Whale.
These issues came to a head last June when Twitter experienced more than ten hours of downtime. However, unlike past instances of significant failure, we said at the time that that we had a long-term plan.

Here are a few nuggets that jump out.

Once all the data was in place we began serving live traffic from the second data center for end-to-end testing and to continue to shed load from our primary data center. Confident that our strategy for replicating Twitter was solid, we moved on to the final leg of the migration, building out and moving all of Twitter from the first and second data centers to the final nesting grounds. This essentially required us to move Twitter a second time. Yes, that’s right, we moved all of Twitter twice!

Twitter discusses three data centers.

First, our engineers extended many of Twitter’s core systems to replicate Tweets to multiple data centers. Simultaneously, our operations engineers divided into new teams and built new processes and software to allow us to qualify, burn-in, deploy, tear-down and monitor the thousands of servers, routers, and switches that are required to build out and operate Twitter. With hardware at a second data center in place, we moved some of our non-runtime systems there – giving us headroom to stay ahead of tweet growth. This second data center also served as a staging laboratory for our replication and migration strategies. Simultaneously, we prepped a third larger data center as our final nesting ground.

Another released pieced of information is Twitter's expected employee growth from 350 to 3,000 in 2 years.

Now headquartered in the South of Market, Twitter is eyeing a move to Brisbane. Twitter CFO Ali Rowghani informed city officials last week that if the proposed tax exemption were approved, the company would move into the vacant Furniture Mart on Market Street just east of Ninth Street. It projects growing its current workforce of 350 to 3,000 by 2013.