Shifting mindset to an Information Factory from a Data Center, the industrialization of the data

There is a different way to think about data centers where the goal of a company is to bring raw unprocessed bits and turn them into higher value bits just like a factory brings in raw materials and transforms the materials into higher value finished goods. The factory uses huge amounts of power in special buildings with lots of equipment and custom processes to support the transformation.   This is the industrialization of the data center.

Barton George writes a post on Big Data any how in general 5% of data is only used.

Big Data is the new Cloud

Rate This

Big Data represents the next not-completely-understood got-to-have strategy.  This first dawned on me about a year ago and has continued to become clearer as the phenomenon has gained momentum.  Contributing to Big Data-mania is Hadoop, today’s weapon of choice in the taming and harnessing of  mountains of unstructured data, a project that has its own immense gravitational pull of celebrity.

So what

But what is the value of slogging through these mountains of data?  In a recent Forrester blog, Brian Hopkins lays it out very simply:

We estimate that firms effectively utilize less than 5% of available data. Why so little? The rest is simply too expensive to deal with. Big data is new because it lets firms affordably dip into that other 95%. If two companies use data with the same effectiveness but one can handle 15% of available data and one is stuck at 5%, who do you think will win?

But, do you think Google, Facebook, Amazon, Twitter, and Zynga use only 5% of the data.  These companies are analyzing all their users and information looking where to make more money.

The new way of thinking is all that data is both market intelligence and the raw materials for information factories.

Barton goes on to point out that Google Facebook, and Yahoo are big Hadoop type of users analyzing unstructured big data.

Deal with it

Hadoop, which I mentioned above, is your first line of offense when attacking big data.  Hadoop is an open source highly scalable compute and storage platform.  It can be used to collect, tidy up and store boatloads of structure and unstructured data.  In the case of enterprises it can be combined with a data warehouse and then linked to analytics (in the case web companies they forgo the warehouse).

And speaking of web companies Hopkins explains

Google, Yahoo, and Facebook used big data to deal with web scale search, content relevance, and social connections, and we see what happened to those markets. If you are not thinking about how to leverage big data to get the value from the other 95%, your competition is.

Some of you may think of this is new, but this is standard practice for many.  The winners think like an information factory integrating across many different systems.  The losers are thinking of a data center as a place their data is stored in silos to support internal organizational structures.

5 months since Mike Manos posts, Shares his Lights Out Data Center achievement

Mike Manos hasn't posted on his blog since May 20, 2011. Mike was nice enough to give me a heads up on what he posted today.

So, what has been Mike up to?  His post as usual is quite long, but having worked with Mike many times I am going to boil down this long post into something that is thought of as a blog post, less than 200 words.  :-)

Here is a graphic of what AOL has launched.

So what did Mike's team do?  They shut themselves in their own world, in a cocoon for months to come up with a better way to run AOL's infrastructure.

Luckily I have a world class team at AOL and together we built and entered our own cocoon and busily went to work. We have gone down the path of changing out technology platforms, operational processes, outdated ways of thinking about data centers, infrastructure, and overall approach. Every inch fighting forward on this idea of unified infrastructure.

And part of that time was identifying the cruft.

As I look at the challenges facing modern IT departments across the world, their ability to “go to the cloud” or make use of new approaches is also securely anchored behind by the “cruft” of their past. Sometimes that cruft is so thick that the organization cannot move forward.

And getting rid of the cruft is required for automation.

One of the key foundations for our ATC facility is our cloud platform and automation layer.

Mike shared some of his achievements at Uptime.

We went from provisioning servers in days, to getting base virtual machines up and running in under 8 seconds. Want Service and Application images (for established products)? Add another 8 seconds or so. Want to roll it into production globally (changing global DNS/Load balancing/Security changes)? Lets call that another minute to roll out. We used Open Source products and added our own development glue into our own systems to make all this happen. I am incredibly proud of my Cloud teams here at AOL, because what they have been able to do in such a relatively short period of time is to roll out a world class cloud and service provisioning system that can be applied to new efforts and platforms or our older products. Better yet, the provisioning systems were built to be universal so that if required we can do the same thing with stand-alone physical boxes or virtual machines. No difference. Same system. This technology platform was recently recognized by the Uptime Institute at its last Symposium in California.

And Mike gives credit to his team.

The culmination of all of this work is the result of some incredible teams devoted to the desire to affect change, a little dash of renegade engineering, a heaping helping of some new perspective, blood, sweat, tears and vision.   I am extremely proud of the teams here at AOL to deliver this ground-breaking achievement.   But then again, I am more than a bit biased.   I have seen the passion of these teams manifested in some incredible technology.

And what did Mike's team do?  They modularized the work.

This time frame was made possible by a standardized / modular way to build out our compute capacity in logical segments based upon the the infrastructure cloud type being deployed (low tier, mid-tier, etc.).   This approach has given us a predictability to speed of deployment and cost which in my opinion is unparalleled.

One of the things Mike can do that he couldn't do at Microsoft is use open source tools.

This time frame was made possible by a standardized / modular way to build out our compute capacity in logical segments based upon the the infrastructure cloud type being deployed (low tier, mid-tier, etc.).   This approach has given us a predictability to speed of deployment and cost which in my opinion is unparalleled.

...

We used Open Source products and added our own development glue into our own systems to make all  this happen.

Mike can now have really interesting discussions on the use of open source tools with the likes of Facebook, Twitter, Zynga, Mozilla, and others.

Steve Jobs Lesson to learn for Data Centers, Start with customer experience

The data center has some phenomenally smart mechanical and electrical engineers designing more efficient power and cooling systems.  DCIM is a hot topic for software to run data centers.  ZDNet has a blog post on the 100-year legacy of Steve Jobs.  One point made that is a quote from Steve Jobs is a lesson that is hard to learn.

“One of the things I’ve always found is that you’ve got to start with the customer experience and work backwards to the technology. You can’t start with the technology and try to figure out where you’re going to try to sell it. I’ve made this mistake probably more than anybody in this room and I’ve got the scar tissue to prove it, and I know that it’s the case. As we have tried to come up with a strategy and a vision for Apple, it started with ‘What incredible benefits can we give to the customer? Where can we take the customer?’ [It's] not starting with ‘Let’s sit down with the engineers and figure out what awesome technology we have and how are we going to market that?’”

Almost everyone does what Steve says not to do.

‘Let’s sit down with the engineers and figure out what awesome technology we have and how are we going to market that?’”

The nice thing is some of my entrepreneur friends are ex-Apple including myself we embrace this approach.

One of the things I’ve always found is that you’ve got to start with the customer experience and work backwards to the technology.

We've been sharing some of our product ideas with innovative data center operators, and we are pleasantly surprised on how much they like our approach to solving the customer problem.  We actually don't even really talk about the technology.  One of the guys who I shared the solution said you guys are using the "Challenger Sale" technique.

In The Challenger Sale, Matthew Dixon and Brent Adamson show how this critical finding has turned conventional wisdom on its head. While most companies focus on building customer relationships, the best focus on pushing customers’ thinking, introducing new solutions to their problems and illuminating problems customers overlook. That is, they challenge their customers.

Here is data that shows the challenger approach wins.

It's kind of logical to buy solutions from those who push your performance.  How many fanatical loyal Apple fans feel like they have products that push their experience of life.  Steve Jobs was a genius who learned some hard lessons that made him better in his 40s and 50s than in his 30s.

Zynga moves from AWS to its own Data Centers

Zynga is one of AWS largest tenants, supporting Zynga's rapid growth.  I discovered most of this information a year ago interviewing some people, and now that there is a public document, I can blog the following information.

VentureBeat's Dean Takahashi reports on the disclosure in the SEC statement.

Zynga planning to diversify beyond Amazon, build its own data centers

July 4, 2011 | Dean Takahashi

4 Comments

in Share

One of the little-known facts about social game giant Zynga is that it’s one of the biggest operators of cloud computing infrastructure, built to support its current customer base of more than 281 million monthly active users.

For much of its four-year history, Zynga has relied on a third-party hosting company, Amazon Web Services, for the hardware infrastructure for its server-based games such as FarmVille on Facebook. But to cut costs and diversify its risks, Zynga is now investing more money in building its own data centers, according to the company’s initial public offering filing with the Securities and Exchange Commission.

Zynga considers the investment in its own infrastructure to be important enough to warrant an investment of $100 – $150 million in the second half of 2011, according to the filing.

Where is Zynga moving to?  DCK reports on some of the sites.

Zynga currently leases data center space from two wholesale data center providers, DuPont Fabros Technology (DFT) and Digital Realty Trust (DLR). In the wholesale data center model, a tenant leases dedicated, fully-built data center space. Thisapproach offers greater control and security than shared colocation space, and is quicker and cheaper than building an entire data center facility. The tenant pays a significant premium over typical leases for office space, but is spared the capital investment to construct the data center.

Several of Zynga’s leased data centers are adjacent to Facebook data center facilities.

How fast can Zynga react in its new infrastructure?  How about 1,000 servers in a day.

Using Amazon EC2 and Leased Data Centers
Zynga has a strong cloud-based infrastructure that balances Amazon cloud instances with its own internal cloud infrastructure.  With the ability to add as many as 1,000 new servers to accomodate a surge in users in a 24 hour period (according to the S-1) a heavy hosting cost is associated with increased user demand.  By building more of its own infrastructure in company-owned data centers, Zynga might be able to reduce that cost.

Zynga has architected its solution for AWS.

Cadir Lee (CTO Zynga) quoted in a VentureBeat post:

It’s not the amount of hardware that matters. It’s the architecture of the application. You have to work at making your app architecture so that it takes advantage of Amazon. You have to have complete fluidity with the storage tier, the web tier. We are running our own data centers. We are looking more at doing our own data centers with more of a private cloud.

Netflix is infamous for being 100% in AWS, and Zynga is going in the opposite direction.

Zynga is going the opposite direction than Netflix. While Netflix is focusing (by using Amazon for most of their infrastructure), Zynga is diversifying (building their own data centers) .

And, what has Zynga learned running in AWS.  Note the yellow below "We have experienced, and may in the future experience, website disruptions, outages and other performance problems due to a variety of factors, including infrastructure changes, human or software errors and capacity constraints."

A significant majority of our game traffic is hosted by a single vendor and any failure or significant interruption in our network could impact our operations and harm our business. Our technology infrastructure is critical to the performance of our games and to player satisfaction. Our games run on a complex distributed system, or what is commonly known as cloud computing. We own, operate and maintain elements of this system, but significant elements of this system are operated by third parties that we do not control and which would require significant time to replace. We expect this dependence on third parties to continue. In particular, a significant majority of our game traffic is hosted by Amazon Web Services, or AWS, which service uses multiple locations. We have experienced, and may in the future experience, website disruptions, outages and other performance problems due to a variety of factors, including infrastructure changes, human or software errors and capacity constraints. For example, the operation of a few of our significant games, including FarmVille and CityVille, was interrupted for several hours in April 2011 due to a network outage. If a particular game is unavailable when players attempt to access it or navigation through a game is slower than they expect, players may stop playing the game and may be less likely to return to the game as often, if at all. A failure or significant interruption in our game service would harm our reputation and operations. We expect to continue to make significant investments to our technology infrastructure to maintain and improve all aspects of player experience and game performance. To the extent that our disaster recovery systems are not adequate, or we do not effectively address capacity constraints, upgrade our systems as needed and continually develop our technology and network architecture to accommodate increasing traffic, our business and operating results may suffer. We do not maintain insurance policies covering losses relating to our systems and we do not have business interruption insurance.

Who are Netflix's proponents after throwing supporters of divide the business under the bus?

Netfilx upset lots of users when they announced the splitting into Netflix and Qwikster.  There were a handful of analyst type of people who were saying that it was a good business decision.  So there were a large group of frustrated users, a small band of business people who said Netflix is doing the right thing and they should hold course.

Now that Netflix has thrown Qwikster under the bus, it also threw the handful of supporters who said dividing the business is the right thing.  How bad do you look saying Qwikster was the right thing, and 3 weeks later Netflix cancels Qwikster.

So, who is left to say Netflix is great company?

This situation will be a big lesson for companies who don't treat customers as their #1 priority.  It is clear to Netflix users that they are 2nd class citizens in Netflix's grand plans.

Also, an unintended consequence is the value of ex-Netflix employees on the job market has decreased.  This situation sure hasn't increased their value, even though there is an increase in quantity of Netflix employees thinking of exiting now that they have seen the stock drop from $300 to $110.  Morale at Netflix cannot be good.  I wonder if Amazon, Google, and a bunch of others have fired up the recruiters to hang around Netflix HQ.

The #1 resource of Netflix is great people.  The #1 proponents is loyal fans.  How many #1 things surround the Netflix brand?

The one thing that could change perception is throwing the CEO under the bus and take the blame.  How much do you think the Netflix stock would go up if the CEO quit?

Think about some of these things when running your data center.  Who is your #1 customer and how do they feel about their service?  Are they a proponent of your operations?  If not, if may explain why the business units are running to the cloud.