Why Twitter dedicated servers for Justin Bieber, 3% of the load

There are 247 news articles about Twitter having dedicated servers for Justin Bieber which is 3% of Twitter traffic.

image

CNET broke the news with citing a tweet.

Report: Justin Bieber is 3 percent of Twitter

by Chris Matyszczyk

If you believe that Twitter is full of inane, immature narcissism, here's one in your solar plexus.

For an allegation has reached my eyes and baffled them into blindness. The allegation is that, at any given moment, at any given movement of your lungs and toes, 3 percent of Twitter's infrastructure is dedicated solely to the one person who most defines our hopes and our times.

I am not speaking of Kim Kardashian, nor of Rep. Jack Kimble. I am speaking of the one person who can unite men and women, young and old, sane and slightly less so: Justin Bieber.

A fascinatingly hopeful post at Gizmodo offers that Dustin Curtis, a designer of some repute, was told by an employee at Twitter that 3 percent of the company's infrastructure is dedicated to the little man with the unreal voice and the even more unreal hair.

(Credit: Screenshot: Chris Matyszczyk/CNET)

Read more: http://news.cnet.com/8301-17852_3-20015781-71.html#ixzz0ywhpUh60

What I haven't seen anyone attempt to explain is why would there be Racks of Servers dedicated?

This is common sense with no information from Twitter folks.

When you want speed in your servers you want to keep your whole search index in RAM in a server. Google, eBay, and Amazon are good examples of companies who use this method in their search servers to support keyword lookup and result match-up.  Any time you go to disk or another server you will slow down the search results by magnitudes.

The costs to put your whole search index in RAM is too high, so break down the problem.  Analyzing the flow of tweets, Twitter figured out many people are focused on a few areas, and probably stick to that topic watching the tweets.

Second Life was started with the idea of virtual reality and avatars, but many people are too lazy to keep up their 3D appearance.  Tweets are much more efficient to support a virtual presence.

In the same way that Saudi Arabia wanted to get access to RIM's encrypted data the information and analytics available in Justin Bieber's servers can be useful.  And, Twitter is probably figuring out ways to monetize access to analytics.

Which brings up an interesting point is Twitter the next service which governments want access to crawl?  For many people, e-mail and blogging is in the past.  And, Twitter is their main method to communicate.

There was a rumor of part of the reason why Twitter chose Salt Lake City is to be near the NSA's new data center in Salt Lake City. 

Room to grow: a Twitter data center

Wednesday, July 21, 2010

Later this year, Twitter is moving our technical operations infrastructure into a new, custom-built data center in the Salt Lake City area. We're excited about the move for several reasons.
First, Twitter's user base has continued to grow steadily in 2010, with over 300,000 people a day signing up for new accounts on an average day. Keeping pace with these users and their Twitter activity presents some unique and complex engineering challenges (as John Adams, our lead engineer for application services, noted in a speech last month at the O'Reilly Velocity conference). Having dedicated data centers will give us more capacity to accommodate this growth in users and activity on Twitter.
Second, Twitter will have full control over network and systems configuration, with a much larger footprint in a building designed specifically around our unique power and cooling needs. Twitter will be able to define and manage to a finer grained SLA on the service as we are managing and monitoring at all layers. The data center will house a mixed-vendor environment for servers running open source OS and applications.
Importantly, having our own data center will give us the flexibility to more quickly make adjustments as our infrastructure needs change.
Finally, Twitter's custom data center is built for high availability and redundancy in our network and systems infrastructure. This first Twitter managed data center is being designed with a multi-homed network solution for greater reliability and capacity. We will continue to work with NTT America to operate our current footprint, and plan to bring additional Twitter managed data centers online over the next 24 months.

The probability of the NSA talking to Twitter at some point of time is almost certainty.

Is Twitter's dedicated servers a help for the NSA?  There are Twitter servers dedicated to President Obama.  And Oprah too. :-)  Are the number of dedicated servers the new status symbol and measure of popularity?  It used to be records sold and movie revenue.  Is it now the billions of tweets?

Read more

William Gibson signs his first Kindle (mine) at a Zero History book signing

A friend and I went to hear William Gibson read a chapter from his new book Zero History.

Zero History

While William was reading a chapter,  I decided to download the book to my Kindle 3.  After reading the chapter, William took 1/2 an hour to answer various questions.

One of the questions was on how William uses Twitter, and he told the story of "Detection of Novelty" and how he used to spend $200-$300 on foreign magazines and randomly flip through them to simply lte him be attracted to what catches his eye.  With Twitter he focuses on finding efficient novelty aggregators that gives him the exposure he used to get from his foreign magazines.

When watching the standing room only crowd holding their hardcopy books, I thought what does William think about book signing with people having Kindles?  I decided to ask William this question which turned out to be the last before book signing.

William responded with humor that he could use a carbide tip to etch his signature in a strategic place, and how some people have brought those silver pens to sign black things.  Then there were some fanatics who wanted William to sign their laptops and motherboards.  William went on a bit more and used my question to end the session and transition to book signing.

image

Luckily, I sat in the front next to the door where they were going to start the line for book signing.  My friend went 1st as he had a hardcopy book.  I was next with my Kindle 3.  I showed William my Kindle 3 screen and chapter 1 of Zero History, and asked him to sign the Kindle 3.

image

William Gibson said this is the first Kindle he has signed.

I asked the last question, on the first day of William Gibson's book signing tour for Zero History, and have the first Kindle he has signed.

image

Boy I am glad I sat in the front and was #2 and not in the line of a hundred people behind me.  ;-)

Is this a new trend to have book authors sign Kindles?  I don't know, but I have the first Kindle signed by William Gibson.

And, here is William's tweet.

Signed very first Kindle at Microsoft. Actually, *touched* very first Kindle. Appealing unit, IMO.36 minutes ago via Twitterrific

Read more

Google confirms Daniel Costello joins the company

DataCenterKnowledge was able to get the confirmation Daniel Costello joined Google.

Daniel Costello Moves from Microsoft to Google

September 7th, 2010 : Rich Miller

Daniel Costello discusses the use of PACs (pre-assembled components) in Microsoft data centers at The Uptime Symposium in May.

Google has confirmed that Daniel Costello, who has been a key member of data center design teams at Microsoft and Intel, has joined the company’s data center staff. “We’re excited to welcome Dan to our infrastructure team,” said a Google spokesperson. “He has a great track record working with Microsoft and Intel, and we’re happy to have him on board.”

Now that Rich has posted, watch the news spread to other media outlets.

Rich referenced my post as part of his.

Does Costello’s departure from Microsoft (first noted by Dave Ohara) tell us anything about what’s ahead for data center design at Google? Or is Google simply the next frontier for a man with Costello’s design experience? Time will tell.

Thanks Rich.

Read more

A Motivation for Greenpeace’s targeting Facebook – Money and Power!!!

It has been interesting watching Greenpeace target the high tech industry.  Greenpeace has gone after Dell, HP, and Apple for their products use of components that have an environmental impact.  Mike Manos and I talked years ago how it was only a matter of time when Greenpeace target data centers.

Why Data Centers?  Because, they are big expensive buildings run by rich companies (Google, Microsoft, Facebook, Twitter, and Financial Companies) that use huge amounts of power.  Greenpeace has a difficult time changing the behavior of electric utilities as they have protested nuclear power plants, hydroelectric dams, etc.

As an alternative Greenpeace goes after the biggest consumers of a product they consider bad for the environment.  One of Greenpeace’s recent targets is Palm Oil and Nestle.  Here is Greenpeace’s post on Nestle on Mar 23, 2010.

Nestlé doesn't deserve a break

On this page

Feature story - March 23, 2010

Nestlé has remained relatively silent since issuing its initial statement of contract cancellations with palm oil supplier Sinar Mas. This is despite the fact that it is still receiving a barrage of complaints and criticism via its Facebook page from people who recognize that Nestlé's concessions made last week are not enough to protect Indonesia's rainforests.

zoom

Nestlé is buying palm oil from companies who are destroying orang-utan habitat.

The contract cancellations do not go nearly far enough to protect Indonesian rainforests because Nestlé will still be using Sinar Mas palm oil, but just getting it from other suppliers.

In response to the ongoing criticism it has faced online Nestlé also began to circulate a Q & A on its palm oil use, which also included its commitment to using only "Certified Sustainable PAlm Oil" by 2015. Again, this is not a solution and it does go far enough. 2015 will be far too late for the already endangered orang-utan and Indonesia's rainforests - which are being deforested at the fastest rate of any major forested country in the world. Earning Indonesia an unfortunate place in The Guinness Book of World Records.

And here are reported results.

NaturalNews) Nestle recently announced that it would stop purchasing palm oil from the world's second largest producer out of concern over rainforest destruction, but Greenpeace alleges that the company has failed to keep this promise.


Following a Greenpeace report accusing Indonesian palm oil producer Sinar Mas of responsibility for widespread destruction of endangered rainforests and peatlands for palm oil plantations, Nestle announced in March that it would cease doing business with the company. In December, Nestle rival Unilever had cut ties with Sinar Mas over the same issue.


"We will continue to pressure our suppliers to eliminate any sources of palm oil which are related to rainforest destruction and to provide valid guarantees of traceability as quickly as possible," Nestle said.

NYTimes reports on Soros donating $100 million to Human Rights Watch

Soros to Donate $100 Million to Rights Group

By STEPHANIE STROM
Published: September 6, 2010

  • George Soros, the billionaire investor and philanthropist, plans to announce on Tuesday that he is giving $100 million to Human Rights Watch to expand the organization’s work globally.

As the NYTimes reports, Soros action drives other money to his causes.

Last year, in the depths of the recession, Mr. Soros gave the Robin Hood Foundation, a charity that fights poverty in New York, a $50 million contribution that helped it raise significantly more than that amount. He also gave every family with children on welfare in New York State $200 to buy school supplies, a grant worth $35 million that enabled the state to gain access to some $175 million in federal money for which it would not otherwise have qualified.

Now, go back to the Greenpeace post on Palm Oil I put above.

image

Note the “Support Us” in red above.

Greenpeace cannot survive without support for its causes. 

In Greenpeace’s core values.

  • In exposing threats to the environment and finding solutions we have no permanent allies or adversaries;
  • We ensure our financial independence from political or commercial interests;

Greenpeace needs power and money to be independent.

Going after Facebook’s coal data centers let’s Greenpeace raise money and show its power to create change.

Do you think Greenpeace is going to back down until they get Facebook to change its power composition proactively?

Oct 1, 2010 is the opening of “The Social Network” the Facebook movie.  Just wait until October for more pressure from Greenpeace.

Read more

American Express’s “Project Green” Data Center

Winston-Salem Journal reports on American Express’s latest Data Center in NC - “Project Green” where details are hard to discover about the data center.

American Express making plans to hide in plain sight

By Richard M. Barron

NEWS & RECORD

Published: September 7, 2010

GREENSBORO

American Express has shrouded its major data-center project in secrecy from the moment that economic developers said in May that Guilford County had landed the $600 million project.

The company has declined requests to comment about the center it plans to build on two sites near Interstate 40 at Rock Creek Dairy Road. Many people working with the company in business or government relationships are not returning calls or taking great pains not to let much information slip out.

The reason becomes clear when you get inside the philosophy of the data industry.

"Do not make public any information about your operations," according to a report by SANS Institute, a data-center security consultant, issued not long after the Sept. 11 attacks.

"This includes but is not limited to location, staff, design or security features, type of equipment, etc. The smallest pieces of information can be used to compromise security."

But, given the facility is $600 million and 510,000 square feet it is kind of ridiculous to think people won’t know where the site is.

For all its secrecy, however, American Express is building its data center in a remarkably public place: Rock Creek Center, the Triad's largest business park with 1,400 acres.

The park is just south of Interstate 40/85, one of the state's busiest sections of road. The data center will sit south of Franz Warner Parkway, which runs through the park, and just north of a housing development with 375 lots.

The company plans to hide in plain sight.

The site will by 75 % vacant.

After it builds its data center and two large power substations, American Express will leave 75 percent of its land essentially vacant.

As typical the cost of the site is 1% of the cost of the building.

American Express paid $450,000 for that site and $5.62 million for the 107.8 acres it bought in Rock Creek Center.

American Express has two parcels and one may be a back-up data center.

Earlier this month, the Greensboro City Council unanimously approved the annexation and zoning for a site on the north side of Interstate 85 that is expected to be used as a backup data center. The annexation will allow city water and sewer to be extended to the 145 acres. The zoning change will allow a business park to be built on agriculture land.

Read more