Facebook shares ideas on Optimizing the Data Center Operations

Facebook’s Delfina Elberly presented on Facebook’s Optimizing the Data Center.

Keynote: Optimizing Data Center Operations

It’s time to recognize that making data center operations more efficient greatly reduces OPEX, increases internal customer satisfaction, and increases employee satisfaction and retention. Facebook has developed a wide range of tools and systems to measure the status and reliability of data center hardware; automate repair processes; deploy staff more effectively; and perform a variety of other functions that help it run one of the most efficient server fleets in the world. In this talk, Facebook data center operations director Delfina Eberly will describe how Facebook addresses operations challenges, with a focus on best practices and systems that can help you unlock more efficiencies.

This slide has concepts shared.

Another concept shared is focusing on service.

 The majority of tools are developed in house, except asset tracking.

For those of you who are curious what Asset Tracking system Facebook uses Delfina didn’t say what the software is.  Should you buy the same software as Facebook uses for asset tracking.  Unfortunately, unless you are running 100,000 servers and making your own servers and servicing them yourselves like Facebook their software stack may not make sense.

I preface these points because don’t run out and grab a copy of Oracle Agile unless you are prepared to build you own servers with BOM discipline.  Google, Amazon, Facebook, and Dell need Agile, most of you don’t.

Delfina presented great concepts though that are good for optimizing a data center.  And I agree with the points Delfina made.  The hard part is executing with software.  I know how to do this, and how to build it given I have a client ask how to do product lifecycle management in the data center three years ago.  And, I’ve been crawling through a bunch of the issues to do this for years.  Disclosure: my brother works on the Oracle Agile product and it was Facebook employees who have said they use Agile, not my brother.

Delfina closed with a point that also describes the approach we use to transform operations.

Operations has a huge opportunity to be transformed.  Can you see the future?  “The Transformation of Operations” is coming.

Katie Fehrenbacher makes a 2nd trip to Apple's Maiden Data Center, will she make a 3rd trip next year?

GigaOm’s Katie Fehrenbacher has a post on her 2nd trip to Maiden, NC to see Apple’s data center.  If you repeat a task you get better with experience and familiarity.  Katie took a trip to NC in July 2012 and wrote a series of posts on her visit including visiting Apple’s data center.

The ultimate geek road trip: North Carolina’s mega data center cluster

by Katie Fehrenbacher

JUL. 8, 2012 - 8:00 PM PST

 

Nov 2013, Katie returned to Maiden, NC and writes a well researched post on Apple’s renewable energy installation of 50MW.  Read Katie’s post to get the full report.  I’ll highlight a few points that shows how well Katie researched the post.

Katie gives background on her method

Apple has long been reticent about speaking to the media about its operations, green or otherwise. But I’ve pieced together a much more detailed picture of its clean energy operations after talking to dozens of people, many of them over the years. And Last week I got a chance to see these fully operational facilities for myself.

I walked around these pioneering landscapes, took these exclusive photos, and pondered why Apple made this move and why it’s important. This is Apple’s story of clean power plans, told comprehensively for the first time.

Katie points out that it is not possible for Apple to directly use the power from the its renewable energy sources.

Apple’s second 20 MW solar panel farm, which is about 15 miles away from the data center near the town of Conover, North Carolina, is also up and running. All told, the three facilities are creating 50 MW of power, which is about 10 MW more than what Apple’s data center uses. Because of state laws, the energy is being pumped into the power grid, and Apple then uses the energy it needs from the grid. But this setup also means Apple doesn’t need large batteries, or other forms of energy storage, to keep the power going when the sun goes down and its solar panels stop producing electricity.

Here is a nice picture Katie took.  One of the comments made said Katie should have used Panorama mode to get a wider view.

Apple’s solar power farm stretches for 100 acres

One of the funnier comments I read was a person saying the use of sheep is “iSheep" to clear the grass growing around the solar panels

You can see in the above picture that the grass is neatly maintained. Apple manages the grass under the panels in a variety of ways, but one of those is a little more unusual. Apple works with a company that ropes in sheep that eat the grass on a portion of the solar farm; when the sheep finish grazing on one spot, they’re moved to the next.

It’s a more sustainable option than running gas-powered mowers across the farm, and also has the added benefit that sheep can get into smaller spaces and up close to the panels. Some companies use goats to eat grass on plots of land, but goats could chew on the farm’s wiring and solar panel parts.

Apple’s 2nd solar array is 15 miles from the data center and putting the power on the grind makes much more sense than trying to bring the power back to the data center.

Apple’s second solar farm about 15 miles from its data center in North Carolina

Since the second solar farm is a ways away from the data center, it’s also an example of why Apple’s business with the utility is important. The power goes into the power grid near the solar farm, and Apple can use the equivalent back at its data center.

And Katie closes identifying Apple’s leadership.

Change often times happens incrementally. From the outside that happened with clean power and Internet companies in North Carolina. But sometimes crucial change happens with a single brush stroke or a single outlier decision. That’s how I see Apple’s clean power facilities in North Carolina — right now, they stand alone.

Disclosure: Katie is a good friend and I work freelance for GigaOm Research.  And after a post like this, it reminds me part of what I enjoy discussing is who is doing some of the coolest stuff in environmental efforts.

Seems kind of obvious that within the next year or two Katie will return to Maiden, and who knows what she’ll be able to write about then.

Katie’s post got picked up by others.

Gigaom checks out Apple NC data center's renewable-energy infrastructure tuaw.com - ‎12 hours ago‎ Apple's huge Maiden, NC, data center has recently become something else -- a net power provider of clean energy to Duke Energy. Gigaom's Katie Fehrenbacher took a look at Apple's new power-production facilities, which generate a total of 50 MW ...   A Closer Look at Apple's North Carolina Renewable Energy Power Plant Mac Rumors - ‎20 hours ago‎ After building large solar power and biogas fuel cell generating facilities at its Maiden, North Carolina data center, Apple is one of the of the largest non-utility clean energy generation companies in the country. As of a report earlier this year, Apple uses 100% ...   Report profiles Apple's pioneering clean energy projects in North Carolina [photos] 9 to 5 Mac - ‎20 hours ago‎ The site decided to take a look into Apple's decision to take renewable energy into its own hands just as North Carolina utility Duke Energy is requesting that the state allow it to sell clean energy to large corporate customers. Google is one of the large ...   Apple's Green Energy Efforts in North Carolina Inspires Industry Shift Wall St. Cheat Sheet - ‎14 hours ago‎ Did Apple's (NASDAQ:AAPL) renewable energy initiatives in North Carolina boost the state's clean energy industry? As noted by GigaOM, North Carolina-based Duke Energy (NYSE:DUK) recently requested regulators for permission to begin selling clean ...

Intermittent Blogging for next month, Mid Nov to Mid Dec 2013

A few days ago I realized I am traveling for work or pleasure every week for the next 4 weeks.  Lots of good things going on, but this means my regular time to blog will be disrupted.  I don’t think of myself as traveling that much, but when I add it up at the end of the year I am traveling a trip a bit less than every other week.  The majority to the bay area which I do so often it doesn’t really feel like a plane ride.  

I have been thinking more about how the industry is going through changes and how I have been changing what I blog about.  The Green Data Center is more accepted.  Google, Apple, Facebook, and Microsoft all have renewable energy announcements on a regular basis.  There are many who don’t do this yet, and thanks to the environmental groups more pressure will build for others to jump on the environmental movement.

I have lots to write.  The problem is my schedule is a little crazy for the next month.

Thanks for following this blog.  

I’ll see a few of regular readers in my travels, and sharing ideas through this blog jump starts the conversations.

AWS throws more weight into SSD

Jan 2010 I speculated that SSD’s would be part of AWS.  I was off by when AWS would start using SSD.

When will solid state memory server be an option in AWS instances?

THURSDAY, JANUARY 7, 2010 AT 12:47AM

I was having another stimulating conversation in silicon valley last night, and one of the ideas that made sense is for solid state memory servers to be part of the cloud computing option.  It’s just a matter of time.  Amazon has their current instance offerings with a division of performance and memory.

Today AWS announced two more SSD powered services.

One, is a coming soon service I2.

Coming Soon - The I2 Instance Type - High I/O Performance Via SSD

Earlier today, Amazon.com CTO Werner Vogels announced the upcoming I2 instance type from the main stage of AWS re:Invent!

The I2 instances are optimized for high performance random I/O. They are a great fit for transactional systems and NoSQL databases like Cassandra and MongoDB.

The instances use 2.5 GHz intel Xeon E5-2670v2 processors with Turbo mode enabled. They also benefit EC2's new enhanced networking. You will see significantly higher performance (in terms of packets per second), much lower latency, and lower jitter when you launch these instances from within a Virtual Private Cloud (VPC).

We'll be releasing more information at launch time. Here are the basic specs to tide you over until then:

Instance Name	vCPU Count	RAM	Instance Storage (SSD)
i2.large	2	15 GiB	1 x 360 GB
i2.xlarge	4	30.5 GiB	1 x 720 GB
i2.2xlarge	8	61 GiB	2 x 720 GB
i2.4xlarge	16	122 GiB	4 x 720 GB
i2.8xlarge	32	244 GiB	8 x 720 GB

The i2.8xlarge instances will be able to deliver 350,000 random read IOPS and 320,000 random write IOPS. Numbers for the other instance types will be proportionally smaller, based on the number of SSD devices associated with the instance.

Stay tuned for more information about the I2 instances.

and EC2 instances with SSD.

A New Generation of EC2 Instances for Compute-Intensive Workloads

Many AWS customers run CPU-bound, compute-intensive workloads on Amazon EC2, often using parallel processing frameworks such as Hadoop to distribute work and collect results. This includes batch data processing, analytics, high-performance scientific computing, 3D rendering, engineering, and simulation.

To date these needs have been met by the existing members of our compute-optimized instance families -- the C1 and CC2 instance types. When compared to EC2's general purpose instance types, the instances in this family have a higher ratio of compute power to memory.

Hello C3
Today we are introducing the C3 family of droids instances. Compared to C1 instances, the C3 instances provide faster processors, approximately double the memory per vCPU and SSD-based instance storage.

As the newest member of our lineup of compute-optimized instances, the C3's were designed to deliver high performance at an economical price. The C3 instances feature per-core performance that bests that provided by any of the other EC2 instance types, at a price-performance ratio that will make them a great fit for many compute-intensive workloads.

Use the Cores
Each virtual core (vCPU) on a C3 instance type is a hardware Hyper-Thread on a 2.8 GHz Intel Xeon E5-2680v2 (Ivy Bridge) processor. There are five members of the C3 family:

Instance Name	vCPU Count	Total ECU	RAM	Local Storage	Hourly On-Demand
c3.large	2	7	3.75 GiB	2 x 16 GB SSD	$0.15
c3.xlarge	4	14	7 GiB	2 x 40 GB SSD	$0.30
c3.2xlarge	8	28	15 GiB	2 x 80 GB SSD	$0.60
c3.4xlarge	16	55	30 GiB	2 x 160 GB SSD	$1.20
c3.8xlarge	32	108	60 GiB	2 x 320 GB SSD	$2.40

Protocols
If you launch C3 instances inside of a Virtual Private Cloud and you use an HVM AMI with the proper driver installed, you will also get the benefit of EC2's new enhanced networking. You will see significantly higher performance (in terms of packets per second), much lower latency, and lower jitter.

Getting Technical
As you may have noticed, we are specifying the underlying processor type for new instance types. Armed with this information, you can choose to make use of specialized instructions or to tune your application to exploit other characteristics (e.g. cache behavior) of the actual processor. For example, the processor in the C3 instances supports Intel's AVX (Advanced Vector Extensions) for efficient processing of vector-oriented data in 256-bit chunks.

Some Numbers
In order to measure the real-world performance of the new C3 instances, we launched a 26,496 core cluster and evaluated it against the most recent Top500 scores. This cluster delivered an Rmax of 484.18 teraflops and would land at position 56 in the June 2013 list. Notably, this is over twice the performance of the last cluster that we submitted to Top500. We also built an 8,192 cluster, which delivered an Rmax of 163.9, putting it at position 210 on the Top500 list.

Launch One Now
The C3 instances are available today in the US East (Northern Virginia), US West (Oregon), EU (Ireland), Asia Pacific (Singapore), Asia Pacific (Tokyo), and Asia Pacific (Sydney) Regions. You can choose to launch C3 instances as On-Demand,Reserved Instances, or Spot Instances.

-- Jeff;

 

Having fun at 7x24 Exchange, Getting some of the Thought Leaders to Socialize more

Normally going to a conference is not considered fun.  It may be educational, but rarely fun.  Many of my data center friends, the thought leaders who speak at events or have built and run some of the biggest data centers, like to get together when they can.  It can be a challenge trying to organize when we can get together for an evening and break away from the conference activities.  Hanging in the bar just doesn’t work for discussing something worthwhile.

Many of us are planning on going to 7x24 Exchange, Nov 17-20, 2013 in San Antonio.

This time we’ll spend an extra day in Texas and head to Austin for beer, BBQ, listen to a band, and socialize at a leisure pace.  I’ve found I get many more insights exchanging ideas this way and many others do too which is why we get together.  Building a social event around another data center event has been working, and this is the first time we’ll spend an additional day in the area. 

Thanks to Kevin,Sonda, Kristen to help pull this off.  There is no way this social could be organized without their help.

And a big thanks to those attending our Austin Social. 

At this time, we are full and are not adding more people to the event.  I am not going to say where we are staying in Austin, where we are going to dinner, or what venue we are going for the band.  Thanks to Kevin who is a native Austin, he has taken care of us.

What I will l say is thanks to another well connected individual, those who golf are going to Fazio Foothills.

The Fazio Foothills course, established in 1986, ranks at the top of all experts’ lists. Golfweek Magazine rates the Foothills #4 among America’s 50 Best Resort Courses. The course is also named the Best Resort Course in Texas and has the distinction of consistently having several holes listed among the State’s Best or Most Beautiful Holes (holes #12, 4, 9, 16, 10, 18) recognized by the Dallas Morning News.

And here is a little secret is a common interest.  You hang out with people who help you get better.  I was referring to the data center conversations, not the golf game. :-)