James Hamilton presents on AWS Data Centers - 2014 AWS Reinvent LV

At AWS Reinvent 2014 James Hamilton gave an update on the state of AWS data centers.

 

SPOT301 - AWS Innovation at Scale

 

This session, led by James Hamilton, VP & Distinguished Engineer, gives an insider view of some the innovations that help make the AWS cloud unique. He will show examples of AWS networking innovations from the interregional network backbone, through custom routers and networking protocol stack, all the way down to individual servers. He will show examples from AWS server hardware, storage, and power distribution and then, up the stack, in high scale streaming data processing. James will also dive into fundamental database work AWS is delivering to open up scaling and performance limits, reduce costs, and eliminate much of the administrative burden of managing databases. Join this session and walk away with a deeper understanding of the underlying innovations powering the cloud.

Let's start with Power.  AWS now has three data centers that are carbon neutral and AWS has negotiated their own power purchase agreements like Google.

Discussing the network infrastructure you can see the current state of AWS locations.

A data center can have between 50k - 80k servers.

Like Google and Facebook, AWS has its own server skus that they have had developed for their workloads.

Here is a video of James as well at AWS Reinvent.

Technology Companies with Seattle Offices

Microsoft and Amazon are well known for having their HQ in the Seattle area.  Geekwire has a post on the range of technology companies that have set up shop in seattle.

Home-grown tech giants such as Microsoft, Amazon.com, Expedia, T-Mobile, Zillow and Tableau have deep roots in Seattle — important companies which have helped define this region’s unique tech ecosystem.

The post does a good job of providing the range of other companies in the Seattle area.  Here are some that are the cloud companies.

Given that news and Apple’s fresh presence — along with recent arrivals of Alibaba, Oracle, HP, and, yes, even SpaceX — I thought the time was right to update the list.
...
CenturyLink: The telecommunications giant isn’t just about broadband Internet. The company is building a huge cloud engineering center in 30,000 square-feet of space in Bellevue at the One Twelfth @ Twelfth building, with plans to add 150 employees in the coming year. CenturyLink arrived in Seattle through the purchase last year of Seattle area cloud computing startup Tier 3, which makes up the backbone of the company’s big cloud push.“Seattle has become this pulse of cloud. It is like the heartbeat,” said former Tier 3 CEO Jared Wray in an interview with GeekWire last month as he showed off CenturyLink’s new cloud engineering center. Previously: Inside CenturyLink’s fast-growing dev center: Shaking up the cloud with innovative work spaces
...
Google: The granddaddy of the Silicon Valley titans in Seattle, the Internet search pioneer established a presence in Kirkland 10 years ago. It now employs more than 1,000 people in the area, split between offices in Kirkland, Bothell and Seattle’s Fremont neighborhood. Last year, the company doubled down on the Seattle area, announcing plans to add onto its Kirkland campus with a new 180,000 square foot development, one with enough room for an additional 700 employees. Previously: Google to double size of engineering center in Microsoft’s backyard
...
HP: In May, HP announced that its Helion cloud engineering effort will be based in Seattle, led by Microsoft veteran Bill Hilf.“We’re hiring like gangbusters,” said Hilf, speaking at the HP offices at Seventh Avenue and Pike Street in Seattle earlier this year. The company employed about 70 folks in the offices at the time, and said that it planned to add more than 200 in the next 18 months. Previously: HP hiring hundreds in Seattle for ‘Helion’ cloud launch, led by former Microsoft exec, betting $1B on OpenStack
...
Oracle: Yes, another long-time nemesis of Microsoft just arrived in Seattle. Oracle just inked a 17,000 square-foot lease in Seattle’s Century Square high-rise, with the new offices led by cloud engineering veterans and former Amazon.com employees Don Johnson and Craig Kelly. They are aiming to hire more than 100 engineers for the new office. Previously: Oracle to hire 100+ engineers in Seattle for new cloud infrastructure center

If any of you are looking for what kind of housing is available in the Seattle areas you can go to one of Seattle Native Tech companies Zillow.

LinkedIn Open Sources CPU Saving big data engine, Cubert

Saving power is something everyone wants to do, and thanks to open source advocates ideas can be spread by sharing.  Gigaom’s Jonathan Vanian posts on LinkedIn’s efforts called Cubert.

Linkedin said on Tuesday that it open sourced a framework called Cubert that uses specialized algorithms to organize data in a way that makes it easier to run queries without overburdening the system and wasting CPU resources.

Cubert, whose name is derived from the Rubik’s Cube, is supposedly as easy for engineers to work with as a Java application and it contains a “script-like user interface” from which engineers can use algorithms like MeshJoin and Cube on top of the organized data to save system resources when running queries.

Here is the LinkedIn Blog post.

 

About Cubert

Data scientists, analysts and engineers look for a computation platform that is designed for their real-life analytics needs, is fast even as the data scales, and is friendly in understanding and controlling the execution plans. 
 
We built Cubert to meet these requirements.


 

Cubert was built with the primary focus on better algorithms that can maximize map-side aggregations, minimize intermediate data, partition work in balanced chunks based on cost-functions, and ensure that the operators scan data that is resident in memory. Cubert has introduced a new paradigm of computation that:

  • organizes data in a format that is ideally suited for scalable execution of subsequent query processing operators
  • provides a suite of specialized operators (such as MeshJoin, Cube, Pivot) using algorithms that exploit the organization to provide significantly improved CPU and resource utilization

Cubert was shown to outperform other engines by a factor of 5-60X even when the data set sizes extend into 10s of TB and cannot fit into main memory.

Debating upgrading GreenM3 From Squarespace 5 to Squarespace 7

It’s been over 3 years since I switched from Typepad to Squarespace 5 for this blog.  Squarespace has upgraded from 5 to 6 and now 7.  There are some technical details on why I like squarespace 5 which don’t work on squarespace 7, but sometimes it is better change than hold on to hold habits.

Squarespace today announced its first major platform update in two years: Squarespace 7, adding new splash pages, templates and integrations with Getty Images and Google Apps.

The release advances the completely rebuilt codebase that arrived with the release of Squarespace 6 in 2012. Squarespace 7 will become available in waves to customers starting tonight. Customers can opt-in to the platform, or choose to hold off on upgrading during the early transition.

Will I change?  Most likely, but going to spend a few more days thinking about it.