Google's Gmail self heals from shooting itself in the foot with config updates, after 25 minutes systems restored, Outage ends

There is tons of press on Gmail outage.  I was on the phone during the time the outage occurred so gmail being down didn’t bother me, but it did bother many others.

Gmail goes down briefly and everybody flips out

Atlanta Journal Constitution - ‎5 hours ago‎

If you're watching this, congratulations! You've survived the Great Google Outage of Jan. 24, 2014. At about 2:15 p.m. eastern time Friday, Gmail users across the world began seeing Temporary Error (500) error message while trying to access their email ...

 

Google's Gmail outage leaves many in the dark [San Jose Mercury News :: ]

Businessweek - ‎14 hours ago‎

Jan. 24--MOUNTAIN VIEW -- An unexplained outage affected countless users of Google's (GOOG) popular Gmail service for more than an hour Friday, while also disrupting the Google Plus social network and some of the company's other Web services, ...

 

Google's reliability team was prepping for a reddit AMA when Gmail went down

Washington Post (blog) - ‎50 minutes ago‎

While most of Twitter panicked over (and Yahoo celebrated) a Gmail outage, Google's Site Reliability Engineering was preparing to do an "Ask Me Anything" reddit thread. Depending on how paranoid you are, that may seem either incredibly ironic or like ...

 

Google services go down as Reliability team takes questions on Reddit

Fox News - ‎10 hours ago‎

Many of Google's services hiccupped briefly on Friday, as an unexplained outage knocked offline such popular services as Gmail, Calendar, Talk, Docs, Drive and more. As of 3:23 p.m. EST, the service was back up and running smoothly, according to the ...

 

Here's what caused that massive Gmail outage

Washington Post (blog) - ‎50 minutes ago‎

The outage, Traynor continued, essentially fixed itself when the system responsible for the malfunction automatically generated the correct configuration and began propagating that throughout Google's live services. Google offered an apology for the mishap ...

Here is Blog Post from Google VP Engineering Ben Traynor.  The brief summary of the problem, and how it self repaired is here.

At 10:55 a.m. PST this morning, an internal system that generates configurations—essentially, information that tells other systems how to behave—encountered a software bug and generated an incorrect configuration. The incorrect configuration was sent to live services over the next 15 minutes, caused users’ requests for their data to be ignored, and those services, in turn, generated errors. Users began seeing these errors on affected services at 11:02 a.m., and at that time our internal monitoring alerted Google’s Site Reliability Team. Engineers were still debugging 12 minutes later when the same system, having automatically cleared the original error, generated a new correct configuration at 11:14 a.m. and began sending it; errors subsided rapidly starting at this time. By 11:30 a.m. the correct configuration was live everywhere and almost all users’ service was restored.

Naive users are comparing Yahoo’s email outage to Google’s gmail.  Did Yahoo self heal?  No.  

Google knows it can win the e-mail battle with better availability.  Things happen, but if you can quickly recover and find the cause the overall site reliability should improve.

With services once again working normally, our work is now focused on (a) removing the source of failure that caused today’s outage, and (b) speeding up recovery when a problem does occur. We'll be taking the following steps in the next few days:
1. Correcting the bug in the configuration generator to prevent recurrence, and auditing all other critical configuration generation systems to ensure they do not contain a similar bug.
2. Adding additional input validation checks for configurations, so that a bad configuration generated in the future will not result in service disruption.
3. Adding additional targeted monitoring to more quickly detect and diagnose the cause of service failure.

Google adds another 59MW of Wind Power in 2015 in Sweden to support future expansion in Hamina, Finland Data Center

Google announced a Swedish wind power project to support its Finland Data Center growth.

More Swedish wind power for Google’s Finnish data center

Google buys 10 years of renewable energy in deal with Nordic wind farm developer Eolus Vind AB, financing construction of four new wind farms.

Stockholm, January 22, 2014 - Google and Eolus Vind AB, a leading Nordic wind farm developer and operator, today announced that Google will buy the entire electricity output of 4 new wind farms, to be built in 4 municipalities in southern Sweden, for a period of ten years, starting early 2015.  The new power purchase agreement (PPA) is Google’s second such agreement in Sweden in less than 12 months, and sixth agreement globally to procure renewable energy for its data centers.

The deal will provide Google’s Hamina, Finland, data center with additional renewable energy as the facility expands in coming years. This cross-border arrangement is possible thanks to Europe’s increasingly integrated energy market - and in particular, Scandinavia’s Nord Pool market - which allows Google to buy renewable energy with Guarantee of Origin certification in Sweden, and consume an equivalent amount of power elsewhere in Europe.  

The press has picked this up as well. When you combine Google with Renewable Energy the press coverage is better than most.

Eolus to Establish Wind Farms That Will Supply Google's Finnish Data Center ...

Wall Street Journal - ‎6 hours ago‎

HASSLEHOLM, Sweden, Jan. 22, 2014 (GLOBE NEWSWIRE) -- Eolus Vind AB and Google have signed a ten year agreement in which Google will buy all electricity generated by 29 wind turbines that Eolus establishes in four wind farms in southern Sweden.

 

Google locks in four new Swedish wind farms for Finnish datacentre

ZDNet - ‎2 hours ago‎

The deal with Nordic operator Eolus is the second 'purchase power agreement' (PPA) Google has made in Sweden to indirectly supply power to its datacentre, more than 1,000km away in Hamina, Finland. The Finnish facility is currently undergoing a €450m ...

 

Google seals second Swedish wind farm deal

Financial Times - ‎7 hours ago‎

Google has made its third bet on green energy in the space of three months, with a deal to buy all the electricity generated by four Swedish wind farms for 10 years. The transaction is similar to one that the technology group sealed seven months ago with ...

 

Google to buy more wind power for Finnish data center

GigaOM - ‎46 minutes ago‎

The additional wind power comes on top of what Google announced last year, that it planned to buy wind power from one large (72 MW) wind farm in Northern Sweden, to be built and operated by Swedish wind farm developer O2 in 2015. The new wind ...

 

Google Logging In To Swedish Wind Farms

North American Windpower - ‎2 hours ago‎

Google has signed a 10-year agreement with Swedish developer Eolus Vind AB to buy all of the electricity generated by four wind farms under development in southern Sweden. Eolus says the projects will supply the internet company's Finish data center with ...

What Google brings to the Nest party, infrastructure that performs and scales

GigaOm’s Om Malik.  Or maybe it should be Om Malik on GigaOm posts on why Tony Fadell decided to team up with Google.

Nest’s CEO Tony Fadell explains why he teamed up with Google: it’s about infrastructure

by Om Malik

17 HOURS AGO

8 Comments

• A
• A

photo: (c) Pinar Ozger

SUMMARY:

In an interview, Nest’s CEO tells us why a union with Google is a good idea (hint: infrastructure and scale) and how he can now get back to designing killer product experiences.

Here are parts that emphasize the google infrastructure.

In an interview with Gigaom following the announcement of the deal, Fadell told us how he wants to refocus on designing product experiences instead of spending his time on scaling and infrastructure — something that’s long been at the core of massive Google. It’s certainly “not about laptops and phones,” joked Fadell; it’s more about a marriage of hardware, software and services, he explained. And the deal has been under discussion for a long time, he noted, Sergey Brin was the one who originally told Google Ventures to start a discussion with Nest.

...

Will Google’s server infrastructure come in handy in a few years? Absolutely. Will their ability to work with algorithms help us? Absolutely. There are a lot of short and long term benefits, but in reality they are going to help us get to new markets at a much faster rate — especially, overseas. I see there are a lot of customers in countries were we are not there yet.

How much does your Cloud cost? Google adds Billing API to to support its Cloud data

It is kind of stupid to wait for your monthly AWS bill to see what you are paying.  Checking your console is one option.  Another would be if there was an API to get the data in an CSV (Excel compatible file).

Google just announced its Billing API for its cloud.

Now Get Programmatic Access to your Billing Data With the New Billing API

Posted: Monday, December 23, 2013

Tools for monitoring, analyzing and optimizing cost have become an important part of managing cloud services. But these tools are difficult to build if the usage data is only in the Google Cloud Console. We are happy to announce a solution to this problem. The Billing Export feature addresses this need, and it is available in Preview.

Once enabled, your daily Google Cloud Platform usage and cost estimates will be exported automatically to a CSV or JSON file stored in a Google Cloud Storage bucket you specify. You can then access the data via theCloud Storage API, CLI tool or Cloud Console file browser. Usage data is labelled with project Number and resource type. You have full control of who can access this data via ACLs on your Cloud Storage bucket. 

For those of you on AWS you can go here to figure out the Amazon billing API.

Programmatic Billing Access

Topics

• When to Use Programmatic Billing Access
• Using Programmatic Billing Access

Programmatic Billing Access leverages existing Amazon Simple Storage Service (Amazon S3) APIs so you can build applications that reference your billing data from a CSV (comma-separated value) file stored in an Amazon S3 bucket.

Note

IAM users with access to the billing pages can set the Programmatic Billing preferences.

Here's how it works:

Programmatic Billing Access Process

1. Log in to the Billing Preferences page.

2. Enable CSV reporting of your billing statement.

3. Sign up for Programmatic Billing Access by providing a bucket location for the CSV files.

4. Set a policy on the bucket granting AWS access to publish your CSV files to the bucket at the specified location.

   Note

   The CSV files are stored in Amazon S3 at standard Amazon S3 pricing.

5. Use an application, such as Microsoft Excel, to parse the billing data. Or, use the existing Amazon S3 API to write an application that accesses your billing data.

   AWS provides SDKs for developing applications in specific languages. For links to the complete set of AWS SDKs, see Sample Code & Libraries.

The following diagram shows how Programmatic Billing Access works.

Here's a bit odd point, why is it inevitable Google build a data center in Hong Kong?

Many have blogged based on Google choosing to not build a data center Hong Kong.

Then I saw this post on why this event is odd.

Google sets aside Hong Kong data center plans. Here’s why that’s a bit odd

Google

An aisle of servers in a Google data center

The point made is that Google will be in every country, so they will return to Hong Kong.

And because high-speed service has been a priority for Google product managers over the years, it’s not hard to imagine that one day a Google data center could be located in every country, or every other country. Google’s got the money and the demand, so while a Hong Kong data center might be off the roadmap for the immediate future, it will probably open someday.

I guess you can imagine that there are users in every country in the world using Google.  Oh, that is pretty much true except where Russia, China, and other countries may make it difficult for Google to do business.  But, given Google has users in every country, does the company need to have data centers in every country?