Loggly, a Hadoop approach in the Cloud to manage servers

Almost everyone puts their management system in the same area as their IT assets. When I worked on management system architecture I asked the question why don’t management systems get located offsite?  This was back in 2005 before the cloud was popular.  Recently, I’ve been asking about a Hadoop base approach to collect IT logs.

Wouldn’t it be cool if there was a cloud base server management system that used Hadoop to do things the big management tools can’t.  And cheaper with a pay as you go basis.

Loggly is a company that uses Hadoop to store log files.  See this job description.

Hadoop Engineer

Forge and weld a different kind of search engine. You are building part of the back end systems that accept data from our customers and push it through to our archiving, indexing and map/reduce framework, then make it available through search and large scale analytics systems. You’re helping form a core team who’s responsibilities are to make us bigger, better and faster. You know what to do, and don’t ask twice.

Here's what makes you tick:

  • have constructed a distributed, elastic system before
  • familiar with both solr and lucene, and realize those projects have in fact merged
  • you conduct map reduce jobs on hadoop for breakfast, or for small afternoon snacks
  • achieved authoring or implementing a high throughput distributed queuing system
  • or have authored or implemented a high performance distributed data store
  • understand that high reliability systems are expected to be highly reliable
  • you’re that guy that comes in, in the middle of the night, and makes magic happen

What is Loggly?

Logging as a service — any time — your way — fast.

Loggly collects, indexes, and stores all log data and makes it accessible through search for analysis and reporting.

You can try Loggly today by signing up for the free product. With no up front investment necessary, you reduce your risk of locking into a software solution. Once you decide to purchase the Loggly service, we run your service at a fraction of the cost you would incur yourself. We manage the infrastructure for you. You don't need to do anything and have your logs at your fingertips at any time from anywhere — fast.

Running in AWS.

Loggly- United States (San Francisco, California)

Loggly is a cloud-based server logging service. Loggly provides a way to collect logs from servers in one centralized location and then quickly search them with an intuitive user interface.

Here is a comparison of Splunk vs. Loggly.

Update:Here’s how chief executive Kord Campbell described the difference between Splunk and Loggly:

We are a hosted solution compared to Splunk’s enterprise software download. Instead of installing your own server, downloading the code, and forwarding logs to that server, you just send them to our system. We run all the servers, storage, code, etc. for you, making life easier in the process. It’s a hell of a lot cheaper too.

We’re leveraging a bunch of Open Source technologies to leap ahead in the search portions of our offering, which makes us more nimble than Splunk. We’re focused on web app developers (like us) initially, providing development and monitoring features for them to maintain their code and systems. Later on we’ll branch out into security, compliance, and analytics.

When it comes to analytics, we’ll be able to use the search system we’ve built to pull data from a customer’s logs, then run a map reduce algorithm on them to crank out statistics on the data. For lots of data. Think of it as a flip side to Google Analytics. They take the log entries from browsers hitting your site – we take the entries from the hits to your server directly, through its logs.

Read more

48 hours at a Data Center Conference, Example Gartner DC LV 2010

I learn a lot going to data center conferences, but I go to almost no sessions.  My calendar is booked with numerous meetings, but I leave time to network and find new connections.

This is my 2nd Gartner DC Conference in LV and it was well worth my time, but I kind of have a unique way of leveraging the conference.

Given my blog I attend the conference as media with a press badge.  Weeks before attending I start to get e-mails and phone calls from public relations company to meet with vendors who have executives at the conference.  I choose carefully who I will set up an appointment with.  Some of the companies on the list were HP, Dell, SGI, Nimbula, Equinix, and APC.

The Gartner keynotes start on Monday morning at 8a.  I arrive in LV at about noon and start my meetings, and within 48 hours I am on a plane leaving LV.

While I am at the Gartner conference I don’t speak to a single Gartner employee.  And, I actually don’t listen to any of their presentations.  There are many who find the sessions educational, but I don’t learn enough new content to make it worth the time.  Spend 1/2 hour with a senior HP executive or 45 minutes listening to Gartner explain its surveys of a group who are not the innovators.  I spend so much of my time talking to innovators, listening to how the masses think is convenient for tracking the market, but not for being more competitive than the rest.

All my interviews are with what people see as what the future data center will look like which many time are greener data centers too.

As soon as I get close to the conference I start saying hi to people who I have seen at other conferences.  My first stop after registering is going to the press room.  The press room is one of the sparsest media rooms - no food, water, coffee.  Some mints.  Rich Miller and Kevin Normandeau from DataCenterKnowledge are the first media guys I see and we catch up a bit.  Two months ago, we were all at AFCOM LV, so it has only been a short period since we chatted. Also, I see Matt Stansberry from SearchDataCenter who I hadn’t seen for a while.

Matt and I discuss Oregon Duck Football as he lives in Eugene, OR and his wife is completing her post graduate degree at UO.  I tell Matt that I educate many people it is he who broke the story on Facebook’s choice for coal power.

Maybe Facebook should have bought a Bloom Box to diffuse Greenpeace’s campaign against a coal powered data center

Thanks to Matt Stansberry’s reporting on SearchDataCenter, attention was drawn to Facebook’s Prineville Data Center being coal powered.

Tiered energy rates bring higher prices for new customers
By 2012, BPA will charge tiered rates for power. Customers that signed 20-year contracts in 2008 will pay tier-one (i.e., inexpensive) pricing for their current electricity demand. These customers use most of the power produced by the dams.

By 2012, Oregon's Bonneville Power Administration will charge tiered rates for power.

To meet new customer demand or increased demand from existing customers, BPA also purchases power from other sources. In 2012 this electricity will be classified as tier two, and it will be charged at a much higher rate than the BPA's current hydropower.

Which brings us back to Facebook: The company's new data center is being built in Prineville, Ore., a small town on Oregon's high desert. Pacific Power, a utility owned by PacifiCorp, will provide the electricity. While Pacific Power gets some hydropower from BPA, its primary power-generation fuel is coal, according to Jason Carr, the manager of the Prineville office of economic development for Central Oregon.

With the price of hydropower increasing in the Northwest, Facebook opted to bet on the incremental price increases associated with coal rather than face tier-two pricing from BPA.

I’ll see Matt and Rich many times at the conference as we interview many times the same executives.  I’ll ask what they are finding interesting.

Besides interviewing there are attendees who come to do business and we’ll meet to discuss what is going on in the industry and where there are new opportunities.  Who is doing some of the best work and who is starting up new projects.  I’ll start looking for new connections and interesting people to have discussions with at the conference.

When the exhibit area opens, I’ll look for people I know, and watch which vendors are getting lots of traffic.  I rarely spend much time at an exhibit unless I know people at the company.  Most of the time I talk to a booth person, I find I can learn more by surfing their website.  One booth I went to was Splunk which is one of the fastest growing IT management tools.  I ended up spending over 1/2 hr at the Splunk booth as the guy I was talking to just happened to work for a good friend of mine at Microsoft and I knew there was a guy who had interesting insight.  Within 24 hrs, I was able to have a telephone conversation with Splunk’s CTO to discuss an innovative use of Splunk which I hope to write more about in the future as we prove a scenario to green the data center using Splunk.

Throughout the 48 hours I talk to business friends in site selection, engineering services, construction, facility operations, containers, server hardware, cloud SW, networking, and management tools.  Looking for how the pieces fit together in interesting ways.  Making introductions, and discussing new ideas.

There were a couple of good “ah ha” moments when I figured out some new things.  One example is the big whales aren’t at Gartner.

In the end I talked to some amazing data center executives, found some new technologies sooner as they were brought to my attention, reinforced established connections, made new connections, and had a good time discussing new ideas.

BTW, I don’t expect the Gartner folks to talk to me as I am not going to pay for their advice as I am not a client.  But I will help talk about what goes on Gartner DC LV.  Everybody gets a different experience than others.   The above is my 48 hrs at Gartner DC LV 2010.

Read more

A Logistics Lesson for Container Data Centers, US Navy’s F-35 Fighter engine too big to be shipped

Container data centers are hot topics and there are lots of new players in the game.  The military has used containers for a long time to ship supplies.  Here is a story about a goof with the F-35 fighter engine for the US Navy supply logistics.

Yet Still Another Embarrassing F-35 Problem

December 3, 2010: The U.S. Navy has yet another problem with the new F-35 fighter it will soon be operating off its carriers. It seems that no one bothered to check if the engine for the F-35C could fit into the C-2 aircraft the navy currently uses to deliver jet fighter engines to carriers. Normally, carriers go to sea with 30-35 spare engines for their F-18 fighters (that the F-35s will replace). In the course of a six month deployment, a dozen or more of these engines will be flown to, or from, the carrier.

The F-35 engine can be disassembled into five major components, and the largest of these can be carried by sling under an MH-53E helicopter or V-22 tilt-rotor aircraft. Both of these aircraft are normally carried by amphibious ships, along with a battalion of marines, and are usually near a carrier task force. But the range for the MH-53E (carrying the heaviest component) is only 550 kilometers, if the weather is good. The V-22 has had problems landing heavy sling loads on carriers, and more research is needed there. The heaviest component, including the shipping container, weighs 4.3 tons, and is too heavy to transfer at sea using the normal methods of underway replenishment (with the supply ship moving along side and using cables and hoses to move material and fuel.) This leaves delivering the engine via the supply ship. This requires very calm weather, and getting close enough to use cranes to haul the engine aboard the carrier. This can be tricky, even in good weather, on the high seas. All this is a big problem, as within eight years, F-35Cs will be operating off Nimitz class carriers, and getting fresh engines on, and broken ones off, will become a real issue. The navy will improvise some kind of solution, but this is not the first major hassle with F-35s operating on carriers.

If you thinking about containers for data centers, make sure you think of the lifecycle and logistics to support the maintenance and repair of containers.

Read more

Whale Hunting at Gartner DC LV 2010

I had dinner with a couple of senior data center executives who engineer and build datacenters for some of the top players in the industry.  These guys are part of the data center industry that build the big data centers for the top players where the business must have large capacity and the best designs.

Part of the conversation was comparing data center conferences to go to.  We were all at Gartner DC LV 2010.

image

Other conferences we agreed were good are DataCenterDynamics, 7X24 Exchange, and Uptime Symposium, SVLG DCEE.  All of these conferences have more of a data center facilities focus whereas Gartner has little facilities discussion. 

Gartner is different with a focus on data center operations from an IT perspective, not a facilities.

But, then I made the point that the big boys in data centers in general don't go to Gartner.  The exception is the executives who attend for presentations and meet with clients.  And, those who sell to the rest of the attendees

For example, Google sent two people who didn't look like they were from the data center group.  Microsoft sent over a dozen, but again not from the data center group.  No Facebook or Yahoo.  No Twitter, Zynga,   No AT&T. No Apple.  A couple from Verizon Wireless, but again no data center group.

Who does intend in mass with 5 or more people? 

Canada Dept of Defense

Delta Airlines

DePaul University

DirectTV

FAA

GSA

Kaiser Permante

Medtronic

McKesson

NASA

PG&E

Royal Bank of Canada

Sandia National Labs

Southern CA Edison

Social Security Adminstration

State Farm

US Dept of Defense

US Dept of Vet affairs

US Marine Corps

Lots of big fish.  But not the whales of data center.

So is Gartner DC LV conference really the data center industry?  Or those who look for Gartner for advice on data centers?

I run into many people I know in the industry at Gartner DC, but now that I think about it is the suppliers of the data center industry I run into at the event, not the end users I know who are the most innovative and biggest. 

The end users I run into at DataCenterDynamics, Uptime, and 7X24 Exchange.

If you are hunting for the big whales in the data center industry Gartner is not the place to look, but there are still plenty of big fish.  On the other hand, getting access to the right people is part of the challenge which is why the exhibit area is used so much.

Read more