Almost everyone puts their management system in the same area as their IT assets. When I worked on management system architecture I asked the question why don’t management systems get located offsite?  This was back in 2005 before the cloud was popular.  Recently, I’ve been asking about a Hadoop base approach to collect IT logs.

Wouldn’t it be cool if there was a cloud base server management system that used Hadoop to do things the big management tools can’t.  And cheaper with a pay as you go basis.

Loggly is a company that uses Hadoop to store log files.  See this job description.

Hadoop Engineer

Forge and weld a different kind of search engine. You are building part of the back end systems that accept data from our customers and push it through to our archiving, indexing and map/reduce framework, then make it available through search and large scale analytics systems. You’re helping form a core team who’s responsibilities are to make us bigger, better and faster. You know what to do, and don’t ask twice.

Here's what makes you tick:

• have constructed a distributed, elastic system before
• familiar with both solr and lucene, and realize those projects have in fact merged
• you conduct map reduce jobs on hadoop for breakfast, or for small afternoon snacks
• achieved authoring or implementing a high throughput distributed queuing system
• or have authored or implemented a high performance distributed data store
• understand that high reliability systems are expected to be highly reliable
• you’re that guy that comes in, in the middle of the night, and makes magic happen

What is Loggly?

Logging as a service — any time — your way — fast.

Loggly collects, indexes, and stores all log data and makes it accessible through search for analysis and reporting.

You can try Loggly today by signing up for the free product. With no up front investment necessary, you reduce your risk of locking into a software solution. Once you decide to purchase the Loggly service, we run your service at a fraction of the cost you would incur yourself. We manage the infrastructure for you. You don't need to do anything and have your logs at your fingertips at any time from anywhere — fast.

Running in AWS.

Loggly- United States (San Francisco, California)

Loggly is a cloud-based server logging service. Loggly provides a way to collect logs from servers in one centralized location and then quickly search them with an intuitive user interface.

Here is a comparison of Splunk vs. Loggly.

Update:Here’s how chief executive Kord Campbell described the difference between Splunk and Loggly:

We are a hosted solution compared to Splunk’s enterprise software download. Instead of installing your own server, downloading the code, and forwarding logs to that server, you just send them to our system. We run all the servers, storage, code, etc. for you, making life easier in the process. It’s a hell of a lot cheaper too.

We’re leveraging a bunch of Open Source technologies to leap ahead in the search portions of our offering, which makes us more nimble than Splunk. We’re focused on web app developers (like us) initially, providing development and monitoring features for them to maintain their code and systems. Later on we’ll branch out into security, compliance, and analytics.

When it comes to analytics, we’ll be able to use the search system we’ve built to pull data from a customer’s logs, then run a map reduce algorithm on them to crank out statistics on the data. For lots of data. Think of it as a flip side to Google Analytics. They take the log entries from browsers hitting your site – we take the entries from the hits to your server directly, through its logs.