sfrancis

ELK as a Service

Recommended Posts

One thing everybody is looking for is convergence, a single tool that does everything for observability. Monitoring, metrics, log analysis - LM does a good job on the first two, but I still need a separate tool to get useful metrics and trends out of my application logs. LM should look into adding ELK-as-a-Service to the LM feature stack (provide customers with an API endpoint they can feed logs to or something), and then customers could have service-level monitoring (URL response times, etc.), plus the traditional LM suite of monitors/metrics, plus LM Cloud, *plus* the most useful info of all: data mined from application logs. That's generally where the really good insights come from (and most of what's unique to each customer's business/offering).

 
ELK is well-known, open source, and fairly mature. Relatively easy to scale as well; should be easy for LM engineering to put together for a proof of concept anyway.
 
Meanwhile, I'm looking at things like Papertrail, Librato and Logz.io for my application logs - but I'd really like to have One Tool to Rule Them All.

Share this post


Link to post
Share on other sites

Hi sfrancis,

Thanks for the comment.  We've discussed log monitoring but not tackled it for a number of reasons, one of which being that the Splunk's, SumoLogic's, etc., of the world are very robust tools that do what they do really well especially when it comes to the terabytes worth of data that is often thrown at them.  Most companies have a solution like this in place already, so integrating with them more tightly is our preferred path at the moment.

Having said that, "push data" is something we've got on the docket for next year.  While that certainly will include technologies like StatsD, it's not to the exclusion of other formatted data (eg JSON payloads) where something like a ELK may be the right endpoint.  We already take in "events", so it's not too much of a stretch.  To be determined, and input welcome. 

Jeff

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.