Finding the Bug in the Haystack with Machine Learning: Logz.io Exceptions in Kibana

Logz.io is releasing its AI-powered Exceptions, a revamped version of our Application Insights, fully embedded in your Kibana Discover experience, to boost your troubleshooting experience and help you find bugs in the log haystack.

Continuous Delivery Yields More Bugs In Production

How many of your production issues stem from bugs in code you deployed? Probably quite a bit. The introduction of agile software methodology and its release early, release often mentality has exacerbated the problem, with more frequent code releases, in earlier stages.

How do you hunt down these bugs in production?

How do you ensure that your deployed code hasn’t caused any issues?

How can you proactively address code quality in your production environment?

Exceptions are the crumb trail you’re probably after. Exceptions contain human-readable messages, parameter values, and detailed stack traces. So you open your Kibana (or another log management system) and start hunting for exceptions. That is, you start running free text searches on exception strings, or filter by ‘log level equals error’ or above.

But let’s face it: modern software is full of exceptions, and finding relevant exceptions in the haystack (or should I say log stack) is quite challenging. We’ve all been there: you start filtering by service name, timeframe, and all sorts of parameters to reduce to a manageable number of log lines, but still end up with way more data than you can digest.

The challenge is not just in sifting through large amounts of exceptions and errors, but also in identifying trends out of all that data. For example, if I roll out new code to production, how can I check that this hasn’t introduced bugs? Are there new exceptions? Are there spikes in certain exceptions? How many exceptions of that sort have I had and when? How does the exceptions correlate with other events in the system?

Enhancing Kibana with Machine Learning Insights

While humans can’t effectively digest such large amounts of data, machines can.

Logz.io Insights Engine runs machine learning analytics algorithms that analyze the exceptions in your log data and aggregate them by their signature. This means that instead of looking through over a million log lines in Kibana, Logz.io surfaces only a short list of common exceptions that you can start investigating.

After running our Insights Engine successfully with our customers, we’ve decided to make it an integral part of your Kibana experience. Now you can access Exceptions (formerly known as Application Insights) directly in Kibana Discover, alongside your Logs and Patterns tabs.

Your Kibana experience just got upgraded:

Did you notice something suspicious in your Kibana dashboard? Switch to Discover, with that same context of filters and time window. But instead of diving into the sea of logs, explore your Exceptions first. Check out the most frequent exceptions in that time frame, or the most recent ones.

Here, you can easily see which exception has been around for a while and is just noise, then filter it out within Kibana with a single click. New exceptions on the investigated context are flagged, so you can focus your efforts there.

For each exception you can see a sample log of that exception with the stack trace and additional metadata labeled on your logs. You can view the exception plotted on a graph over time, to help identify a change of trend or abnormal spikes. Once you hone in on a suspicious exception, you can filter for it with a single click, to narrow down your Discover context and see only logs of that kind.

Beyond the logs of the error itself, you’d probably like to understand what happened in the “vicinity” of that error. Use the powerful Kibana capabilities like filter by tags, free text search and select time range, and focus on the relevant context of the issue. You can toggle between Exceptions and Logs within that context inside Discover in order to correlate your exception with other events in that same context, such as new code pushed to your environment or an upgrade of a database.

Moving from Reactive to Proactive

Exceptions are useful for much more than reactive investigations of production issues. With Exceptions, developers and DevOps engineers can take a proactive approach to the health of the production environment.

Did you push new code to production? Checking Exceptions can tell you if this triggered any new exceptions or spiked existing ones. With Exceptions, developers can take full ownership of their code and releases all the way to production, and reduce the emergency calls they get to put out fires.

Looking for an effective way to ensure the quality of your code in production? Monitor Exceptions and identify exceptions that don’t make sense (after all, they’re called ‘exceptions’ for a reason). Even for the exceptions you can live with, you should still monitor for any changes in the normal state. Exceptions which you deem to be normal, can be filtered out of your view in a single click to reduce the “noise” in Kibana.

The Exceptions feature joins Logz.io’s machine learning capabilities of Log Patterns and Cognitive Insights. The Patterns feature uses machine learning to identify common patterns in your logs (regardless of whether erroneous or not), which helps analyzing your log data and filtering out noisy one. Cognitive Insights automatically cross-references your logs with crowdsourced data to uncover potential production issues and provide actionable references from the web.

You can also build Exceptions and Insights into your DevOps pipeline automation: use the API to consume Exceptions programmatically and send them to downstream systems, such as reporting or ticketing systems. This enables creating elaborate automated pipelines, with quality gates and issue tracking, to increase the overall quality of the software that reaches production.

Endnote

Kibana is a powerful tool for investigating your logs for issues. We’ve enhanced it with machine learning based insights, and baked that into Kibana to give you a better experience. Machine learning can help surface issues from the mounts of data. Logz.io Exceptions’ integral work within the Kibana context enables seamless toggle between insights, logs and visualizations, for a more efficient and timely investigation. Try Exceptions now, or reach out to us for more details. For more information, check our technical documentation for Exceptions.