Announcing Log Patterns: Saving time and money for engineers

Logz.io was born out of a frustration with the realities of managing an open source monitoring and troubleshooting stack across distributed systems at cloud scale. Today, we introduce another tool to streamline log management: Log Patterns.

Our founding mission was motivated by our desire to make it easy for engineers to analyze machine data using the most powerful open source tools without the cumbersome and time-consuming tasks of maintaining, configuring, hosting and scaling these systems.

And so, we built an observability platform with investigative powers like those of popular open source tools Elasticsearch and Prometheus — but enhanced — that is able to work with modern technologies, integrate with DevOps tools and processes, and most importantly, and scale easily and automatically to monitor the growing amounts of data that need to be processed at any given time.

Even then, we could never have foreseen the massive amounts of machine data that engineers would be analyzing on a daily basis in modern environments. Our service scales like no other to handle that data, but engineers still face the growing problem of time and energy spent on performance diagnostics and root cause analysis (47% engineers say they spend more than 2 hours finding and troubleshooting each production issue they encounter!). The most advanced queries and dashboards in the world can’t help if you don’t know what you’re looking for. And as machine data volume and velocity has exploded, resulting in millions of logs to analyze from distributed sources, that’s often the case.

There must be a better way…

So having solved for one frustration, we decided it’s time to tackle another. Today we’re excited to announce Log Patterns, the newest addition in the suite of advanced analysis AIOps tools, to help engineers speed up and simplify troubleshooting. 

The reality is that the logs produced by all these systems are often verbose, repetitive, or unimportant for investigative work. Log Patterns looks forr just that – recurring patterns in your log data that may have otherwise gone unnoticed. Without needing a single query, it will analyze each and every incoming log message using AI and advanced clustering techniques, and then group them into more manageable patterns so engineers can more quickly cut through all the noise to identify the important or unusual events rather than be distracted by those that are recurring or redundant. 

Example:

Account 358 was created , waiting for kibana indexes to be created
Account 1265 was created , waiting for kibana indexes to be created
Account 871 was created , waiting for kibana indexes to be created
Account 1291 was created , waiting for kibana indexes to be created
Account 309 was created , waiting for kibana indexes to be created

Pattern:

Account Number was created , waiting for kibana indexes to be created

We started using Log Patterns began as an internal tool, but its effect on productivity was so pronounced that internally and like it so much that we decided to make it available to everyone. The patterns are displayed right within Kibana for the simplest possible experience, right alongside the interface we all use for querying log data. And of course, we wouldn’t be doing our job if we didn’t make sure that all of this happens automatically, in real-time, and regardless of the amount of data being shipped. 

Solving more than we expected: Time AND Money

With Log Patterns, engineers will save valuable time by filtering the millions of lines of data and analyzing only the specific patterns that need investigating. The ability to filter by rare and very specific patterns also helps engineers isolate errors (and threats) more easily, resolving production and/or security issues faster, thereby potentially saving sensitive customer data from breach. As for the patterns that are truly irrelevant to the issues they are troubleshooting, engineers can simply discard them. This last part is particularly interesting, because it naturally results in engineers saving money. By removing noisy and irrelevant logs from logging pipelines, teams no longer have to pay to store that data. 

We at Logz.io ha’ve already saved ourselves a ton of time and frustration troubleshooting with Log Patterns. Now we’re excited to see what it can do for your customers. And this is only the first step. Log Patterns is the foundation for which we’ll build more advanced features to help engineers accelerate time to resolution and continue to be more productive and effective.

Get a thorough walk-through of Log Patterns’ utility with this tutorial on troubleshooting with Log Patterns.