At AWS re:Invent recently, we excitedly announced Logz.io Infrastructure Monitoring! This product is the third pillar of our cloud observability platform, together with Logz.io Log Management and Logz.io Cloud SIEM. This announcement comes at the end of a very busy 2019 during which we introduced a series of major enhancements to the platform, all aimed at making the lives of the engineers monitoring, troubleshooting and securing their apps a whole lot easier.  

The end of the year is a good opportunity as any to provide a recap of the key announcements we’ve made and get perspective on upcoming 2020 DevOps trends and coming features from Logz.io. But it’s also a great opportunity to express my personal appreciation and kudos to the dedicated engineering and product teams responsible for constantly and consistently delivering innovation. 

Let’s begin.

Cloud Observability and Infrastructure Monitoring (beta)

Modern engineering teams rely on three main types of signals for observability: logs, metrics and traces. While our users have always used our platform for analyzing their time-series metrics, we quickly understood that there were better open-source tools for the job. 

Understanding engineers preferred Grafana for visualizing their metrics, we began by developing a data source plugin for Grafana which allows users to analyze the metrics stored in Logz.io with their own Grafana instance. But this still wasn’t enough. When troubleshooting an issue, users still needed to jump from one tool to another. Correlating between metrics and logs this way is simply not efficient. 

Last week at AWS re:Invent we announced a more complete experience in a new monitoring product and new metrics UI. Users can now use Logz.io for analyzing logs and metrics using the best open source tools on the market.

Logz.io Infrastructure Monitoring offers easy setup with built-in integrations and canned dashboards and extended retention for historical time-series analysis. More importantly, Logz.io supports a seamless correlation between metrics and logs, all within the same platform.

But this is just the start. We are now working hard at adding Jaeger into our platform, an addition that will consolidate our Cloud Observability Platform. More news on that coming soon!

Cloud SIEM

The responsibility for securing modern IT environments is increasingly been cast on the very busy shoulders of DevOps engineers. The slow but constant rise of DevSecOps reflects this trend, but as the results of our 2018 DevOpsPulse survey show, these engineers feel they don’t have the right tools or know-how to properly secure their apps and the infrastructure they’re deployed on. Our work here in 2019 anticipates the importance of Cloud SIEM among 2020 DevOps trends.

Logs contain a wealth of information about what happened, when and why. They can be used for operational use cases but also for security analytics. A large number of our users were already using our Log Management product as the backbone of their SOC when a year ago, we announced our Cloud SIEM (Security Incident & Event Management) product, purpose-built for securing cloud workloads. 

During 2019, Logz.io Cloud SIEM was enhanced by a long list of new features: Reports, Threat Intelligence Feeds, Lookups, Drilldown, tighter integration with security systems (Suricata, Wazuh, Zeek, AWS GuardDuty) and new dashboards and rules for various security data sources. You can find out more about Logz.io Cloud SIEM here.

Log Patterns

Logz.io’s core product is Log Management. Providing the world’s most popular open-source log management platform — the ELK Stack, as a cloud service, Logz.io Log Management offers users a long list of advanced analytics and data management capabilities designed to help engineers be more productive. 

From the onset, we knew that modern logging pipelines involved extremely high volumes of logs. We wanted to help the engineers tasked with sifting through these huge datasets when troubleshooting critical issues in their environment. That’s why we built AI-powered tools that use various machine learning techniques to automatically surface critical logs that would otherwise have gone unnoticed, with tools such as Cognitive Insights and Application Insights.  And a few months ago, we announced the release of Log Patterns, our third AI-powered analysis tool!

Log Patterns crunches up millions of log messages into what are much smaller, manageable groups of logs. This provides engineers with the ability to quickly cut through the noise, identify unique or unusual events, as well as recurring and repetitive events. This helps to make troubleshooting a much faster and easier process, but also to optimize logging pipeline and cut costs. You can read more about Log Patterns here.

Doubling down on alerting

Engineers rely on alerts to get notified when something goes wrong in their environment, and be more proactive in how they manage their log data. This explains why our alerting feature is one of the most popular features in the platform. 

Over the course of 2019, we introduced a long list of enhancements to this feature, making it the leading alerting engine on top of the ELK Stack. We’ve improved the performance of the underlying engine to near-real-time alerting, added a new JSON output, applied improvements to the alert creation and editing experience, added support for lookups, and a lot more.  

This one of Logz.io’s key features and we are already in the process of making it even better. Stay tuned for some innovation in this field—an expected major 2020 DevOps trend—coming soon to a theater near you. 

Upgrade to Kibana 7

Both our Log Management and Cloud SIEM products are built on top of the ELK Stack. Our users know that they can enjoy all the new functionality added to the open-source stack without the hassle of maintaining it and upgrading it. 

Kibana 7 offers users improved query performance, a collapsed side navigation, a new time picker, new filtering, a better dashboarding experience, and overall — a much sleeker and fresher look and feel. 

More information on Kibana 7 and Logz.io is available here.  

Data management & optimization

A lot of pain in the world of log management stems from the ever-increasing amount of operational noise created by logs. This noise poses an analytics challenge — how does one sift through millions of log messages a day — but also a very real cost challenge. Data storage can cost organizations millions a year. 

Logz.io invests a lot of time and resources into helping our users overcome these two challenges. Insights™ was developed to reveal hidden issues hiding within the data and cut troubleshooting time. In addition, a series of cost optimization features, such as Data Optimizer™ and Log Patterns were developed to help build cost-efficient and optimized data pipelines. 

This year we introduced a new concept of On-Demand Logging. A new feature called Drop Filters allows users to decide what logs to actually ingest in the system and which to weed out. The logs that are not ingested are not billed for. More information on Drop Filters is available here.  

Enhancing user experience

We know that ease-of-use and performance is crucial for our users and helps make monitoring and troubleshooting a more efficient process. That’s why we put a focus in 2019 on enhancing the UI and UX of the product by introducing a number of new features and improvements. 

Our upgrade to newer versions of Elasticsearch and Kibana have helped improve the speed of querying and dashboarding. The onboarding experience has changed for new users, who can now go through live tutorials within the product as well as download demo datasets to help learn the ropes faster. A new self-payment system has been put in place to make the upgrade process painless. And the list goes on. To help you keep up to date with all these features, we’ve also added a What’s New section in the product. 

Wrapping up 2019 

It’s hard to recap an entire year of innovation. I did not mention all the other smaller features added, such as new integrations with various tools and platforms, the new reporting feature, AWS IAM authentication improvements, the drill-down feature, multi-tab support, and many many others. 

To stay up to date with the latest and greatest I can only encourage our users to use the What’s New feature within the platform and of course, follow this blog. 

2019 was a super-packed year for Logz.io. The industry as a whole, and the observability space specifically, has seen some dramatic changes. New technologies have emerged together with new challenges. As we approach 2020, you can rest assured Logz.io will be there, at the edge, to face these new challenges head-on. We look forward to summarizing our upcoming 2020 features in a similar article next year!