Where’s Open Source Observability Headed in 2022?

For the last five years, Logz.io has tracked and measured the pulse of DevOps, as well as adoption of key trends and technology, through our DevOps Pulse survey and report.  One of the obvious focus areas for us, as a company whose products are based on industry-leading open source, is the increased rise of incredibly useful open-source observability solutions, in general. 

This year’s survey, which featured a sample of over 1,000 organizations, encapsulates the viewpoints of those who are more apt to adopt open-source solutions.

There’s little debating – open source is critical for observability today. It’s not just the massive move that’s currently ongoing from proprietary data collection and agents to open source, driven primarily by OpenTelemetry. It is also the creation of standard ways to describe metrics, logs, and traces within hugely popular open source projects. 

Open-source efforts have evolved considerably in the past five years since we initiated this survey. These projects and technologies are much more accessible to engineers, enabling them to do so without having to engage with commercial solutions or pay high costs to get started. Yet, as these efforts scale across organizations, additional challenges have predictably begun to crop up. This is less often a major problem since the tools are still typically used in pockets, but as this gets strategic, it’s a bigger challenge for the involved organizations.

Large Enterprises Lead the Open Source Observability Move

The shift is also becoming increasingly visible today. Large banks and other conservative enterprise businesses are actively advancing open-source as a strategic initiative. They are doing this not only to save money on tooling, but also to future-proof their agents and data collection, especially since data collection may live inside the code (SDKs) and is not always extracted with agents.

In addition, as we enter an economic downturn, the high cost of commercial solutions will increasingly come under greater pressure from management. Frequently, open-source solutions can save over 50% on fully-loaded costs (such as engineers and infrastructure costs). Even if you opt for a fully-managed SaaS solution based on open source, 50% cost savings is a baseline. People also represent a big part of these open source costs. The talent shortage is real, and resources often need to be aligned to the most critical business problems. Self-managing and running open-source solutions, which are easily consumed as a service, does not drive the business forward.

These are all reasons why 90% of DevOps Pulse respondents are using some open-source solutions for observability, but, at the same time, only 10% of respondents are 100% open source. Removing proprietary technologies is becoming strategic for more organizations to future-proof their investments and avoid being controlled by other organizations. It also increases some risks, since open-source projects do shift over time, and getting support and security patches can be a concern. We had some perfect examples of this over the last 6 months with log4shell, spring4shell, and countless other major security holes discovered lingering in open-source libraries.

Why Are Enterprises Relying on Open Source Observability?

Open source was selected by the survey participants for many reasons, but the first three were ease of integration, community, and cost. Let’s look at each separately:

• Open source ensures integration with other technologies remains open and accessible to all users, avoiding lock-in.
• When projects are part of well-run software foundations, the community is strengthened,  projects are always open source, and they are marketed to the user base. This ensures they will be around for a long time and there will not be vendors taking over projects, an unfortunate trend in 2021 with Elasticsearch, Grafana, and more recently Cortex projects. 
• Finally, the cost discussion is somewhat misleading. Although open-source projects are no cost to download and use, they often require more care and feeding by engineering teams to keep them running, patched, and configured. This burden rests on the team, who instead could be building business critical applications.

Regarding vendor lock-in, in 2021 34% of respondents wanted to avoid vendor-lock in, versus 13% in 2020. This was likely driven by the actions of a few companies to re-license software and even close source software, as was undertaken by Elasticsearch, Kibana, and Grafana projects. Users feel it would be better not to be tied to a vendor, as we know we will have to change solutions every three-to-five years on average.

Running an observability team is hard work, and finding the expertise during this massive shortage of engineering talent is a big problem. Although as technologists we like to focus on technology, we often forget how integral people and processes are to success. I spend 20% of my time on recruiting and talent, specifically to ensure the teams are properly staffed and effective. This should be the bar for any manager. This was the 3rd-largest challenge in gaining observability at 36% of respondents.

Similarly, 36% of respondents also stated they were challenged by scaling and managing open-source monitoring tools. Although open source has come a long way, challenges remain in usability and scalability for these tools.

The Path Forward for Open Source Observability

Thankfully, due to cloud providers and specialized vendors, we have many alternatives to building scalable open-source solutions for ourselves. We can always outsource it to someone who does it better and keeps everything up to date for us, and many of these vendors build additional capabilities on top to make it even better for users.

I hope you review the webinar where I co-presented some insights and findings from this year’s survey. Have a look at our other coverage of the survey. Please follow us on Twitter and LinkedIn and look out for the forthcoming survey later in 2022!