Surveying the Tides of Open Source & Cloud-Native Observability

An Intellyx BrainBlog by Jason English, for Logz.io

We can plausibly say the enterprise development market turned the tide on cloud-native development in 2020, as most net-new software and serious overhaul projects started moving toward microservices architectures, with Kubernetes as the preferred platform. 

According to the 2020 DevOps Pulse survey of more than a thousand DevOps practitioners from mostly mid-to-large sized development and IT Ops shops, 56% of respondents store more than half their infrastructure in the cloud, while 42% store 75% or more.

Open source software is underpinning this sea change. Kubernetes itself floats upon open source contributions, while itself further promoting a wide community exploration of even more innovative cloud-native projects and open source tools for container delivery, orchestration, storage and security.

And on the crest of this tidal shift? Observability, which goes beyond monitoring to provide clarity into the inner workings of software.

With so much business riding on software, the global market demand for observability is exploding with multi-million-dollar venture investments and multi-billion-dollar vendor acquisitions (and valuations). While there are market leaders doing very well selling commercial observability software, that’s only a small fraction of the action in the cloud-native realm.

Instances of installed open source monitoring and cloud-native observability tools in production may outnumber the instances of all vendor solutions by 10 times or more. Why?

Out-of-the-Box Cloud-Native Observability Tooling

The ELK Stack for log management and reporting, Prometheus for monitoring, Grafana for metrics and dashboarding, and finally Jaeger for tracing, were all born in a microservices world. These projects are now maturing within a cloud-native ecosystem.

When companies embark on a Kubernetes project, open source tools are a default. Devs most often consider OSS as part of the out-of-the-box kit you get when going cloud-native. Open Source tools should be easy to download, free of license costs, and most importantly battle-tested and refined by their respective communities. 

But by no means does that mean companies are making sense of all the data pouring forth from open source instrumentation. The most common reported challenges to gaining observability into cloud-native environments cited in the study were Kubernetes-related architecture (59%) then followed by managing and scaling open source monitoring tools (40%).

Despite almost ubiquitous adoption of open source instrumentation software for logs, metrics and traces, there’s still plenty of room left for vendors to improve cloud-native observability development and operations.

Functionality gaps are a feature, not a bug, specified by the open source community and the CNCF. The Kubernetes platform intentionally ‘leaves the wires hanging’ for vendors to offer best-of-breed solutions. Those hanging threads invite contributors to guide innovation to meet whatever future market demands will exist for orchestration, security, storage, networking and of course, observability.

“Clearly we are hearing from teams that they want to use open source monitoring tools, because Kubernetes and microservices architectures are entirely open source,” said Jonah Kowall, CTO of Logz.io. “However, they are having trouble running Kubernetes, because they just don’t have visibility into what’s happening.”

What Data is Pushing this Wave?

In the DevOps Pulse survey, 44 percent of respondents cited difficulties monitoring and troubleshooting Kubernetes in production, making observability the leading cause of concern — ahead of management, security and all others.

Drilling down into the sources of data for observability, log analysis (90%) is still the most common method for gathering info. Next is infrastructure monitoring (78%) with APM (34%) and distributed tracing (26%) bringing up the rear.

While tracing seems a distant fourth, most planned to implement a tracing tool soon to better identify faulty services and interdependencies. Specifically, 44% planned on activating it within the year and 20% within two years.

Cloud-Native Development Trends

Increasing cloud-native development patterns will only generate further instances of open source telemetry tools. To meet the need, the cloud-native community and leading vendors are promulgating the OpenTelemetry project, to standardize data collection methods, making telemetry data portable and queryable, whether supporting a do-it-yourself open source stack or a leading full-service vendor platform. OpenTelemetry v.1.0.1 went live the first week of February 2021.

Following this trend, Logz.io is going all in on open source, providing a highly scalable multi-tenant SaaS ‘wrapper’ that encapsulates the functionality of an ELK stack. Kibana dashboards, metrics panels, and the Jaeger UI have combined within a panoramic portal of observability that maintains and correlates threads of a given developer’s work across all the tools.

Leading employment marketplace ZipRecruiter faced just such a challenge. Its own ELK configuration was slowing down and waning against heavy query traffic and high data ingestion rates. But that changed after using a multi-tenant instance of Logz.io’s Elasticsearch-as-a-service. ZipRecruiter’s SREs found that the same queries consistently performed 3X faster and scaled fluidly with spikes in demand.

Best of all, the skill investments engineers made in using open source telemetry tools could be immediately leveraged in a combined platform, while leaving laborious management and scaling tasks up to the vendor’s platform.

“We try to remain as true to open source as possible, making it easy for you to migrate to us, and easy for you to migrate away from us — if you decide that you want to take your telemetry data with you and run your own stack,” said Kowall. 

The Intellyx Take

Clearly the future of cloud native observability at enterprise scale is dovetails with open source tools providing a richer flow of telemetry data needed for observability.

Companies that view open source observability tools simply as a way to reduce the TCO of software licenses are overlooking hidden costs. You have to also account for complexities in managing and supporting these tools at scale. And, of course, you’ve got to keep an eye on ballooning labor and cloud infrastructure costs.

Most importantly, don’t miss the boat in the value of time and talent investments in open source observability options. They retain the benefit of being future-proof thanks to continued global community engagement and vendor innovation. 

A desire for open standards will allow cloud-native observability to keep improving the reliability and resiliency of our software, whether it runs on Kubernetes — or whatever new layer of abstraction drifts into view next.

© 2021, Intellyx, LLC. Intellyx retains editorial control over the content of this article. Intellyx advises companies on their digital transformation goals, and publishes the weekly BrainCandy and Cortex newsletters and their Cloud-Native Computing poster. At the time of publishing, Logz.io is an Intellyx customer. Image credits: NASA Goddard, Seaweed Farms; Tide gears, scattered1, flickr open source.