The Rise of Open Standards in Observability: Highlights from KubeCon
TL;DR Key updates:
- Query Language Standard for Observability: new CNCF working group led by eBay, Netflix
- Standardizing the Prometheus Remote-Write Protocol
- Sunsetting OpenCensus in Favor of OpenTelemetry
- Elastic Common Schema merge with OpenTelemetry Logs Specification
- OpenTelemetry Logs Bridge Specification merged, progress to stabilize Logs API/SDK
Today’s IT systems are ever more fragmented. It is commonplace to see polyglot systems, written in multiple programming languages, and using a plethora of tools and cloud services as infrastructure building blocks, whether data stores, web proxy or other functions.
In this dynamic cloud-native realm, open standards and open specifications have become integral drivers of compatibility, collaboration, and convergence – the Three C’s of Open Standards, if you will. These principles enable developers and organizations to build upon existing solutions, fostering the growth of vibrant ecosystems, and without getting vendor-locked.
“Open standards and open specifications have become integral drivers of compatibility, collaboration, and convergence – the Three C’s of Open Standards”
Being at KubeCon, I witnessed the center stage that open standards and specifications have taken in advancing observability, within the Cloud Native Computing Foundation (CNCF) realm. In this post I will highlight several of the significant updates released during and around KubeCon, to get more community members aware and involved.
Query Language Standard for Observability
Today practitioners struggle with a plethora of languages for observability data systems with little consistency or interoperability between them. People may find themselves juggling between Lucene for querying the logs, PromQL to query their metrics, or perhaps using the SQL-like InfluxQL language for their metrics, and other options out there. Some vendors have attempted to create a unified query language, but that has come with the caveat of vendor-locking.
This is the time for an open and vendor-agnostic open specification, something that I got to discuss with Chris Larsen of Netflix at KubeCon. Chris, together with Vijay Samuel of eBay, established a new Observability Query Language Standard (QLS) workgroup to tackle this topic. The working group has recently formed under the CNCF’s Technical Advisory Group for Observability (TAG Observability in CNCF lingo), and you can read the charter document.
The first meeting of the new working group took place May 30th, 2023, and it is starting to shape up. I’m glad to see this initiative being driven by end users rather than vendors, as I think this will set it up for greater success, and I’m looking forward to seeing more end users as well as vendors joining it. Check out my fireside chat with Vijay Samuel of eBay at OpenObservability Talks for the latest on this initiative:
Standardizing the Prometheus Remote-Write Protocol
The Prometheus monitoring project is highly popular, especially for Kubernetes monitoring, with both of these projects hosted under the CNCF. Prometheus heavily relies on the Remote-Write protocol for transmitting metrics data. Seeing its wide adoption in the cloud native sphere, it was only natural to standardize it to enable broader interoperability across metric and time series data.
Recognizing the need for a structured approach, the Prometheus community recently looked to formalize the protocol as IETF open standard RFC2119. This significant step facilitates consistency, compatibility, and streamlined integration with vendor solutions and other Prometheus instances. The official Remote-Write v1.0 specification is now public, you can find the documentation here.
Sunsetting OpenCensus in Favor of OpenTelemetry
Back in 2019, OpenCensus and OpenTracing, two prominent open specifications for collecting distributed traces and metrics, have merged to form OpenTelemetry, a unified open source observability framework under the CNCF.
Now the time has come to bid OpenCensus farewell. OpenTelemetry has achieved feature parity with OpenCensus across various programming languages, including C++, .NET, Go, Java, JavaScript, PHP, and Python. With stable releases of both Tracing and Metrics SDKs, the community encourages users to migrate from OpenCensus to OpenTelemetry, as both OpenCensus and OpenTracing are considered deprecated.
This transition not only offers enhanced functionality but also presents a richer ecosystem of instrumentation libraries, exporters, and an active open source community. The GitHub repos for OpenCensus will be archived on July 31st, 2023. Read all the details on what it means and how to migrate to OpenTelemetry in this post.
Elastic Common Schema Merge into OpenTelemetry
It is a known fact that the old way of logging, with unstructured text flat files, is not sustainable in today’s observability space. OpenTelemetry’s long-term goal has been to standardize on a structured, strongly typed and machine readable log format.
To reduce competing standards and foster alignment within the open source observability ecosystem, the Elastic Common Schema (ECS) has merged with OpenTelemetry Semantic Conventions. Elastic’s accumulated experience in log analytics, based on the popular ELK Stack user community, brings important experience and thought into OpenTelemetry. This collaboration signifies a pivotal milestone in establishing common semantic conventions for logs.
By unifying these frameworks, developers will be able to simplify workflows, integrate log data seamlessly, and enhance compatibility with other observability components. The generous contribution by Elastic amplifies the spirit of collaboration and standardization within the community. You can read more about the merge here, and you can get involved through the Semantic Conventions Working Group.
Progress Towards Stabilizing Logs in OpenTelemetry
The ECS merge into OpenTelemetry is not the only advancement in the logs front. The first major milestone in stabilizing logs happened a year ago, when Logs OTLP (OpenTelemetry Protocol) and Logs Data model were announced stable in June 2022.
Ensuring the stability and maturity of logging capabilities in OpenTelemetry, the project is now actively working on log API and SDK stabilization. The merger of the Logs Bridge Specification was announced around KubeCon, which will facilitate the smooth flow of log data between systems and tools. You can read more on that here.
Concurrently, efforts are underway to separate the specification and semantic conventions, enabling the stabilization of the specification while the ECS merge progresses. These endeavors aim to promote clarity, extensibility, and long-term compatibility, further enriching the observability landscape.
This joins the broader maturity of the OpenTelemetry specification, which is about to release OTLP version 1.0. The OTLP specification defines the encoding, transport, and delivery mechanism for telemetry data, across traces, metrics and logs.
Update (3 July 2023): OTLP 1.0 is officially released.
From Collaboration to Standardization: The Real Power of Open Source Lies with Open Specifications
The recent updates showcased at KubeCon underline the increasing significance of open standards and open specifications in the observability domain. Embracing open source-based specifications not only fosters interoperability but also fuels innovation through collaborative efforts and shared visions.
Furthermore, the successful merge of partially-overlapping specifications, such as OpenCensus and OpenTracing or ECS and OpenTelemetry, as well as successful cross-project collaboration such as with OpenTelemetry and Prometheus, empower developers to seamlessly integrate diverse components, enhance troubleshooting, and derive valuable insights from complex distributed systems. It also brings the best minds of the community under one roof.
As the observability ecosystem continues to evolve, open standards and open specifications will play a pivotal role in creating a thriving and inclusive community.