Kubernetes and Beyond: A Year-End Reflection with Kelsey Hightower

With 2023 drawing to a close, the final OpenObservability Talks of the year focused on what happened this year in open source, DevOps, observability and more, with an eye towards the future. 

I was delighted to be joined by a special guest, Kelsey Hightower, a renowned figure in the tech community, especially known for his contributions to the Kubernetes ecosystem.

We reflected on the significance of open source in the tech landscape, where the power of collaboration gives birth to groundbreaking ideas. We delve into the maturity of various projects, including Kubernetes, and the realization that stability becomes crucial as these projects mature. Kelsey touches upon the evolving landscape of cloud-native technologies, acknowledging that leaving the cloud doesn’t mean abandoning cloud-like technologies. 

The discussion highlights the success of industry efforts to standardize the cloud, making technologies like S3 protocol accessible beyond vendor-specific platforms.

Where is Observability Headed in 2024 and Beyond?

On observability, we explored the convergence of logs, traces, and metrics and Kelsey provided valuable insights into the evolution of these observability components. We delved into the challenges and opportunities in normalizing semantic conventions and specifications across observability tools.

Kelsey shared pragmatic insights into the role of AI in observability and challenged the notion that AI should replace existing UIs. He argued that the current UIs already reflect the mental models of experienced individuals. 

He envisions a future where AI aids in making data-driven decisions but cautioned against getting distracted by the hype surrounding AI.

“The future is going back to the way humans already work, but just teaching our systems to allow that level of customization,” he said.

The Future of Kubernetes: From Birth to Maturity

Kelsey draws parallels between the trajectory of Kubernetes and Linux, envisioning Kubernetes as a foundational component that transcends specific implementations. He explores the potential of workload-specific platforms, predicting a shift where Kubernetes might become less visible but remains a crucial foundational element.

Kelsey shares his perspective on the evolving nature of Kubernetes, emphasizing the importance of structuring declarative APIs and separating control loops from CRD implementations. The conversation closes with a glimpse into the future of infrastructure as data, opening up new possibilities for user experience and interface design.

We also went deep on trends in containerization, tracing roots back to Docker. Docker brought containers into the limelight, popularizing Linux Containers and harnessing the container runtime capabilities within the Linux kernel such as LXC. However, as Kelsey pointed out, what was once a source of excitement eventually became a commodity.

Kubernetes emerged as a transformative force, pushing Docker and other container runtimes to a lower layer. Kelsey drew a parallel with the evolution of Linux, suggesting that Kubernetes might follow a similar trajectory. If, 20 years from now, we’ll still be talking about Kubernetes as the front end of what we do, that would mean no progress happened, Kelsey noted.

“I think if you’re being honest and you really want the same evolution that brought us to Kubernetes, you should probably still want the same evolution that makes Kubernetes disappear,” he said.

Where Kubernetes and Specialized Workloads Are Going

The conversation then shifted to the expansive use cases of Kubernetes, ranging from traditional backend applications to edge computing and IoT. I expressed concerns about stretching Kubernetes too thin, especially when applied to specialized workloads like AI/ML (artificial intelligence and machine learning).

Kelsey countered this by illustrating how Kubernetes can effectively handle AI workloads, breaking down the steps involved in running AI processes within a Kubernetes environment. He emphasized that Kubernetes’ generic nature allows for pluggable components, making it adaptable to a variety of use cases.

Kelsey discussed the balance between supporting specialized workloads and maintaining simplicity in Kubernetes. He highlighted the need for frameworks and platforms to be built on top of Kubernetes, citing examples like Cloud Foundry and Heroku. 

While acknowledging that Kubernetes may not be perfect for every use case, he emphasized its role as a foundational layer that can be augmented by specialized platforms. I still think that expanding API to support specialized workload can have an impact on stability, reliability and security, that needs to be taken under advice.

Unpacking Complexity and Abstractions in Observability

The conversation took a detour into the complexity of Kubernetes, as experienced by many users of the open source. Many even contrast that with the perceived simplicity of using virtual machines (VMs). Kelsey dissected the challenges of managing VMs, from configuring operating systems to load balancing and network connectivity. Kelsey even went as far as dealing with machine racks, power and cooling, which to me was too far, as neither Kubernetes uses nor VM users handle datacenter management these days. 

He argued that Kubernetes, while still requiring management, simplifies these complexities by 

providing a layer of abstraction. The equivalent in VMs would amount to many additional tools and frameworks that one would need to set up, where Kubernetes packages that all in, and  transparently exposes these elements to the end user. 

Moving beyond Kubernetes, the conversation delved into the Cloud Native Computing Foundation (CNCF) and the growing complexity of the cloud-native landscape. I expressed concerns about the overwhelming number of projects and options within CNCF, causing confusion for users.

Kelsey countered by distinguishing between choice and complexity, asserting that transparency in the form of various projects is beneficial. He compared this to a watch, stating that a complex watch with visible components is often perceived as more intricate than a sealed watch with hidden mechanisms.

The discussion on observability brought to light a crucial question: Why are we observing? Kelsey challenged the prevailing approach of collecting massive amounts of data without a clear purpose. He envisioned a future where observability is driven by specific goals, leading to actionable insights.

Drawing an analogy with cars, he emphasized that observability should focus on essential metrics, like speed, with a clear purpose in mind, in this case not getting a speeding ticket. The conversation highlighted the need for the industry to mature and move towards building more predictable systems to reduce the reliance on extensive observability.

“We observe for a purpose,” he said. “I think where we’re going is that more systems will get closer to the reason why we observe, and that will be the new UI.”

The Future of Observability: A Call for Maturity

In addressing the future of observability, Kelsey stressed the importance of systems becoming closer to predictable finite-state machines. He advocated for software engineering to mature, with an emphasis on building software with clear specifications and limited side effects.

While acknowledging the challenge of changing the perception that stability is boring, he emphasized the necessity of moving away from the current paradigm of constant firefighting and embracing a more mature approach.

The conversation with Kelsey provided valuable insights into the evolution of observability, the role of Kubernetes and the cloud native ecosystem, and the challenges facing the software industry. As the industry continues to grapple with complexity, the call for maturity in software engineering and observability becomes increasingly relevant. 

Observability, when driven by purpose and a clear understanding of system behavior, holds the key to a more stable and efficient future in software development.

Want to learn more? Listen to the full discussion on the latest episode of OpenObservability Talks: Kubernetes and Beyond: A Year-End Reflection with Kelsey Hightower.