2019 Area of Focus: Observability
Due to the massive explosion of data and the various components DevOps Engineers must look after, ensuring product performance has become more difficult than ever. New technologies have emerged to increase innovation and product functionality, yet these same technologies can obscure visibility into a system’s performance.
Observability is a methodology that has evolved to help engineers gain greater visibility into their system’s performance. But what exactly is observability? What strategies are engineers employing to ensure observability? And most importantly, why are engineers looking to achieve it? That’s what we uncovered in the DevOps Pulse 2019.
Read the survey below to gain an understanding of the evolution of various DevOps trends as well as a thorough examination of how your peers in the DevOps industry approach observability.
Who took the survey?
The majority of DevOps Pulse respondents were developers practicing DevOps, SREs, platform engineers and software engineers with ten years of experience or more.
Just like last year, in 2019, the overwhelming majority of our respondents were male, emphasizing the tremendous amount of work that still needs to be done to promote gender equality in engineering. While the number of female and non-binary individuals increased slightly since 2018, the number is still extremely low with 91% male, under 6% female, and fewer than 1% non-binary.
What is your role in the company?
Experience level
Gender: comparing 2018 and 2019
What is the size of your team?
Key Findings: Observability (our statements on trends from data)
- Tracing has yet to be fully adopted. 66% of DevOps Pulse respondents do not use distributed tracing tools, but Jaeger is the most popular tool among those who have adopted this technology.
- Tool sprawl is a significant and widespread issue for software engineers. 63% of DevOps Pulse respondents report using more than one observability tool, while close to 14% use five or more.
- Logging is critical for observability. Over 73% reported using log management and analysis tools to gain observability. Infrastructure monitoring and alerting took second place, both at about 40%.
- Open source observability stacks are largely preferred over their proprietary counterparts. ELK is the most popular logging tool, Grafana is the most popular metrics tool, and Jaeger is the most popular tracing tool. Machine Learning is gaining momentum as an observability solution. Almost 40% of DevOps Pulse participants use or are considering machine learning solutions to improve observability.
- As DevOps has become mainstream, R&D teams are sharing the responsibility for observability across multiple roles. DevOps teams are still largely responsible for ensuring observability, but Developers and Operations are not far behind.
- Serverless is the biggest technical obstacle to observability. Despite more than 40% of respondents adopting serverless, 47% claim serverless technology presents the most challenges for obtaining observability.
Defining Observability
Observability has become a popular term in the DevOps and SRE space. But the precise definition has yet to be established. While the majority of DevOps Pulse respondents believe observability is simply using logs, metrics, and traces together, a large number have a less specific approach, defining it as a measure of how well a system’s state can be inferred from output data.
In addition, monitoring seems to be the most popular method for achieving observability, closely followed by alerting, tracing, and instrumentation.
Observability has become an increasingly popular term, what does it mean to you? (Select all that apply)
Which practices/methods do you feel are most critical for achieving observability? (Select all that apply)
The observability stack
The majority of DevOps Pulse participants are using more than one monitoring or troubleshooting tool with 63% using two to four tools and almost 14% using five or more. This clearly demonstrates that tool sprawl exists across the different DevOps roles and responsibilities. As a result, engineers lack a comprehensive and reliable source for managing and maintaining production environments.
It’s also interesting to note that the majority of respondents prefer open source observability solutions over their proprietary counterparts. This reveals the prominent role open source technologies play in modern engineering.
How many observability tools are you using?
What type of observability tools are you using? (Select all that apply)
How many of your observability tools are open source?
Which log management tools are you using?
Which metric monitoring tools do you use? (Select all that apply)
Which tracing tools do you use?
Observability methodologies and strategies
DevOps Engineers bear the most responsibility for achieving observability, as 64% of DevOps Pulse respondents report. However, with DevOps becoming more mainstream, R&D teams are sharing the responsibility for observability across various roles. Close to half of respondents report that developers are also responsible for observability, while 39% say their operations teams are involved as well.
Overall, DevOps Pulse respondents feel they can achieve a moderate level of observability into their systems. Logs are the primary data source used for signals, followed by metrics and events. Surprisingly, only 1.16% use traces. However, 23% use all four data sources combined. Despite the adoption of new technologies, the number of engineers who take two hours or more to identify and troubleshoot issues has increased.
This may be due to various observability challenges the most common of which are complex architectures (almost 58%), lack of knowledge (47%), and too many tools (40%). Serverless was named as the main technical obstacle for observability (almost 48%) followed by containers (30%) and service meshes (30%).
Which practices/methods do you feel are most critical for achieving observability? (Select all that apply)
What is the primary data source you use for signals?
What’s your most common use case for log analytics? (Select all that apply)
What do you feel are the biggest challenges for achieving complete observability? (Select all that apply)
Which technologies do you feel are an obstacle to observability? (Select all that apply)
Who bears the majority of responsibility for ensuring observability? (Select all that apply)
How long does it take your team to identify and troubleshoot a production issue?
Do you use AI/ML to isolate critical events?
On a scale of 1-5, what level of observability are you able to achieve? (1 being low observability and 5 being extremely high)
DevOps tools and evolving trends
It should come as no surprise that CI/CD has become integral to the DevOps movement, with the vast majority of organizations implementing these strategies. In addition, technologies such as microservices, container orchestration, and serverless are on the rise.
The number of engineers exclusively using microservice architecture has jumped from 29% to 37% in just one year. This spike is a tremendous contrast to 2016 and 2017 when adoption stayed consistent at around 18%. Serverless technology is still growing steadily, increasing from a 30% adoption rate in 2017 to 41% in 2019.
By far, the most popular DevOps technology is container orchestration services, which grew from a 25% adoption rate in 2016 to 42% in 2017 and nearly 61% in 2018. Today, more than 63% of DevOps Pulse respondents report using container orchestration services. Kubernetes leads the way as the most popular container orchestration tool, with close to 43% using Kubernetes or a Kubernetes-based solution.
What architecture are you using to build your application?
Are you using serverless in your application logic?
Do you use container orchestration services, and if so - which?
Do you have a Continuous Integration (CI) strategy in place?
Do you have a Continuous Delivery strategy in place?
How frequently does your R&D team deploy code into production? (backend, frontend, etc.)
2020 Predictions
In 2019, the growing adoption of technologies such as microservices, serverless and Kubernetes as well as the rising popularity of CI/CD shows that DevOps tooling has become firmly mainstream and suggest that DevOps processes and methodologies have matured.
Yet we can see from this report that engineers are spending more and more time finding and resolving issues. In 2019, the number of engineers spending more than 2 hours to identify and troubleshoot issues grew to 44%. These numbers indicate that while engineering teams continue to adopt new technologies, these technologies also obstruct visibility into system performance, contributing to a lack of obserability. 58% of our respondents confirm this hypothesis, stating that the number one barrier to observability is complex architectures. This is due to the increasing complexity that technologies such as serverless and Kubernetes bring to modern architectures.
In 2020, we will see:
- Engineering teams invest more in observability as broader awareness of this concept and practices associated with it develop. This is especially true in the enterprise, where observability will dominate the conversation.
- Distributed tracing grow in adoption, with Jaeger paving the way as the industry’s standard distributed tracing tool.
- Increased focus on reducing tool sprawl as vendors consolidate tools for monitoring, troubleshooting, and security.
- Open source continue to be the go-to choice for engineering teams.
It’s an exciting time to be in DevOps and be on the forefront of what is sure to be a transformational year.