Survey: Complexity and Costs Threaten Health of Strong DevOps Pulse

“Life moves pretty fast. If you stop and look around once in a while, you could miss it.”

– Ferris Bueller

Oh how quickly the times change. Over the past 5 years, as Logz.io has executed its annual DevOps Pulse Survey and created the accompanying report, the state of “modern” cloud engineering and monitoring practices have advanced at a torrid pace.

As a result, so has the project itself. Back in 2017 when we started polling the industry and collecting data, we sought to validate the usefulness of DevOps itself. Likely to no one’s surprise, our audience heartily endorsed this growing model – with 90% of respondents stating that DevOps and increased collaboration had a positive impact on the quality of their products.

Over the years we’ve moved through primary topics including the increasing security concerns, the rise of “full-stack” observability, and the growing proliferation of microservices architecture, among others.

DevOps Pulse for 2022

This year’s survey, [with data collection in late 2021], continues to reinforce all of these prior trends, and, in addition to touting the growing maturity of cloud and DevOps practices, elicits some fascinating trends, including

• Most respondents cite increased maturity in their cloud and DevOps journeys, and a lion’s share of organizations (77%) reported that their observability strategies are well underway – yet, a few big challenges loom, notably:
  
• Roughly 64% percent of respondents reported that MTTR during production incidents was over an hour, compared to only 47% one year ago. This means that despite all the hard work, troubleshooting is getting more difficult. Why? For one thing:
  
• Monitoring Kubernetes and microservices remain difficult: with over 52% of respondents citing Kubernetes, microservices, and serverless among their primary challenges in gaining observability into their cloud environments. On a related note:
  
• Distributed Tracing remains nascent: while adoption of Log Management (67%) and Infrastructure Monitoring (59%) is now relatively high, only 27% of respondents report that they have deployed Tracing capabilities.

This is important on the point of monitoring of Kubeternes, serverless, etc. because Tracing is the element of the observability stack that, when effectively deployed, should/could play the biggest role in helping organizations visualize and troubleshoot these systems.

Also, there’s a lot of appetite for Tracing. An overwhelming 80% of 2021 respondents voted “3” or higher when asked to rate the importance of tracing to their overall observability strategy on a 1-5 scale. This is compared to 68% who ranked it 3 or higher last year. Further, in the 2020 survey, 32% of respondents said they had no current plans to invest in distributed tracing. This year, that number fell to 25%.

Planned deployment of Tracing is also growing: compared to the 2020 survey, more organizations that have not yet deployed tracing have immediate plans to do so. While 65% of respondents reported plans to deploy tracing in the next 1-3 years in 2020, in 2021 that number grew to over 75%.

But, what about cost and complexity?

Now, let’s turn to perhaps the most important macro-trend in this year’s survey – and the one that we see impacting DevOps organizations the most in their day-to-day efforts – issues related to the growing cost and complexity of observability itself.

More than ever, we hear from practitioners that they are drowning in observability data, that this is making life difficult in terms of lowering signal to noise ratio [another reason for longer MTTR times, for sure] and that the cost of attempting to monitor all this data using traditional tooling and practices is not working anymore. To wit:

• Observability tool sprawl persists: with 90% using multiple tools, 66% using at least 2-4 observability systems, and roughly 24% employing anywhere from 5 to 10. In 2020, roughly 20% were using only 1 tool, with only 10% using over 5.
  
• Observability costs and data volumes are growing concerns: with some 27% of respondents ranking total cost of ownership and the large volumes of data being ingested into the tools among their main challenges in maintaining effective observability into the cloud.

• Lacking visibility into observability costs also persists: as 43% of respondents indicated that efforts to increase visibility into observability costs has become a primary goal – suggesting many organizations are struggling to do so in the face of spiraling data volumes and popular vendor pricing models.

While none of these numbers are huge, we absolutely see this as a trend that bears greater consideration both today and into the future. 

Organizations simply cannot follow a strategy of dumping all of their observability data into a platform or many platforms, at a growing expense, without any real expectations of gaining greater visibility! Helping customers tackle this very issue is our absolute top priority at Logz.io – but let’s stick to summarizing the research in this piece.

All in all, the 2022 DevOps Pulse Report, which you can download here, shines a light on a lot of amazing growth in our space as DevOps practitioners hone their skills, and both processes and tooling evolve.

Register today for our upcoming webinar on the report conclusions, and stay tuned as the project rolls into its sixth year!