OpenSearch and Logz.io – Taking Observability to the Next Level

If you’re in the cloud engineering and DevOps space, you’ve probably seen the name OpenSearch a lot over the last couple of years. But, what is your current understanding of OpenSearch, and the components around it? Let’s take a closer look.

To understand OpenSearch, we should start with a simple review of the basic concepts of observability, and why it matters. Building and maintaining observability into cloud environments is a critical element of how  we can understand precisely how a system works, allowing us to gain detailed visibility into relevant information. Within the context of cloud applications and infrastructure, this matters on both an operational and business level – to monitor performance and importantly to streamline troubleshooting. 

Practically speaking, we make software systems observable by using instrumentation in our code to surface information for monitoring, details that would otherwise be impossible to know from the outside – such as why user latency is high for a specific type of request. 

Taking this methodical approach to instrumentation and monitoring subsequently leads  to wider availability of good data, which can help with everything from day to day operations to enabling the technical shift to cloud native. It can also result in cultural improvements within an organization, such as better transparency and visibility. Decisions can now be  made using more exacting measurements as opposed to guessing.

For years now, organizations have been following these practices and deploying various monitoring and observability tools, including a number of open source technologies. Among the most popular is the ELK stack, and the Elasticsearch search and analytics solution. In 2021, Elastic, the company that develops Elasticsearch, announced that it was changing its licensing off of a pure open source approach. As a result, the industry went looking for other options.

That brings us to OpenSearch. The AWS open source project forked off of Elasticsearch as a response to this situation has proven to be a gamechanger in the observability space in just a short lifetime. All of this was the subject of a recent webinar we co-hosted with DevOps.com and AWS, and you can watch the replay here:

In short, OpenSearch can provide end to end observability. It is community-driven, open source search and analytics-based on Elasticsearch 7.10.2. This service provides elements such as natural language text search, high volume data ingestion, and analysis on time based visualizations. A wide range of organizations are moving to OpenSearch quickly based on all the excitement and innovation going on around the project, and Logz.io is one of those companies.

How OpenSearch and Logz.io Work Together

Why should you use OpenSearch and Logz.io? Put simply, Logz.io is a full stack, unified observability platform built on OpenSearch – along with other open source projects including Prometheus and Jaeger – -providing a full observability solution spanning logs, metrics, traces and security monitoring. While contributing directly to the OpenSearch project and the open source community, we also strive to make open source simpler by extending these tools to provide better observability using numerous expanded capabilities.

For instance, a current widespread challenge that we see among organizations is the explosion of available observability data, which is causing massive issues in allowing people to extract valuable insights, while also causing costs to skyrocket. To address this and a wide range of observability requirements, Logz.io has recently added an array of new capabilities  to combat these issues via our Open 360™ platform, including:. 

•Logz.io Telemetry Collector: A single agent to collect data (speeding time to value, and simplifying onboarding)

•Logz.io LogMetrics Index: Transforming specific log types into metrics for trend analysis (to help lower related costs)

•Logz.io Data Optimization Hub: Helping to reduce costs and increase simplify by surfacing unneeded data (to reduce complexity and costs)

•Logz.io Filtered Log Restore: Adding filters to “archive and restore” from S3 buckets (to make related querying easier and  reduce costs)

•Logz.io Trace Sampling Wizard: To generate OpenTelemetry config files instead of requiring you to  build them manually (simplifying instrumentation to collect trace data with OTEL).

•Adding generative AI to Cognitive Insights: enriching Open 360’s exiting Cognitive Insights model with known solutions to emerging issues.(to speed troubleshooting and optimize resources)  

•Kubernetes 360: Unifying the best of open source logging, metrics and traces in a single platform purpose-built for Kubernetes monitoring (to address related complexity)

To further optimize your observability using OpenSearch you can also check out Logz.io on the AWS Marketplace offering simple migration. 

Watch the on-demand webinar for more information on this and try a demo to see these tools in action!