Product Spotlight: Smart Tiering + LogMetrics to Optimize Costs

Is all observability data worth the same cost? 

If you guessed no, then you’d obviously be correct. Anyone familiar with the very nature of gaining targeted observability knows that some data points hold more value than others. Yet, many observability platforms still treat all types of log data the same, and as a result, related costs remain uniform. 

One of the most persistent observability challenges today is the cost of indexing log data. Combined with the traditional approach of “index and monitor everything,” some might argue that, as a result, many organizations’ observability strategies are broken.

Organizations can tame the costs of their log data by finding innovative ways to extract value from the logs without indexing them.

That’s why we built LogMetrics, which provides low-cost log visualization, as part of our new Open 360™ platform. Customers can easily convert logs to metrics to minimize indexed log data, while still providing detailed access to the critical insights that you ultimately need to troubleshoot your environment. 

Read on to learn how you can accomplish this with Logz.io. 

Data Costs Should be Variable

Data types are used in different ways, so there should be a practice in place that aligns the value of the data with its cost. For example, as data ages, we’ve seen that it is accessed far less frequently than data we want to observe in real-time. 

When optimizing logging costs, you can think about your data in terms of their use case and how often they’re being accessed, such as the following: 

• Critical troubleshooting data frequently used for real-time debugging
• Monitoring data needed to surface production issues
• Aging data that is infrequently accessed
• Compliance data that is seldom used

It’s important to be able to access all of your data, but is it necessary to be paying for the highest level of availability for all of it? As cloud workloads and telemetry data volumes grow, it is becoming increasingly difficult for engineering teams to keep their costs under control. We all want to have access to all of our log data while optimizing query latency, but how can we minimize the cost of doing so?

Diving into LogMetrics

For log data that is used to monitor for production issues, LogMetrics reduces indexing costs by converting logs to metrics. Once converted to metrics, you can clearly visualize and analyze your log data within our product while saving on costs and storage.

The immediate benefit of employing LogMetrics comes from the fact that metrics storage requirements are much smaller than those of logs. This makes it perfect for log data that doesn’t need deeper analysis and might benefit from being aggregated and visualized for easier comprehension. 

For example, some of your logs generated from a Kubernetes cluster might need to be investigated and read. These records tend to describe all of the changes made by Kubernetes – to see when there is a failure, you’d need to investigate specific log files. 

In other cases, you may not need the same availability for the log data. When observing the frequency of HTTP status codes, for example, vital information can be more easily visualized using a graph. Log data like this is a perfect use case for LogMetrics, as you can save on costs while still getting the full benefit of our log visualization capabilities.

Combining LogMetrics and our new Smart Tiering capabilities

Previously, Logz.io Smart Tiering allowed you to define a data management policy that divides your data across tiers based on your desired balance between cost, performance, and availability. 

While Smart Tiering is an effective solution for managing your data to reduce costs, we knew there was a way to improve the storage cost of log data by incorporating LogMetrics and revamping our data tiering system around it. Today, we have the new Smart Tiering system that improves on our previous solution to further reduce the cost of observability data.

Our first new tier is called our Troubleshooting Tier, which consolidates the functionality of our Real-Time and Smart Tier. All of your data in the Troubleshooting Tier will be enabled for full log data analysis, and all data is cooled after 5 days to reduce costs while ensuring high query performance. 

Our next tier is now called the Monitoring Tier, which provides lower-cost log data visualization through LogMetrics monitoring. This tier is best used for log data that can be summarized through visualizations and don’t necessarily need to be read. 

Lastly, we have our Long-term Tier, which enables you to utilize our lowest-cost storage for your log data in AWS S3 or Azure Blob. These logs can be re-indexed at any time for use.

The new Smart Tiering system will help distribute your log data across multiple storage options, and optimize your costs depending on how you use the data.

Summary

Cost optimization means getting more value for less cost. 

It’s not enough just to provide the infrastructure for your telemetry data. We strive to make sure that your data is managed in the best way possible at the lowest cost to you. 

With LogMetrics, we’ve created a new way to eliminate the cost of indexing your log data without sacrificing the critical information you get from it. 
Lower your data observability costs today by signing up for our free trial.