Reducing Operational Noise (and Cost) with Data Optimizer

By: Daniel Berman

If you’re in charge of running DevOps or IT operations in your company, this question might sound familiar: “Why do we need to save all of this log data?” Or perhaps, “I wish we could save this data for longer!”

The reason these questions resonate is that they reflect a compromise most companies are forced to make when selecting a log analysis solution.

Why is that?

The existing data paradigms in the log analysis industry are based on two interrelated axis: data volume and retention. As organizations grow, so does their log data. This growth in data, in turn, drives up costs as well. The paradox is that despite this growth in volume and cost, the actual value extracted from the data stagnates, if not declines due to the operational noise created by the data.

Moreover, in many scenarios and use cases, a large amount of the data does not require long-term retention. For troubleshooting and forensics, for example, most teams would require no more than 3-5 days retention, while for business or security use cases, a company might need retaining the data for a year, or even longer. The pricing models used by log analysis solutions today disregard this simple truth, and offer an all-or-nothing model that forces companies to pay for retention plans that do not take this differentiation into account, are inefficient and needlessly costly.

Introducing the Data Optimizer

Logz.io has developed a technology that will help alleviate the paradox detailed above by allowing companies to differentiate between important data that requires extended retention and data that does not.

When the need arises, Logz.io users can now select to aggregate data to a dedicated Timeless Account, with no retention limits. They can set an Optimizer rule that defines exactly what data to store in this account, when to store it and in what format.

Let’s take a closer look at how the Data Optimizer works.

Creating New Timeless Accounts

New Timeless Accounts are created, listed and managed from the Timeless Accounts page, under Settings (your regular retention plan, what we call a Time Based retention plan, is managed from the Manage Accounts page).

On this page, you will be able to see an overview of your plan — the allowed data volume, the number of allowed accounts and a breakdown of any existing Timeless Accounts.

If this is the first time you are creating a Timeless Account plan, you will see an account has already been created. This is the default Timeless Account, created with a maximum volume allowance based on your plan.

Click this default Timeless Account and reduce the data volume assigned to it. Now that you have some available space to play around with, click the Add new button to create a new Timeless Account.

All you have to do now is enter an account name, the data volume for the account and a list of the accounts with allowed writing and reading access. Hit the Create button to add the new account.

You are only allowed to create up to 5 accounts, and of course, if the data volume in existing Timeless Accounts has exhausted your plan, you will not be allowed to create your new account.

Hovering over the pie displays some basic info on the relevant account. You can select a pie segment for a specific account to open up a section with more details such as the account name, data volume for the account, the accounts with writing and read access and a graphical depiction describing the indexing rate of log data into the account.

Creating A New Optimizer

So how do you store data into a Timeless Account?

This process is handled by what is called Optimizers. These Optimizers are based on Kibana queries and aggregations and contain schedule configurations that allow users to decide what and when to save data to a Timeless Account.

This configuration is performed from the Create an Alert/Optimizer page, which in turn, can be accessed either from the Discover page in Kibana (Create an alert button) or directly from an Application or Cognitive Insight.

Similar to configuring an alert, you can customize the query in the Query box at the top of the page. You can use grouping to group results using a specific field (you have 3 grouping levels to use).

In the Definitions section, enter a name and description for the rule (e.g. application errors).

In the Action section, select Optimizer and then select the Timeless Account you want to save the data from the relevant drop-down menu.

Configure the specific scheduling rule — what type of aggregation type you want to use and how often you want Logz.io to save the data to the account.

Hit Create when done.

That’s it. The Data Optimizer will begin to aggregate data into the selected Timeless Account using the definitions you entered above. You will be able to query this data from the Discover page in Kibana, just as you would analyze your other data.

Knowing What Data to Retain

OK, so instead of saving all your data for an extended time period, you can decide what to set aside in a Timeless Account. But what data is important? How do I know what logs should be set aside to start with?

Well, this is the million dollar question. The answer to this question depends of course on the data and the environment generating it. Examples of important data requiring extended retention range from performance metrics to application error logs. The decision is yours.

Summing it up

Logz.io’s Data Optimizer offers a new way of building and thinking of your retention plans by giving you the management granularity and flexibility you need to decide what data to save for a longer period of time and what data to discard after a short retention period. Logz.io will help you with identifying the logs that matter so you know what data to set aside in timeless accounts that live forever.

Why pay for retaining logs that don’t matter?