Gaining insight into how much data you’re shipping to Logz.io, or any other logging platform for that matter, is important to gauge how much of the allowed quota in your plan is being utilized.
That’s why we recently introduced the ability to effectively monitor account utilization metrics. Each Logz.io account can now save utilization metrics on a set schedule (every 10, 30 or 60 mins). These metrics include the used data volume for the account as well as the expected data volume for the current indexing rate.
Once recorded, you can use these metrics to manage your Logz.io environment more actively — create an alert should a certain threshold be exceeded or create a dashboard monitoring your data volumes. This article will explain how.
Enabling account utilization metrics
Your first step is to enable account utilization metrics for your account or accounts.
This can be done by going to the Manage Accounts tab on the Settings page. Select an account, and under Advanced Settings, select the Save account utilization metrics check-box.
You can then define the interval you want those metrics to be captured (default is every 10 minutes, but you have the option to choose 30 and 60 minutes as well).
Hit the Apply button to apply your changes.
Account utilizations logs
After a while, a new “logzio_account_utilization” log type will begin to show up in Kibana with the reported metrics. To search for these logs in Kibana, use this query:
Here is what the different fields in these logs mean:
- account_id – The Logz.io ID of the sub account in our system
- account_name – The name of the Logz.io account / sub account
- expected_utilization_EOD – The expected utilization % by the end of the day based on the plan volume for the sub account
- expected_volume_in_GB_EOD – The expected utilization in GB by the end of the day
- grace_capacity – The overage configured on the account in % (default is 120%)
- plan_volume_in_GB – Data in GB allocated to the specific account
- type – logzio_account_utilization is message type where these metrics are stored
- utilization – current utilization is %
- volume_in_GB – current utilization in GB
Installing the account utilization dashboard
You can begin to analyze and visualize these metrics in Kibana, but to make it easy for you, we’ve prepared a basic dashboard called “Data Volume Dashboard” that is available in ELK Apps — our library of visualizations and dashboards for different log types.
So, open ELK Apps, search for “data volume”, and click the Install button to install the dashboard.
The dashboard contains two visualizations – one is a time series line chart broken down by sub account, and the second is a data table that gives you a full GB breakdown for those accounts.
Let’s understand what we’re seeing in the dashboard.
To build a line chart that shows utilization % throughout the day, we configured a line chart like below. For the Y Axis, we used a Max aggregation of the utilization field. For our X Axis, we used a date histogram and a split series using a Terms aggregation of the account_name field.
This visualization gives us a time series view of the utilization percentage, per sub account, throughout the day.
For the data table, we are using a Max aggregation for three metrics: plan_volume_in_GB, volume_in_GB, and utilization. For buckets, we used a Terms aggregation of the account_name field. This will give you a data table of current utilization for each sub-account across all of those metrics.
Of course, these same principles can be applied to other visualizations as well (gauges, meters, etc.).
We are continuously trying to help our users make the best out of their log data and optimize usage of our service. A key ingredient in this recipe is to give our users the tools to better understand what data is important or not and to optimize storage and account utilization accordingly.
These latest improvements will help users monitor noisy accounts, in order to fine-tune logging accordingly and in general — make better use of the services they are paying for. We are currently working on additional features that will empower users even further in this respect, so stay tuned for news.