One of the great things about my job is having the pleasure of both witnessing the innovation being driven by our development team, and spreading the message about it to our community, and the world at large.
2017 has been quite the year for Logz.io, with a large number of new features and technologies added to the platform. We’ve done our best to update you on the major additions, but the end of the year is a good opportunity as any for a recap just in case you missed out on any of these announcements.
Here’s an overview of the major new additions to Logz.io:
- Data Parsing
- Kibana Snapshots
- Audit Trail
- Live Tail
- Sub Accounts
- Shared Objects
- Customized Alerts
- Application Insights
- Data Optimizer
Let’s take a closer look, shall we?
Correct parsing is a key aspect of efficient and successful log analysis. If your log messages are not processed correctly and broken up logically and accurately, you will find it much harder to analyze them in Kibana.
Logz.io supports automatic parsing for a variety of different log types but the addition this year focused on allowing our users to manually configure the parsing of their logs within a dedicated page in the UI. Users can select log messages from their data to use as samples and use color patterns to match between the fields in the logs and the parsing results.
More information about Data Parsing and how to use it is available in this post.
Simply put, Kibana Snapshots allows you to share a snapshot of a Kibana visualization or dashboard with your Slack team members.
Kibana already allows cross-team sharing of visualizations and dashboards using share URLs, user tokens, and filters. Logz.io users can share these objects safely and securely, and Kibana Snapshots further enhances this sharing experience by allowing you to instantly share visualizations over Slack.
To share a snapshot with your colleagues, just click the small snapshot icon available on each visualization and dashboard, and decide who you want to share the snapshot with (email address, Slack and other supported endpoints).
System and application audits allow admins to keep tabs on user activity. This is important both for internal control and monitoring but also for compliance with external certification programs.
Logz.io’s Audit Trail feature that was added this year, tracks user activity in your account, and records any action that was performed such as logging into the system, saving a new Kibana query, creating an alert, adding a user, updating a dashboard, etc (a full list of audited actions is available here).
What do you call an ELK with a tail? Logz.io’s Live Tail!
As useful as Kibana is for analyzing log data, sometimes the need for live tailing a specific file arises. Using tail -f for viewing the output of a specific process in a log file is great but what if you had the ability to see a stream of all the logs coming into your centralized logging system?
Logz.io’s Live Tail added a much-requested feature to Kibana and allows you to see a live feed, in real time, of the logs being shipped from different data sources. You can filter these logs, highlight syntax, and use a variety of additional features to handle the stream of data.
More information about Live Tail is available here.
This feature allows users to control and manage their account more efficiently by giving them the ability to run sub-accounts under their main account. This helps organizations to logically differentiate logs according to the apps, services, teams, or the environments generating or managing them.
Users can define different data volumes and retention periods for different environments under their main account. For example, under the main account, a manager could define one sub-account for development, one for staging, and another for production.
The Shared Objects feature allows the sharing of objects (dashboards, visualizations, searches) between different sub-accounts. Meaning that if shared, there is no need to build these objects from scratch in each individual account.
Each main or sub-account can be configured, either when created or at any point in time thereafter, to gain access to the objects configured by another account. This is done using the New/Edit Sub Account dialog, accessed from the Settings → Manage Accounts page.
Logz.io’s built-in alerting engine allows users to get notified when a log-based event occurs. This year, we added new capabilities to this mechanism that allow users to make sure that the alerts include only the data you are really interested in and they are written in a readable format. Using these customization features, users can decide whether to receive the alert in JSON or in a table format. In the case of the latter, users can decide which fields, and which parts of the data within these fields, to see in the alert.
Also added are two additional grouping levels that allow users to group together values for fields in the alert.
More information about these alert customization options are available in this blog post.
Application Insights is the latest machine learning feature added to Logz.io suite of enterprise-grade analytics tools.
This feature gives users more visibility into their environment by identifying new exceptions and errors thrown by their applications, and by giving them the tools and contextual information to easily resolve them.
Users can now easily see new exceptions and errors and correlate them with changes applied in the environment and other types of events identified by Logz.io (Cognitive Insights) using interactive visualization tools and a variety of other available actions.
Read more about Application Insights in this article.
As companies grow, so does their log data. This growth in data comes at both an operational cost — it is becoming increasingly challenging to analyze the growing volume of data and extract insights from it, and an actual cost — more data means more storage and more expensive logging tools.
The Data Optimizer seeks to alleviate the resulting paradox, between the growth in data on the one hand, and the decrease in the value that can be extracted from it on the other, by allowing companies to reduce logging costs and noise.
Instead of storing all of their log data for months at a time, users can decide to aggregate and summarize data in dedicated Timeless Accounts, with extended retention. Using Optimizers, users can summarize their logs, in essence reducing their original size, and aggregate the results in these accounts.
Read more about the Data Optimizer in this article.
Looking forward to 2018
This a list of the major improvements added during 2017 to Logz.io. Needless to say, the list does not include additionally related announcements and changes — such as the addition of Timelion, achieving security compliance and certifications (SOC II Type 2, HIPAA, ISO27001, GDPR, PCI level 1) and more.
Looking forward, I’m extremely excited about the plan for 2018.
We will be open sourcing some of the components we developed as part of our ELK-based architecture and there is a long list of major features in the pipeline. Rest assured, you will know as soon as these become available, so stay tuned.
As always, we’d love to get your feedback. So if you have any questions or ideas, feel free to let us know at: email@example.com.