Kibana is one of the major reasons that developers, DevOps engineers, and IT teams opt for using the ELK Stack (Elasticsearch, Logstash and Kibana) for centralized logging.
For those of you who are not acquainted with the tool, Kibana is an open source data visualization tool that is most commonly used in conjunction with Elasticsearch and Logstash (or any other log shipper) as the stack’s “pretty face.” Kibana is popular for many reasons — being able to slice and dice your data in any way you want is probably the most obvious one.
But nothing in life is perfect, and Kibana is no exception to this rule. While relatively simple to use in general, some features require a high technical bar. Take Kibana visualizations, for example. These can be extremely difficult to create for beginners and advanced users alike. More so, Kibana, as supplied out-of-the-box, lacks some capabilities that are crucial in enterprise deployments and that users often complain are missing (such as an alerting mechanism).
Understanding this need — especially the following feedback from our users — we at Logz.io have added a number of complementary features that make Kibana an even better tool to use to visualize and analyze Big Data.
1. ELK Apps
While being able to visualize data and build monitoring dashboards is one of Kibana’s most popular features, the truth is that it is not always easy to use. Being able to configure accurate aggregations for the X and Y axes of a specific chart can be extremely time-consuming. This issue is exacerbated by the fact that different logs are built differently, meaning that a configuration for an Apache visualization, for example, will not necessarily work for an IIS or NGINX visualization.
Since we went live with ELK Apps, we have received dozens of contributions from users. The library currently consists of 142 apps for: Apache, NGINX, AWS (ELB, CloudFront, CloudTrail, VPCFlow, S3), Docker, MySQL, Nagios, IIS, HAProxy, and general system logs. We’d love to see your contribution as well!
Using ELK Apps is extremely simple: Just browse through the library and install your app of choice with one simple click. Users can also contribute their own apps to the library.
2. Cognitive Insights
One of the biggest challenges of analyzing log data in Kibana — and indeed in any log analysis platform — is the ability to find the information that matters. There is simply too much data to sift through, and most of the time people are not even sure what to look for. Even when people do know what log message they need to analyze, querying Kibana to find that specific log is not always the easiest of tasks.
Often enough, events are taking place in your environment that you are not even aware of. These events might not matter, but they also may be indicators of a catastrophe about to take place that will have the potential to seriously affect your business.
Cognitive Insights is a new AI-powered feature by Logz.io that cancels out the noise made by the large volumes of data in systems and pinpoints the log messages that you need to look at. It does this by correlating your log data with a huge (and growing) database of searches, alerts, and forums on the Web.
The result of these correlations is “Insights” — specific log messages that include details and contextual information on events as they occur in real-time, including meta descriptions and a list of online resources where the issue has already been discussed and troubleshooted.
Log analysis platforms such as the ELK Stack are mostly used for after-the-fact forensics and troubleshooting. But once you’ve found that one SQL error that caused your website to crash, how do you make sure that you will receive an alert the next time that a catastrophe is taking place?
Logz.io comes with a built-in alerting mechanism that allows you to create alerts on top of Kibana queries.
Alerts can be sent to your email address or you can use webhooks to send notifications to Slack, HipChat, JIRA, or any other third-party chat platform that works with webhooks.
4. User Management
Being able to manage who has access to the log data is a basic requirement for any company and a crucial element in complying with international security and auditing standards.
Authorization and authentication are key to protecting the logs that are coming into your ELK Stack, but the problem, again, is that there is no easy way to implement a data protection strategy. Elastic offers Shield, an add-on product for controlling and granting access to Kibana, and, of course, you could try to hack your own solution.
As an end-to-end service, Logz.io comes with built-in user control and management. This includes authorization and authentication, as well as simple user management. You can easily invite new users, suspends old users or edit permissions.
In addition, you can safely share Kibana searches and visualizations with user tokens to decide who in the organization can see what. You can read more about that here.
5. Proximity Events
Another nifty feature that we’ve added to Kibana is the ability to easily see the messages that were logged directly before and after a given message in Kibana.
In a centralized logging system in which Kibana displays messages being ingested from multiple data sources, it’s crucial to be able to understand whether there is a relationship between the different messages that are being logged chronologically.
Take, for example, an Apache 500 response that takes place right after a bad MySQL transaction. These are related log messages that point to the exact root cause but may go unnoticed because they are recorded in two different data streams. Being able to see proximity events allows users to see the big picture and the overall context in which a specific event occurs.
What Kibana 5 Will Add
The list above reflects some of the benefits that Logz.io adds to Kibana but it definitely does not include them all.
There is no doubt that Kibana is a rich visualization tool that can be used for multiple use cases, but the truth remains that in large deployments, the features on offer might feel constraining.
It’s worth noting that the upcoming release of Kibana 5 adds plenty of new features and improved UX, but it does not necessarily compensate for the missing functionality described above.