6 Things to consider when choosing a log management solution

The days when you could simply SSH into a server and perform a fancy grep are long gone. If you’re reading this article, chances are either you are looking to move from that obsolete approach to a centralized logging approach with a log management tool, or you are looking for an alternative log management tool to replace your existing solution.

Problem is, there are so many different tools out there, making a choice can be overwhelming. So how do you pick the right solution?

We’re all snowflakes. We work differently and have requirements that vary from team to team, organization to organization, and use case to use case. It would be presumptuous and highly erroneous on my part to prescribe a one-size-fits-all framework for choosing a log management service. Still, there are some common key requirements that any log management solution today must meet to be even considered, whether for IT operations, security analytics, DevOps or compliance.

Yes. Logz.io meets all these requirements. So do other solutions in the market, to some degree or other. My purpose here is not to highlight Logz.io but to provide readers with an understanding of what they cannot afford to miss out in their search process.

1. Data collection

Let’s start with the basics — collecting the logs from your data sources. Every log management tool promises to easily collect your log data, but it’s up to you to make sure that this promise holds water for your specific environment.

If your application is deployed with Kubernetes, for example, can the log management solution integrate natively with the Kubernetes logging architecture? If not, how much extra configuration is required? Is data collection automated when you auto-scale or do you have to set up integration for each new pod or node as it is deployed? And once collected, does the solution process that data or do you have to apply additional parsing to the logs manually pre-ingestion?

Centralized logging is a basic concept in theory but implementation can be complicated in modern architectures. Be sure you verify that data collection is indeed as simple and seamless as promised.

2. Search experience

Every log management tool must enable users to easily search their log data across multiple data sources. This sounds super-simple, but it actually involves much more than just entering a query in a search field.

First, performing searches must be easy. This means that the search syntax should be simple enough to perform simple text searches on the one hand and be robust enough to support more complicated queries.

Second, additional search features such as autocomplete, autosuggest, or the ability to easily add field and time-based filters can have a dramatic impact on the search experience

Third, and perhaps most importantly, searches must be fast and return results quickly. Which brings me to the next point.

3. Scalability

As we all know, log data today fits the three “V”s that define big data. The sheer volume, velocity and variety of log data makes log management tools crucial for effective monitoring and troubleshooting.

Engineers need to be able to easily access and analyze the huge volumes of log data their applications and infrastructure are generating. When an issue occurs, they cannot afford to wait for a minute or two until a query returns results. They need speed, regardless of the amount of data they are collecting and querying.

Nor can they afford to lose data. Every log message is important. And so, the log management solution you select must be scalable enough to support data bursts, data growth and cloud scale.

Most vendors tout scalability. Not all of them actually provide it. Make sure these solutions can put their money where their mouth is.

4. Security

Cyber threats and compliance requirements are pushing more and more organizations toward tighter security protocols. Log data contains a lot of sensitive information about your business, and potentially about your customers as well. So the security of your logs is of paramount importance.

Any log management solution you opt for must support SSL encryption for data in transit and role-based access, allowing users to be defined as admins or users as well as suspended or deleted. Account admins must be able to manage and control user access, including the provisioning of new users with a defined access level.

Depending on your business, you should be looking at solutions that are compliant with relevant regulatory requirements. If you are in the healthcare industry for example, HIPAA compliance will be important. The PCI DSS compliance framework will be relevant if your business handles or processes credit card data. I recommend reviewing what compliance needs you might require in this article on our blog.

5. Advanced analytics

Most log management tools in the market provide the means to query the log data and analyze it with the help of different types of charts and graphs. Some solutions provide more robust visualization capabilities than others, which should be a consideration as well. However, if you are drowning in TBs of log data a day, this will only take you part of the way.

You can query your data to your heart’s delight, but your troubleshooting process can end up being extremely frustrating as you might not be quite sure what to query in the first place. In today’s modern IT environments, you need to think about log management tools that help you overcome the big data challenge with advanced analytics capabilities.

More and more solutions are offering machine learning and anomaly detection to streamline troubleshooting and improve monitoring by giving you the tools to detect issues early on. As you examine these solutions, be sure to examine these advanced capabilities to gauge their effectiveness. Can they help your use case? Do they really provide added value or are they simply jumping on the AI bandwagon?  

6. Cost-effectiveness

Log data is extremely verbose and noisy in nature. Even if you are an extremely log-driven organization and have implemented structured logging from your very first line of code, your logs will eventually grow in volume and exact a considerable cost from your business.

The pricing model most log management tools use is based on data volume and retention. This makes sense, especially if you consider the fact that these services are paying for storage on the cloud themselves. The problem is that reality is a bit more complex, and sometimes your use case might not fit this all-in-one, take it or leave it, pricing model. What if you want to retain some of the logs for a limited amount of time and retain other logs for a year or more? What happens if you exceed your quota, something that happens often with data bursts?

As you look at the different log management tools on the market, make sure you understand your use case and are able to define your needs clearly. Look for solutions that are flexible, offer granular pricing and that are willing to work with you to find a model that is tailored to suit your needs.

Summing it up

If you’re on the hunt for a log management solution, there’s something about how you are currently working with log data that isn’t working. It might be the process of manually grepping scores of log files located on multiple hosts, or maybe it’s extremely bad search performance with the log management tool you are currently using.  

At the end of the day, your next solution needs to empower you to solve the problems you have, not impede you by creating new problems. Despite the long list of alternative solutions that complicates the selection process, how you narrow down the list of options is simple.

Ask yourself these qualifying questions:

Will the log management tool make my life Easier? Is it easy to deploy, integrate with and use? Does it play nicely with my environment? Will migrating to it be a simple process? Can it support the scale I require?

Will the log management tool make me more Efficient? Can it help me save time and resources? Can it help me overcome the “needle in the haystack” challenge and identify issues more quickly.