How Intent Media Made Sense of 500 GB of Log Data Using Logz.io

About Intent Media

Intent Media uses predictive analytics to help travel brands maximize revenue per visitor on their websites. With sophisticated machine learning capabilities, the company displays ads or conversion-boosting user experiences to website visitors based on a unique understanding of where a given user is in their shopping journey.

The company operates in more than 40 countries with hundreds of airline, hotel, car rental, and online travel agency partners around the globe

The Need for Centralized Logging

Before using Logz.io, the development team at Intent Media had begun to explore the ELK Stack as a platform for centralized logging. The company’s architecture, consisting of multiple, distributed services running on EC2 and several moving into ECS, required a logging solution that would make it easy to access and analyze log data.

Being open source and driven by a large community, ELK seemed like the right solution to try. However, after some initial attempts to handle their own ELK deployment, Intent Media began to understand the ramifications of maintaining a growing stack. The management and maintenance overhead costs pushed the company to search for a robust, reliable and secure hosted ELK solution. Logz.io answered these requirements.

Setting Up the Integration with Logz.io

Since Intent Media already had an ELK Stack of their own installed, the learning curve and integration with Logz.io’s service was simple. All it involved was pointing the data sources to Logz.io’s listeners.

In practical terms, this involved using Filebeat, a reliable and lightweight log shipper, to track log files and send them to Logz.io with SSL encryption.

Once the data was shipped, the support team at Logz.io helped ensure the data was parsed correctly. One example was involved parsing multiline Java exceptions correctly so they were logged as one message line. This step was crucial for Intent Media as it ensured more efficient log analysis and visualization.

Seeing the Forest for the Trees

When first getting started with Logz.io, the company was shipping over 500 GB of data a day. This was too large a data set to handle, but the enhanced visibility into the data gained by using Logz.io allowed the company to downsize that number to 5 GB a day, ultimately enabling a more liberal logging strategy.

Logz.io is now used at Intent Media to analyze multiple log types, including Apache and Nginx web server logs and Java application logs. Using visualizations, the team is able to correlate events across these data sources and identify spikes in errors:

Using Cognitive Insights — Logz.io’s machine learning technology, IntentMedia is able to decrease the signal-noise ratio and identify specific issues that would otherwise have gone unnoticed.

Monitoring Exceptions and Errors

Intent Media closely tracks requests sent by their customers to the various services. Logz.io is used primarily to monitor, and subsequently handle, exceptions and errors resulting from these requests. Should a specific value in a request fail validation, for example, exceptions are triggered and then monitored in a Kibana dashboard.

Alerts are added in specific cases, so the team is notified on time when an error is taking place in real time. For example, an alert was configured on an odd and infrequent database error. Logz.io’s integration with Slack is used so alerts get sent to the relevant organization members.

When new code is deployed, Logz.io’s Live Tail is used to see a raw feed of the logs generated. Using regex-based filters, the data is filtered so the team can focus on specific logs and the pieces of data that interest them the most.

The End Result

Over 30 developers across multiple development teams at Intent Media are now using Logz.io on a daily basis, shipping a total of approximately 5 GB of data daily. Different teams have access to separate sub-accounts, allowing them to test logging strategies in a siloed environment.

Using Logz.io for centralized logging, Intent Media has managed to not only pinpoint errors in the application more accurately and in real-time, but build a more robust and reliable centralized logging system.

Rob Park, VP of Engineering at Intent Media sums it up: “Logz.io enabled us to get our logging practices under control. We had dozens of ad servers all generating a ton of noise in their logs. And worse, you had to SSH to the box to see anything in there with tail or grep. Now we don’t need to SSH, we’ve cut down the noise by about 99%, and you can

actually discover things that you want to know.”

Turn machine data into actionable insights with ELK as a Service