How Mirakl Monitors Hundreds of APIs with Logz.io

Mirakl is the global leader in online marketplace platforms and the provider of Mirakl Marketplace Platform, a SaaS-based, secure and highly scalable platform that allows brands to host transactions between buyers and sellers of products.

Mirakl Marketplace Platform fits seamlessly with a retailer’s brand and existing presence to offer increased product choice and profits without inventory and logistics costs and restraints. Modular and adaptable, it is compatible with existing ecommerce solutions.

Based in Paris, Boston, London and Munich, Mirakl was founded in 201 2 and lists some of the world’s largest retailers as customers, including Galeries Lafayette, Best Buy, Conde Nast, Carrefour, Game and Darty.

Battling with ELK

Mirakl was looking for a centralized logging solution to monitor how customers were using their APIs. An ELK Stack was deployed to answer this need, but after some time, it became extremely hard to handle.

Mirakl waged war with ELK for a few months. While the stack was usable, developers encountered frequent crashes and found that to ensure it was always available, they were spending both an increasing amount of time maintaining the stack and an increasing amount of resources on the AWS infrastructure on which it was deployed. Scaling ELK to cater for the growing amount of data became a major challenge.

Looking for alternatives

Nagi Letaifa, Tech Lead at Mirakl, felt that the company was better off looking for a different solution that would minimize hosting costs as well as enable the development team to focus on improving the product rather than maintaining the underlying logging infrastructure.

After all, as Nagi points out, “In a young and fast-growing startup, you need to focus on your business and keep adding value to your customer as fast as your can — managing tools should never get in the way.”

Mirakl began looking for an alternative that was robust, secure, easy to use, fast, and always available. The solution that complied with these requirements was Logz.io.

Transitioning to Logz.io

It was important for Mirakl that the testing and transition processes for adopting any alternative solution be fast and simple. The move to Logz.io complied with these needs and was seamless.

Testing and moving to Logz.io was smooth, and it took fifteen minutes to see the first logs appear in Logz.io. All it involved was pointing the Logstash forwarders to ship to the Logz.io listeners instead of our Elasticsearch instances. We also received help in migrating our Kibana visualizations from version 3 to version 4.

Nagi Letaifa, Tech Lead

Monitoring APIs

Mirakl’s offering is heavily based on the hundreds of APIs that customers use to integrate with the product’s various services. These APIs are serviced via Apache Tomcat web application servers. By monitoring Tomcat access logs, the Mirakl team can accurately analyze how users are consuming these APIs.

For example, Mirakl developers can tell whether a user is entering the right parameters when passing an API call. Or, they can analyze the response time and status for API calls to see if there are any errors taking place.

A dedicated dashboard was created for each API, giving the team at Mirakl full visibility into how users are consuming API.

Implementing Alerting for APIs

Monitoring is good, but being proactive about it is even nicer. Mirakl uses the alerting mechanism built into Logz.io to complement monitoring with proactive actions.

For example, an alert is configured for each user to notify Mirakl when specific APIs are not being called, signifying that orders are not being made in the customer’s marketplace.

Nagi points out that this feature, together with user control and SSO, were major bonuses because “in the open source ELK, these features are not included out of the box and mostly require additional costs.”

The End Result

The entire development and support teams at Mirakl use Logz.io to analyze and monitor approximately 50 GB of data shipped daily.

We ran ELK internally for quite some time with all the overhead of the maintenance. Logz.io enables us to focus on our growing business and know that our logging solution is always there. From my point of view, it’s perfect.

Nagi Letaifa, Tech Lead