How Droplr Monitors Millions of Users with Logz.io

About Droplr

Droplr is a Bend, Oregon and Wroclaw, Poland – based company whose mission is to help people succeed at work by taking away needless complexity, streamlining communication and promoting collaboration. Started by Levi Nunnink and Josh Bryant, and run by Gray Skinner, Droplr develops remote collaboration solutions recognized by international users.

Droplr offers a number of remote work enabling functionalities. The tool is most popular, however, for how simple it makes sharing information and exchanging feedback. It’s used by client-facing and remote teams, but not only. It’s a real game changer for anybody who needs a tool that will help them share content on the spot and with minimum effort, and that integrates seamlessly with other tools.

The Serverless Revolution

While serverless is a relatively new concept, adoption is well underway. Still, many companies are shy about exploring and implementing a serverless architecture, not to mention publishing their successful migration story online.

Droplr’s engineering team shared their story about moving to serverless, covering all the main steps of the process — planning, testing, security and monitoring. Logz.io plays an integral part of the multi-layered monitoring approach used by Droplr, and is used as the primary logging in the new architecture.

Searching for a Logging Solution

Droplr’s services are now built almost entirely upon a serverless architecture. Lambda functions handle both the processing of background jobs and all the public-facing microservices. The latter are now HTTP-invoked Lambda functions. Requests and responses are passed via the AWS API gateway, which in turn is placed behind CloudFront.

CloudFront enabled Droplr to monitor all incoming and outgoing HTTP traffic using access logs, so the question was what log analysis tool to use to use. Droplr began exploring different logging solutions. AWS lacked the analysis features Droplr required, while other logging solutions did not integrate with AWS to a satisfying degree.

Because of previous experience with the ELK Stack, the team preferred an ELK-based solution that did not require investing resources in maintenance. Offering ELK as a scalable and secure service, Logz.io was evaluated and found to suit Droplr’s requirements and architecture. As Antoni Orfin, Solutions Architect at Droplr, puts it: “Logz.io fit our serverless — and “pay for what you use” — concept perfectly.”

Onboarding with Logz.io

Migrating to Logz.io was seamless and took less than a day’s work. CloudFront logs are shipped to an S3 bucket. From there, they are pulled using Logz.io’s built-in support for AWS.

Since Droplr’s CloudFront logs contain some custom fields used for measuring bandwidth abuse, some initial parsing work was required. This was performed with the assistance of the support team at Logz.io and all processing is now performed on Logz.io’s end. CloudFront logs are parsed automatically to allow efficient analysis and visualization.

Monitoring File Downloads

More than half a million users download files from Droplr every day. It goes without saying that these downloads need to be carefully monitored. Limitations on file downloads need to be enforced and abusive behavior identified on time.

Droplr analyzes and monitors file downloads using the CloudFront access logs. These logs contain typical HTTP-related fields (e.g. status, method, IP, result), and as mentioned above, have also been customized to include some Droplr-specific metrics. Kibana dashboards have been developed on top of the logs to give the team a good indication on when the system is being used as expected.

Triggering Lambdas with Logz.io Alerts

Droplr has also developed a way to automatically block a user once anomalous behavior is identified in the CloudFront logs.

This method is based on two components that integrate Logz.io with Droplr’s serverless architecture. On the Logz.io side, an alert has been configured based on specific defined thresholds. Once triggered, this alert calls an http endpoint which is actually a Lambda function. This function is tasked with blocking the user.

End result

The entire IT team at Droplr use Logz.io as their primary log analysis tool. Access is given to the development team as well upon request, for visibility into how the services are performing in production. Approximately 5 GB of CloudFront logs are shipped to Logz.io a day, helping Droplr not only monitor HTTP traffic but also save the company money.

Antoni Orfin, Solutions Architect at Droplr, sums it up: “After a few months of using Logz.io, we were amazed at how great it fits our serverless architecture. Using Logz.io has helped us to effectively analyze abusive usage of Droplr and has ultimately saved us tens of thousands of dollars a month.”

Turn machine data into actionable insights with ELK as a Service