Computing environments are constantly changing. Back when an on-premises server hogsted your work, your infrastructure and applications were easy to track. Now that you’re developing in the cloud, things are more challenging. You’re learning that each team within your organization uses a different monitoring tool. At this point, you may be wondering if it’s time to build your own monitoring solution with open source tools at its core that everyone can use.

There are several reasons why you would want to do this. You might want to present a “single pane of glass” to monitor the health of your environment, or perhaps you require certain features that are not found on any commercial product. Perhaps, you might show interest in contributing to the monitoring space’s open source community.

Open source monitoring offers a number of benefits, the most significant of which is the ability to start small and gradually scale. Open source software doesn’t punish you with licensing fees or additional expenses, making it remarkably cost-effective. Additionally, with open source monitoring solutions, security and support services are shared costs. The community helps deliver updates to the software as your team gets up to speed.

Flexibility is probably the greatest advantage of open source monitoring solutions. Open source components mesh well with numerous types of deployments. Because there’s a buffet of options to choose from, you can piece together multiple technologies to fit your needs. Adopting open technologies will ensure that your services remain connected and enable you to avoid walled garden monitoring complexities. Furthermore, these tools play well with outside services that rely on similar solutions. That flexibility will save you from developing costly proprietary technologies. 

In this blog post, you’ll learn about the costs of building your own monitoring solution using open source tools. We’ll dive into the components necessary by monitoring solutions and the libraries you can use to start as well as the approximate costs associated with them.

Building an In-House Monitoring Solution

When designing your infrastructure, you’ll need to consider these three tiers of your application:

1. The web tier
2. The application tier
3. The database tier

The costs of the components you use in each tier will drive your monitoring solution’s budget, so you’ll want to make these decisions carefully.

The Web Tier

The web tier is how people will access your new monitoring solution. You can run this as a virtual machine, as a container, or in a platform-as-a-service (PaaS) solution like AWS ElasticBeanstalk or Azure App Services. The more you can limit the scope of this tier, the more money you’ll save. Running your web tier on a PaaS solution like Azure App Services rather than on a virtual machine will substantially reduce your costs. 

Virtual machines that are powerful enough to run everything needed in this tier typically cost between $100-250 per month (this amount will vary based on your scale), whereas an Azure App Services instance will run you less than $80 for comparable performance.

The Application Tier

The application tier crunches all of the numbers behind the scenes for the web tier, handling and processing all of the events that you’re monitoring. Again, this tier can consist of a couple of virtual machines or even an API, but a wiser construction will include microservices for request processing.

The Database Tier

The database tier stores and retains data for use by the application and web tiers. As with the previous tiers, a VM with enough attached storage in the cloud is an expensive option compared with one of the PaaS solutions provided. For instance, Amazon RDS pricing is significantly more economical than running an EC2 instance with enough compute and storage power to replicate the same level of performance.

Open Source Monitoring Software

Given what “Free and Open Source Software” (FOSS) means, most decision makers in your organization should be able to get on board with that price tag! However, it’s important to note that “free and open source” isn’t the same thing as “zero cost.” You have to build the solution that fits your needs, provide support and ongoing maintenance yourself, and adhere to the shared responsibility security model. All of these activities come with costs.

Use of the popular monitoring solution Grafana provides a great example. Grafana is an open source dashboard library that can help you visualize all of the metrics an application ingests from its monitoring endpoints. Grafana helps reduce the cost of developing, maintaining, and supporting a completely custom dashboard by providing templates and tools that can quickly get you up to speed. The solution itself is functionally intuitive, plus a bustling community is available to help tackle unforeseen hiccups. And of course, time is worth something; time is money.

Delivery

Delivery creates another unexpected cost when you choose to build your own monitoring solution. In a traditional vendor interaction, the vendor provides the tools needed to deploy a solution on-prem or in the cloud. When working with your own monitoring solution, you have to figure out a way to put your monitoring solution into production yourself.

If your organization strives to be a DevOps organization, you must deliver a dynamic and responsive monitoring solution that can scale to your needs. Of course, the DevOps methodology comes with its own costs—a learning curve, additional tools and techniques, and potentially new team members with unique skill sets—all of which will need to be included in your budget.

Scaling

When dealing with scale, how you process that data will make or break your success. To buttress the process, you have to incorporate storage and buffering tools into your stack.

A buffering mechanism like messaging queue (MQ) Kafka will mitigate bursts of data during production incidents. The ELK Stack and Kafka together constitute an incredibly popular solution. Other options here that work with ELK and Logz.io include RabbitMQ, and Fluentd as part of the so-called EFK Stack (Fluentd in place of Logstash).

Other services can help with storage, like Prometheus (and its autonomus single server nodes), InfluxDB, or Graphite.

APIs vs. Event-Driven Architecture 

APIs are fairly easy to set up. However, they tightly couple all of your requests and responses. In the long run, this shortcut may end up being costly. Event-driven architecture allows you to loosely couple each request and response, making for a much more reusable and repurposable solution. While its starting cost is greater, event-driven architecture will help you scale more easily. An API requires you to follow a linear scale in your infrastructure to reach the scale level that you need. It goes without saying that you’ll need to beware of any potential technical debt you might build into a system. If you’re using an API, you’re already behind the curve. An event-driven architecture starts you off on the right foot.

Data Retention

Finally, you’ll be ingesting a large number of data points when you create your own monitoring solution. Determining what you’re going to do with those data points and how long you want to keep them will dramatically affect your storage costs.

You might want to get started by keeping only relevant data. Having up-to-the-minute monitoring data is very important, but data older than 90 days may not be relevant to understanding your production environment. You can simply remove data older than 90 days, or you can compress and cold store it in a lower-cost solution like Amazon Glacier in case you ever need it. With Logz.io, you have data retention for up to 18 months.

Summary

A custom-built, in-house monitoring solution can be beneficial to your organization. If planned, resourced, and implemented properly, your solution has the potential to be more efficient and effective than most of the monitoring solutions on the market today.

Remember, though, that the term “free” in open source software doesn’t necessarily mean “zero cost.” This blog post has discussed ways to be prepared for any budgetary challenges involved with building such a solution.

Using open source software for solutions isn’t limited to the monitoring process. There are many other applications that facilitate the use of open source solutions in your environment. If you want to read more, you can check out this article on how to manage costs in your ELK stack.

At the end of the day, you have to ask yourself if you have the time, resources, and dedication to pull off a company-specific monitoring solution. Do the prep work. Then, if any of these unforeseen costs surprise you, consider a managed solution. .