Logz.io’s New Integration with AWS Kinesis: Send Metric Data Without a Single Line of Code

After creating your Logz.io account, the first step for onboarding is to send you log, metric, and trace data. Logz.io makes this flexible – allowing for multiple ways to get data into your Logz.io account depending on your use case and technology stack. 

Today, we’re excited to announce another easy and fast way to get AWS metric data into Logz.io: by setting up a CloudWatch metrics stream and a Kinesis Firehose. This integration is currently available in the eu-central-1 & us-east-1 regions.

In minutes, AWS users can begin streaming metric data to Logz.io to monitor signals like CPU, memory, latency, and others across their resources and services. 

From there, they can easily build interactive dashboards (or use prebuilt dashboards for specific resources), set alerts, and correlate their metrics with associated logs and traces within Logz.io.

A quick review of sending metric data to Logz.io from AWS CloudWatch

AWS Cloudwatch makes it fast and easy to collect AWS metrics, but many prefer to analyze that data in a different interface – including many Logz.io customers.

Before this new integration, the primary way to send CloudWatch metrics to Logz.io was with the Prometheus Cloudwatch exporter. Logz.io customers can use our Infrastructure Monitoring that delivers Prometheus-as-a-service  – which can easily ingest metrics collected by Prometheus and store them cost effectively for long periods of time. 

From there, customers can monitor their metrics and correlate them with logs and traces. The Cloudwatch exporter is a popular way to get those metrics to Logz.io.

Now, as an alternative to those not already using Prometheus, customers can use a new integration with AWS Kinesis, which you can read all about in the following section. This saves the time that would otherwise be needed to configure the Prometheus Cloudwatch exporter.

A quick review of sending metric data to Logz.io from AWS CloudWatch

This new approach for sending metric data is great for those looking for a fast and easy way to start sending their metric data – that don’t mind a bit of extra costs. 

This integration can be set up in minutes without a single line of code and enables near real-time analytics – as you’ll see – but it streams lots of data to Logz.io using two AWS services: Cloudwatch and Kinesis. In future versions of this integration, it will be easier to narrow down the metrics to reduce costs.

Let’s begin a step-by-step tutorial for setting up the integration.

Start by going to Amazon Kinesis and creating a delivery stream with ‘Data Firehose’.

Next you’ll need to select a source and destination: select ‘Direct Put’ as the source and ‘HTTP endpoint’ the destination.

Hit ‘Create delivery stream’.

Now, it’s time to configure the destination settings. Give the HTTP endpoint a name – which should probably include ‘logz.io’. 

Set the HTTP endpoint url as: listener-aws-metrics-stream-us.logz.io

Or, if you’re sending data to a european region, set it to: listener-aws-metrics-stream-eu.logz.io

Next add your Logz.io token to the access key – find this by going to ‘Settings’ in Logz.io and selecting ‘Manage Tokens’ (you need to be a Logz.io admin to access this).

Keep ‘Content encoding’ on ‘Disabled’ and set your desired ‘Retry Duration’. Configure ‘Buffer hints’ to your preferences.

Next is to determine your Backup settings in case something goes wrong and you need to recover the data – in most cases it’s only necessary to back up the failed data. Simply choose an S3 bucket or create a new one to store data if it doesn’t make it to Logz.io. Then, select ‘Create a delivery stream’.

Now it’s time to connect CloudWatch to our Kinesis Delivery Stream. Navigate to Cloudwatch and select ‘Streams’ in the Metrics menu. Select ‘Create metrics stream’.

Next, you can either select to send all your CloudWatch metrics to Logz.io, or only metrics from specified namespaces. In this case, we chose EC2, RDS, Lambda, and ELB.

Under ‘Configuration’, choose the ‘Select an existing Firehose owned by your account’ option and choose the Firehose you just configured. 

If you’d like, you can choose additional statistics in the ‘Select Statistics’ box, which usually provides helpful metrics in terms of percentiles to monitor like latency metrics (i.e. what are my services with the highest average latency?). This will end up adding to your costs, so choose wisely. 

Lastly, give your metric stream a name and hit ‘Create metric stream’. 

That’s it! Without writing a single line of code, we configured an integration with AWS and Logz.io that enables fast and easy infrastructure monitoring through CloudWatch data collection.

Your metrics will be stored in Logz.io for 18 months out of the box – without requiring any overhead management. You can also begin to build dashboards and alerts to begin monitoring – like this EC2 monitoring dashboard below.

To learn more about building metrics dashboards in Logz.io – check out this video.

Want to try it yourself? Create a Logz.io account today, navigate to our infrastructure monitoring product, and start streaming metric data to Logz.io to start monitoring.