cloudwatch lambda shipper

The popularity and adoption of AWS Lambda have grown exponentially over the past few years and for a good reason. Aimed at making the life of developers simpler, Lambda allows developers to run their code without worrying about the operations aspect of executing it. Lambda will handle the infrastructure for you, so all you need to do is supply your code.

We see Lambda as a key player in answering our customer’s needs, and as a quick and simple way to ship their log data into Logz.io. A substantial amount of our users use AWS CloudWatch to collect and monitor logs, and as such, we are happy to introduce a Lambda shipper that collects data from a defined CloudWatch log group and ships its content into Logz.io.

Designed to make shipping your CloudWatch logs a simple task, the Logz.io Lambda shipper can reduce costs based on Lambda’s pay-per-use pricing.

To help you use this Lambda function, this article describes the necessary steps for creating a new function in Lambda, setting your CloudWatch trigger, and shipping your data into Logz.io.

Creating a new Lambda function

Sign in to your AWS account and open the AWS Lambda console. Click Create function, to create a new Lambda function.

Create Function

 

On the Create function page, you are presented with two options: Author from scratch or Blueprints. 

Select Author from scratch, and enter the following information:

  • Name –  enter a name for your new Lambda function. We suggest adding log type to your name.
  • Runtime – from the drop-down menu, select Python 2.7 as the function’s runtime.
  • Role – press on Create new role from template(s) , and under Existing role, select Basic Edge Lambda permissions

When done, hit the Create Function button in the bottom right corner of the page.

 

Uploading and configuring the Logz.io Lambda shipper

Our next step is to upload the Logz.io Lambda shipper, and configure it to ship data into your Logz.io account.

First, you will need to zip two files from our GitHub page. Download ‘lambda_function.py’ and ‘shipper.py’ locally and do:

Then, in the Function Code section, under Code entry type select Upload a .ZIP file, and upload logzio-cloudwatch-log-shipper.zip

You can see the code in the Function Code section, open the Code entry type menu, and select Edit code inline

In the Handler field, verify that the function name matches the .py file name and that the handler name matches the Lambda handler inside the .py file (should automatically match).

Function Handler

It’s now time to configure the function.  

Scroll down to the Environment variables section to set your Logz.io token, URL and log type:

  • TOKEN: Go to your Logz.io app and press the account button in the top right corner, you can find your token in account settings.
  • TYPE: Enter your log type you are going to use with this Lambda, notice that you should set a new Lambda for each log type you are using. Go here, to find all the log types we support.
  • URL: If you are in the EU region insert https://listener-eu.logz.io:8071 otherwise use https://listener.logz.io:8071. You can tell which region you are in by checking the login URL. If your environment says app.logz.io then you’re in the US, if it says app-eu.logz.io then you are in the EU.
  • FORMAT: If the ‘FORMAT’ JSON is set the lambda function will attempt to parse the message field as json and populate the event data with the parsed fields.

Environment Variables

The last thing to configure is some general execution settings.

These are found in the Basic Settings section. We recommend to start by setting memory to 512(MB) and a 1(MIN) timeout, and then subsequently adjusting these values based on trial and error and according to your Lambda usage.

Basic Settings

Defining the CloudWatch log event trigger

We are almost done! Our last step is to make sure that CloudWatch will trigger our function when a new log event occurs for a specific CloudWatch log group.

Under Add triggers at the top of the page, select the CloudWatch Log trigger.  

In the Configure triggers section, you will then be required to enter the log group from which the Logz.io Lambda collects the logs. You will also need to enter a filter name.

Remember, we set only one log group to trigger the function. If you want to use more than one log group just repeat the steps to trigger Lambda, but this time select a different log group.

demo

Click Add to add the trigger and Save at the top of the page to save all your configurations.

You can quickly verify our Lambda is triggered by an event from the chosen log group by going to CloudWatch console, press Logs and in the selected log group you should see the Lambda function name under Subscriptions.

create metric

That’s all, your Lambda function is now running and will automatically send your logs to Logz.io when a CloudWatch event triggers it.

Open the Discover tab in Kibana to view the data.

Kibana

Summing it up

Although it has its limitations, Lambda can be a powerful tool that will not only make code development easier, but also help organizations save a few bucks. If implemented correctly, Lambda can even replace some of your EC2 instances.

In this article, we briefly discussed the advantages of AWS Lambda and how well it integrates with AWS CloudWatch. Lambda is an easy-to-implement method for shipping AWS service logs from CloudWatch into Logz.io.

As always, we’d love to get your feedback. So if you have any suggestions or questions, feel free to contact us: info@logz.io.