A Guide to AWS Lambda Logging

A Guide to AWS Lambda Logging with Logz.io

In this post, we will discuss some key considerations and strategies to collect and analyze your AWS Lambda logs. This will include 1) what to know about logging Lambda functions, 2) how to ship log data to a centralized logging solution, and 3) how to search and visualize log data on monitoring dashboards.

A Few Things to Know about AWS Lambda Behavior

Lambda logging is essential because once lambdas are triggered, all of the underlying compute resources are automatically managed for you. 

This means you won’t have control over the runtime environment where you log into the servers to see the process logs, cpu usage, memory footprints, etc. Rather, it’s a black box service that executes the code, then pumps out the execution logs and metrics into CloudWatch. 

Metrics are great for telling you that a problem might be occurring, but logs are needed to tell you exactly what is causing the problem.

Lambda functions are not designed for long running jobs – they run within a configured timeout (max 15 minutes). Functions could time out for lots of reasons – inadequate timeout settings, low memory, cpu configurations, high network latency etc. Because of that, these are all signals we’ll want to monitor so we can troubleshoot any problems as they arise. 

Also, Lambda functions can get throttled whenever there are lots of parallel events. This can easily go unnoticed without the proper monitoring. 

Logs can capture all of this information. But all lambdas run independently from each other; they have their own logs and operational footprints like memory usage, execution time, billed duration, and errors. As you add more lambdas to your environment, it can be difficult to track the data across your functions and the consequences of that data. 

For example, below is a scenario where a Lambda is used to invite the user, index them in Elasticsearch, and record the user’s information in a warehouse. 

Multi-Lambda Architecture Example
Multi-Lambda Architecture Example

Let’s say the ‘invite lambda’ started to timeout – the issue would then propagate to the other two lambdas down the chain, which would show up in the logs and metrics. But if the logs and metrics aren’t visualizing alongside each other and correlated, then the ‘invite lambda’ issue could easily be observed as multiple, separate issues in the other lambdas.

These are some of the dynamics that are good to know when logging lambdas. Now, let’s dive into how to actually do it.

AWS Lambda Logging with Logz.io

With Logz.io, you can centralize all of your Lambda logs for storage and analysis on a managed SaaS platform. It’s based on the ELK Stack, so users can analyze their logs in the most familiar open source logging tools, without managing it themselves. You set up an account for free here

Let’s go through the steps to collect and analyze your Lambda logs.

When a Lambda function completes an execution, it publishes the logs to Amazon CloudWatch. So we’re going to set up a Lambda specifically to forward these logs to Logz.io.

We’ll set up a cloudwatch-logs-shipper-lambda, a lambda that listens to changes in the CloudWatch log groups that are collecting logs from our functions. After a certain threshold, it forwards those logs to Logz.io. 

Find the details here for setting it up.

In this example, our Lambda called generate-random-errors produces random warning logs, error logs, and timeout logs based on the given configuration. The Lambda function is given the following configuration which tells the function to generate 30 timeouts, 135 logs, 63 errors and 32 warnings:

{
"timeout": 30,
"log": 135,
"error": 63,
"warn": 32
}
 Lambda setup to generate random errors,logs, metrics for monitoring
Lambda setup to generate random errors,logs, metrics for monitoring

To review the architecture above, the generate-random-errors function is triggered, then automatically publishes the logs in CloudWatch. From there, the cloudwatch-log-shipper-lambda forwards those logs to Logz.io. 

Once you’ve created a Logz.io account, you’ll be able to watch your logs stream into Logz.io in real time with Live Tail.

Live tailing of logs on Logz.io
Live tailing of logs on Logz.io

Analyzing Logs in Logz.io

And once you’re ready to search and analyze the logs, the Discovery tab in Kibana offers search filters and a drag and drop time chart to query the log data.

Logz.io Discover
Logz.io Discover

Below, we pulled up all the ‘ERROR’ and ‘TIMEOUT’ Lambda logs with a free text search to immediately surface logs that could indicate problems. 

Free text search is not recommended in large volumes of data. Instead, hit + Add filter (underneath the red box in the image below), select the field you want to search for, and select a field value. You’ll want to get familiar with the logs themselves so you know what to search for.

Of course, in this case, our Lambda function is producing random logs, so there is no problem here.

Create Alerts with Logz.io
Create Alerts with Logz.io

Once you have queried the logs, you can create alerts from the same query as highlighted above, so that we’ll be alerted when a certain number of ‘ERROR’ and ‘TIMEOUT’ logs occur in a certain timeframe. Alerting is an example of a feature we’ve built on top of Kibana.

The alerts can be routed to any endpoint of your choice: email, PagerDuty, VictorOps, Slack, etc. In the case below, we’re using Slack. 

Lambda function alerts in #lambda-alerts Channel on Slack
Lambda function alerts in the #lambda-alerts channel on Slack

In addition to alerts, Logz.io has added other features on top of Kibana. This includes Log Patterns which clusters all your logs into similar groups, so you can quickly skim all the contents of your log data. The Exceptions tab automatically surfaces critical exceptions in your log data. We aren’t focusing on them in this tutorial, but it would be worth it to check out more details in these videos about Log Patterns and Application Insights.

Creating Lambda Monitoring Visualizations and Dashboards

We can also create various Kibana visuals like a bar graph chart, pie chart, line chart or many other visualization options. We can combine them to a dashboard to give a full context of the Lambda logs as shown below. Now we can zoom in to any time period to see the amount of errors, logs, and warnings, which can quickly show us how the data changed over time.

Creating a bar visualization in Logz.io
Creating a bar chart in Logz.io

The bar chart diagram above shows the visual of the Lambda errors being plotted as the bar chart against the timestamp in the x-axis. It’s a simple and effective way to see patterns of the errors over time. 

If we were to see a sudden spike in this chart, it would indicate some kind of problem in our lambdas.

The diagram below is the Kibana dashboard where we can glue related visualization and queried/filtered logs in a single dashboard. As you can see, we created more visualization like the error chart to include successful invocations, timeouts, and warnings. 

Dashboards are extremely helpful to get a glimpse of many indicators in a single view. 

Creating a dashboard of bar visuals in Logz.io
Creating a dashboard of bar visuals in Logz.io

It is good practice to combine all the related Lambda function visuals like errors count, success count, warning, timeouts, and plain text logs as shown above. Doing this will help you visually correlate all the information from logs to get much more context.

Building a Game Plan

Go into your Lambda monitoring journey with a game plan. First review the metrics we covered in the ‘Key AWS Lambda metrics to monitor’ section to decide exactly what you want to monitor.

Once you know which metrics you want to monitor, the next question becomes how you’re going to collect and analyze the data. Prometheus and the ELK Stack are the most popular monitoring tools for metric and log analytics respectively. 

The data we used in this example was very small scale. More realistic production environments require tools to effectively ingest and store huge quantities of data. Using ELK and Prometheus for this is doable – some of the best engineering teams in the world use these tools. The question for you is whether your team has the time and resources to manage them. 

Logz.io is a good alternative for teams that want somebody else to manage these open source tools…and/or because they want to unify log and metric analytics together…and/or because they like the features we’ve built on top of Prometheus and the ELK Stack.

If you fall under any of those categories, try Logz.io for free here.

The Complete Logging Solution – Correlations, Patterns, Insights & Parsing

Internal

Organize Your Kubernetes Logs On One Unified SaaS Platform

Learn More
× scaleup-logo Join our annual user conference Register Now