In this post we will discuss some key considerations and strategies to monitor your AWS Lambda functions. This will include: which Lambda metrics you’ll want to monitor, how to collect AWS Lambda metrics with Prometheus and Logz.io, how to create a monitoring dashboard with alerts, and how to search and visualize your metrics.
Key AWS Lambda Metrics to Monitor
The following section briefly highlights key Lambda metrics to monitor to effectively understand their behavior and identify problems. There are three key categories of AWS Lambda metrics:
1. Invocation metrics
- Invocations – The number of times your function code is executed, including successful executions and executions that result in a function error.
- Errors – The number of invocations that result in a function error.
- DeadLetterErrors – For asynchronous invocation, the number of times Lambda attempts to send an event to a dead-letter queue but fails.
- Throttles – The number of invocation requests that are throttled, which when a Lambda intentionally rejects a request and does not execute the function.
- ProvisionedConcurrencyInvocations – The number of times your function code is executed on provisioned concurrency.
- ProvisionedConcurrencySpilloverInvocations – The number of times your function code is executed on standard concurrency when all provisioned concurrency is in use.
2. Performance metrics
- Duration – The amount of time that your function code spends processing an event.
- IteratorAge – For event source mappings that read from streams, the age of the last record in the event.
3. Concurrency metrics
- ConcurrentExecutions – The number of function instances that are processing events.
- ProvisionedConcurrentExecutions – The number of function instances that are processing events on provisioned concurrency.
- ProvisionedConcurrencyUtilization – For a version or alias, the value of ProvisionedConcurrentExecutions divided by the total amount of provisioned concurrency allocated.
For more information on these metrics, check out the detailed documentation in the AWS docs.
Now that we know which metrics to monitor, let’s set up some dashboards to monitor them. While Lambda metrics are automatically published to Cloudwatch, most engineering teams prefer other solutions to collect and analyze their metrics.
Let’s check out how we can use Prometheus and Logz.io to collect metrics at scale so they can be searched and analyzed.
Collecting and Monitoring Lambda Metrics with Prometheus and Logz.io
Now comes the fun part. To put the metrics described above into action, we’ll need to collect the metric data and build dashboards to monitor them. We’ll do these things with Prometheus and Logz.io.
But metrics don’t show the full picture of what’s happening with your lambdas. Once we identify problems in metrics monitoring dashboards, we’ll also need log data to troubleshoot the problems. We will also show how to correlate your metrics with the associated logs on Logz.io.
Let’s get started.
Collecting and Analyzing Metrics with Prometheus and Logz.io
Before we get to set up the monitoring dashboards, we’ll need to get the metric data into our monitoring solutions.
In this section, let’s pull Lambda metrics from CloudWatch with Prometheus. And then if you’d like to try it, you can follow along to forward those metrics to Logz.io’s Prometheus-as-a-service for scalable storage and analysis.
If you’re NOT interested in using Prometheus to collect your Lambda metrics, check out these directions for sending your metrics straight from CloudWatch to Logz.io with the otel/opentelemetry-collector.
How to Pull Lambda Metrics in Cloudwatch to Your Prometheus Server
Assuming you already have Prometheus installed, let’s start pulling Lambda metrics from Cloudwatch (where they are automatically published) to a Prometheus server so we can monitor them.
To scrape the metrics, we will use yet-another-cloudwatch-exporter. Capable of exporting metrics for a lot of AWS services, we’ll stick to exporting AWS Lambda metrics with it for now.
Here is an example of the
yet-another-cloudwatch-exporter config file (in yaml):
discovery: jobs: - regions: - us-east-1 type: lambda enableMetricData: true metrics: - name: Duration statistics: [ Sum, Maximum, Minimum, Average ] period: 300 length: 3600 - name: Invocations statistics: [ Sum ] period: 300 length: 3600 - name: Errors statistics: [ Sum ] period: 300 length: 3600 - name: Throttles statistics: [ Sum ] period: 300 length: 3600 - name: DeadLetterErrors statistics: [ Sum ] period: 300 length: 3600 - name: DestinationDeliveryFailures statistics: [ Sum ] period: 300 length: 3600 - name: ProvisionedConcurrencyInvocations statistics: [ Sum ] period: 300 length: 3600 - name: ProvisionedConcurrencySpilloverInvocations statistics: [ Sum ] period: 300 length: 3600 - name: IteratorAge statistics: [ Average, Maximum ] period: 300 length: 3600 - name: ConcurrentExecutions statistics: [ Sum ] period: 300 length: 3600 - name: ProvisionedConcurrentExecutions statistics: [ Sum ] period: 300 length: 3600 - name: ProvisionedConcurrencyUtilization statistics: - Maximum period: 300 length: 3600 - name: UnreservedConcurrentExecutions statistics: [ Sum ] period: 300 length: 3600
Once the right credentials and configurations are all set you can run the cloud watch metrics exporter locally like below:
docker run --rm -v $(PWD)/credentials:/exporter/.aws/credentials -v $(PWD)/yace.yml:/tmp/config.yml -p 5000:5000 --name yace quay.io/invisionag/yet-another-cloudwatch-exporter
Then you will be able to see metrics in port 5000 as shown below. You can see that
dev-generate-random-errors Lambda metrics are being reported. Please find the details on running and configuring yet-another-cloudwatch-exporter in their official github repo.
Using PromQL Queries
Now that Prometheus is configured successfully, we can query AWS/Lambda our metrics using the PromQL query language.
The diagram below shows the query result of our
aws_lambda_errors_sum of the
dev-generate-random-errors Lambda metrics.
Prometheus is a widely popular way to collect metrics from Lambda functions. But oftentimes, Grafana will analyze these metrics due to Prometheus’s limited visualization capabilities. Learn more about how these tools work together here.
This example shows a small volume of metric data. But Prometheus servers and roll up servers can be a lot to manage, upgrade, and maintain at large scales. Plus, you’ll need a totally separate solution for logging, which can slow down your monitoring and troubleshooting workflows and can lead to tool sprawl.
These are the reasons Logz.io built Prometheus-as-a-service, which collects all your Prometheus metrics on a scalable SaaS platform for long-term storage. Plus, it unifies your metrics alongside your log analytics, which consolidates your and makes it faster to monitor and troubleshoot your lambdas.
Forwarding Prometheus Metrics to Logz.io for Scalable Storage
Logz.io offers a scalable Prometheus-as-a-service solution called Infrastructure Monitoring.
All we have to do to forward our Prometheus metrics to Logz.io Infrastructure Monitoring is add the
remote_write url to our
prometheus.yml config file. This will automatically forward all the metrics collected by Prometheus to a remote destination – in this case, Logz.io.
Here is an example
prometheus.yml file. Note that you’ll need to add the account token – which you can find in your Logz.io account page – to the file:
global: scrape_interval: 60s evaluation_interval: 60s external_labels: p8s_logzio_name: LambdaMetrics scrape_configs: - job_name: 'cloudwatch_yet-another-exporter' scrape_interval: 30s static_configs: - targets: [<exporter-host-ip>:5000] - job_name: 'prometheus' scrape_interval: 5s static_configs: - targets: ['localhost:9090'] remote_write: - url: https://listener.logz.io:8053 bearer_token: <Metrics Account Token> remote_timeout: 30s queue_config: batch_send_deadline: 5s max_shards: 10 min_shards: 1 max_samples_per_send: 500 capacity: 10000
Now, all your metrics automatically will both scrape from CloudWatch and automatically forward to Logz.io for long term storage and analysis. In other words, you can keep your existing Prometheus servers in place, without needing to manage the data storage at scale. Plus, your metrics can be unified with log and trace analytics:
Creating AWS Lambda Metrics Monitoring Dashboards
Now we can set up monitoring dashboards in Logz.io that monitors our
generate-random-errors Lambda. Creating new monitoring visualizations (called panels) is easy. Hit the ‘Add panel’ button, select the data source, and add the PromQL query to visualize the desired metrics:
The query above monitors all our errors from the function. After hitting save, you can see the new visual in the top right panel below.
We also added panels that monitor latency (which measures the average and max durations of all the invocations), throttles, invocations, and global concurrent executions – a good start on the list of metrics to monitor in the ‘Key AWS Lambda metrics to monitor’ section.
But nobody wants to stare at a dashboard all day to see if something goes wrong, so we can set alerts to do that for us.
Setting Up Logz.io Alerts
The alert below triggers automatically when the Lambda error count is greater than 17 (random count) on every minute evaluated for the last 2 minutes. You can set the threshold by dragging and dropping the highlighted heart up and down the y axis.
Then, we can configure the alert to notify us when it triggers with endpoints like Slack, PagerDuty, VictorOps, ServiceNow, OpsGenie, Gmail, and many other alerting endpoints:
In the real world, after being alerted of spiking Lambda metrics, we’d immediately want to investigate the related logs to debug the problem. This would normally require accessing a separate system to begin querying the logs.
But Logz.io also offers a Log Management solution, so we can correlate our error metrics with the associated logs to immediately bring up the logs that explain the spiking metrics by hitting this link we’ve added:
Hitting this link brings us straight to the relevant logs so we can debug any problems in our lambdas.
But how did we get our Lambda logs into Logz.io you may ask? Check out our Lambda logging blog to learn more.
Your Monitoring Game Plan
Go into your Lambda monitoring journey with a game plan. First, review the metrics we covered in the ‘Key AWS Lambda Metrics To Monitor’ section to decide exactly what you want to monitor.
Once you know which metrics you want to monitor, the next question becomes how you’re going to collect and analyze the data. Most of today’s ‘monitor-ers’ choose Prometheus. It’s easy to set up and integrate with modern stacks to collect your metrics. But there are a few considerations.
The first is scale and time. The data we used in this example was very small-scale. More realistic production environments require tools to effectively ingest and store huge quantities of data.
Using Prometheus for this is doable – some of the best engineering teams in the world use it. The question for you is whether or not your team has the time and resources to manage it.
The next consideration is whether you want your metrics and other monitoring data (logs and/or traces) all in one place, or if you’re okay with having them spread out.
Logz.io is a good alternative for teams that want somebody else to manage these open source tools and/or because they want all of their monitoring data together.
If that’s the case for you, try Logz.io for free here. Otherwise, Prometheus is your best bet.