Serverless Monitoring: Logs, Metrics & Traces with AWS Lambda

Blog / How To

#AWS

#Cloud Monitoring

#CloudWatch

#Docker

#Docker Monitoring

#Instrumentation

#Lambda

#Serverless

By: Mike Elsmore

I’ve been primarily a Javascript developer for a long time now, it’s been my go-to language for the better part of a decade now, I even wrote a post on how to implement observability in a traditional Node.js application. Now, on top of hacking around in JS, I also love building things for AWS Lambda which is AWS’s option for Functions-as-a-Service (a.k.a., serverless computing).

Now, with the example from that tutorial we had access to the host machine to get/send serverless logs, plus gather metric data but that’s not the case with Lambda. Today’s we’re going to cover serverless logging and serverless monitoring with AWS Lambda and CloudWatch.

What’s different with Lambda?

So most developers shipping server-side Javascript applications are shipping stacks on top of virtual machine or container, much like the previous demo. However, with Functions-as-a-Service you lose access to the OS and machine itself; you only ship and use the application code you give to the vendor, in this case, AWS. What this means is you can’t read log files or access the system/OS level for data. Thus, you have to access the logs and metrics in a different way.

What did you build the Lambdas in?

Personally, I write my functions in Javascript. When I’m writing functions for Lambda — or it’s alternatives really — I like to use the Serverless framework. It provides a common boilerplate, configuration process, and deployment tooling for those serverless functions

#Pay attention to the capitalization of "serverless" — lower-case "s" refers to serverless computing in general, while upper-case/capitalized "Serverless" refers to the specific framework of that name.

How to Build Out Serverless Logs, Metrics & Traces

Let’s walk through serverless monitoring — the logs, metrics, and traces — in order to work out how to manage all three.

Serverless Logs

Just like in the Node.js tutorial, we’re going to use Winston to take advantage of its structure and ability to send logs to different receivers. So, we’ll add the packages, npm i -S winston winston-logzio. This gives us the default Winston to use and our Logz.io Transporter.

Then, we need to create the logger object:

#javascript
const winston = require(‘winston’);
const LogzioWinstonTransport = require('winston-logzio');
const logzioWinstonTransport = new LogzioWinstonTransport({
  name: 'winston_logzio',
  token: process.env.SHIPPING_TOKEN,
  host: `${process.env.LISTENER_URI}:5015`,
});
const logger = winston.createLogger({
  transports: [
    new winston.transports.Console(),
    logzioWinstonTransport,
  ]
});

Then, to actually generate the serverless logs, we just need to do something like:

#js
logger.info("127.0.0.1 - just a little information about home");
logger.warn("127.0.0.1 - a warning about home");
logger.error("127.0.0.1 - home is throwing an error");
logger.debug("127.0.0.1 - helps find bugs at home");

This undertakes the great task of sending serverless logs to Logz.io directly through the Winston Transporter. Just to note, this is very much the same with any Node.js application. But what if you don’t want the overhead of sending the data whilst the code is running, what do you do then?

At this point, you have to capture the logs from AWS CloudWatch instead. How do we do this? This is where it gets a little funny. We use a Lambda to capture the logged data from the Lambdas that are sent to CloudWatch.

To facilitate this, we have got an AWS Lambda coded and ready for you to deploy, you can see the full CloudWatch guide here.

I opted for the “Automated CloudFormation deployment” method since it does most of the wiring of the Lambda for you.

The only thing you need to do after script deployment is add the event trigger for CloudWatch Logs, explained here.

So, we’re capturing and structuring the log data within code, and we have the choice to send directly from Lambda or capture from CloudWatch. So, what’s next?

Metrics

Unlike the Docker version, I was using previously to gather the metric data from AWS for the Lambdas is going to a bit more hard work. Here at Logz.io, we’ve provided a Docker image that connects and collects metrics data from CloudWatch, then forwards it to our platform.

In theory, you could run this anywhere you can run Docker. But, it’s always a better idea to have it inside your AWS infrastructure. That way you can configure the security settings you require.

So how do we ship this as part of our Lambda code?

As we’re using the Serverless framework, we can naturally extend with CloudFormation Resources to add the elements needed.

Here is the complete resource list, for which I’ll give you a quick run through underneath.

#yaml
#you can add CloudFormation resource templates here
resources:
  Resources:
    Cluster:
      Type: AWS::ECS::Cluster
      Properties:
        ClusterName: deployment-example-cluster
    LogGroup:
      Type: AWS::Logs::LogGroup
      Properties:
        LogGroupName: deployment-example-log-group
    ExecutionRole:
      Type: AWS::IAM::Role
      Properties:
        RoleName: deployment-example-role
        AssumeRolePolicyDocument:
          Statement:
            - Effect: Allow
              Principal:
                Service: ecs-tasks.amazonaws.com
              Action: sts:AssumeRole
        ManagedPolicyArns:
          - arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy
          - arn:aws:iam::aws:policy/AmazonEC2FullAccess
          - arn:aws:iam::aws:policy/AmazonECS_FullAccess
    ContainerSecurityGroup:
      Type: AWS::EC2::SecurityGroup
      Properties:
        GroupName: CollectorContainerSecurityGroup
        GroupDescription: Security group for Collector containers
    TaskDefinition:
      Type: AWS::ECS::TaskDefinition
      Properties:
        Family: deployment-collector-task
        Cpu: 256
        Memory: 512
        NetworkMode: awsvpc
        ExecutionRoleArn: !Ref ExecutionRole
        ContainerDefinitions:
          - Name: deployment-example-metrics-collector
            Image: logzio/docker-collector-metrics
            Environment:
              - Name: LOGZIO_TOKEN
                Value: ${ssm:LOGZIO_METRICS_TOKEN}
              - Name: LOGZIO_MODULES
                Value: aws
              - Name: AWS_REGION
                Value: ${self:provider.region}
              - Name: AWS_NAMESPACES
                Value: "AWS/Lambda,AWS/S3"
              - Name: AWS_ACCESS_KEY
                Value: ${ssm:METRICS_AWS_ACCESS_KEY}
              - Name: AWS_SECRET_KEY
                Value: ${ssm:METRICS_AWS_SECRET_KEY}
            LogConfiguration:
              LogDriver: awslogs
              Options:
                awslogs-region: !Ref AWS::Region
                awslogs-group: !Ref LogGroup
                awslogs-stream-prefix: ecs
        RequiresCompatibilities:
          - EC2
          - FARGATE
    Service:
      Type: AWS::ECS::Service
      Properties:
        ServiceName: deployment-example-service
        Cluster: !Ref Cluster
        TaskDefinition: !Ref TaskDefinition
        DesiredCount: 1
        LaunchType: FARGATE
        NetworkConfiguration:
          AwsvpcConfiguration:
            AssignPublicIp: ENABLED
            Subnets:
              - ${ssm:METRICS_SUBNET}
            SecurityGroups:
              - !GetAtt ContainerSecurityGroup.GroupId

In comparison to the Docker file, this is a little bit more complex.

This section of the serverless.yml is doing quite a few things. For one, it’s creating the ECS (Elastic Container Service) Cluster that the Docker containers will deploy to.

Additionally, it’s generating the logging group they use for CloudWatch and the role in which they are run (including the security group).

And finally, it defines the container itself with the environment variables required to run. That way, it can successfully read the correct metrics data and send it to Logz.io.

Tracing

In my Node.js tutorial, I covered the instrumentation of tracing with Jaeger through the express application. The way you configure and start tracing is exactly the same, except now your entire Lambda function is your trace (and quite possibly your entire span).

Standing up the collector, however, is just a little bit complex. To use the Jaeger Collector that we’ve prepared to send tracing data to Logz.io, you’ll need to use something like the following example:

#yaml
#you can add CloudFormation resource templates here
resources:
  Resources:
    Cluster:
      Type: AWS::ECS::Cluster
      Properties:
        ClusterName: deployment-example-cluster
    LogGroup:
      Type: AWS::Logs::LogGroup
      Properties:
        LogGroupName: deployment-example-log-group
    ExecutionRole:
      Type: AWS::IAM::Role
      Properties:
        RoleName: deployment-example-role
        AssumeRolePolicyDocument:
          Statement:
            - Effect: Allow
              Principal:
                Service: ecs-tasks.amazonaws.com
              Action: sts:AssumeRole
        ManagedPolicyArns:
          - arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy
          - arn:aws:iam::aws:policy/AmazonEC2FullAccess
          - arn:aws:iam::aws:policy/AmazonECS_FullAccess
    ContainerSecurityGroup:
      Type: AWS::EC2::SecurityGroup
      Properties:
        GroupName: CollectorContainerSecurityGroup
        GroupDescription: Security group for Collector containers
        SecurityGroupIngress:
          - IpProtocol: tcp
            FromPort: 14268
            ToPort: 14268
            CidrIp: 0.0.0.0/0
          - IpProtocol: tcp
            FromPort: 9411
            ToPort: 9411
            CidrIp: 0.0.0.0/0
          - IpProtocol: tcp
            FromPort: 14269
            ToPort: 14269
            CidrIp: 0.0.0.0/0
          - IpProtocol: tcp
            FromPort: 14250
            ToPort: 14250
            CidrIp: 0.0.0.0/0
    TaskDefinition:
      Type: AWS::ECS::TaskDefinition
      Properties:
        Family: deployment-collector-task
        Cpu: 256
        Memory: 512
        NetworkMode: awsvpc
        ExecutionRoleArn: !Ref ExecutionRole
        ContainerDefinitions:
          - Name: deployment-example-traces-collector
            Image: logzio/jaeger-logzio-collector:latest
            PortMappings:
              - ContainerPort: 14268
              - ContainerPort: 9411
              - ContainerPort: 14269
              - ContainerPort: 14250
            Environment:
              - Name: ACCOUNT_TOKEN
                Value: ${ssm:LOGZIO_TRACING_TOKEN}
              - Name: AccountToken
                Value: ${ssm:LOGZIO_TRACING_TOKEN}
            LogConfiguration:
              LogDriver: awslogs
              Options:
                awslogs-region: !Ref AWS::Region
                awslogs-group: !Ref LogGroup
                awslogs-stream-prefix: ecs
        RequiresCompatibilities:
          - EC2
          - FARGATE
    Service:
      Type: AWS::ECS::Service
      Properties:
        ServiceName: deployment-example-service
        Cluster: !Ref Cluster
        TaskDefinition: !Ref TaskDefinition
        DesiredCount: 1
        LaunchType: FARGATE
        NetworkConfiguration:
          AwsvpcConfiguration:
            AssignPublicIp: ENABLED
            Subnets:
              - ${ssm:METRICS_SUBNET}
            SecurityGroups:
              - !GetAtt ContainerSecurityGroup.GroupId

Much like how we configured ECS to run our metrics collector, we’ve had to do the same for the Jaeger collector. The primary difference being that you need to send data to the jaeger collector, so you have to open the ports that the collector uses, like so:

#yaml
        SecurityGroupIngress:
          - IpProtocol: tcp
            FromPort: 14268
            ToPort: 14268
            CidrIp: 0.0.0.0/0
          - IpProtocol: tcp
            FromPort: 9411
            ToPort: 9411
            CidrIp: 0.0.0.0/0
          - IpProtocol: tcp
            FromPort: 14269
            ToPort: 14269
            CidrIp: 0.0.0.0/0
          - IpProtocol: tcp
            FromPort: 14250
            ToPort: 14250
            CidrIp: 0.0.0.0/0

You now have the Jaeger collection, and thanks to the Node.js tut, you know how to send spans as part of traces.

If you’d like to see how the whole application is wired together, feel free to check out the GitHub Repository. Check back for further info on serverless computing and Serverless frameworks in the future.