Last month, we announced Logz.io Security Analytics — a security app built on top of the ELK Stack, offering out-of-the-box security features such as threat intelligence, correlation, and premade integrations and dashboards.
In this article, I’d like to show an example of using both the ELK Stack and Logz.io Security Analytics to secure an AWS environment. To do this, we are going to ship GuardDuty data into Logz.io Security Analytics and then construct a security dashboard using Kibana.
What is GuardDuty?
AWS GuardDuty is a security service that monitors your AWS environment and identifies malicious or unauthorized activity. It does this by analyzing the data generated by various AWS data sources, such as VPC Flow Logs or CloudTrail events, and correlating it with thread feeds. The results of this analysis are security findings such as bitcoin mining or unauthorized instance deployments.
Your AWS account is only one component you have to watch in order to secure a modern IT environment and so GuardDuty is only one part of a more complicated security puzzle that we need to decipher. That’s where security analytics solutions come into the picture, helping to connect the dots and provide a more holistic view.
Shipping from GuardDuty into Logz.io
GuardDuty ships data automatically into CloudWatch. To ship this data into Logz.io, we will first create a Kinesis stream and use a Lambda function to consume it and send the data to Logz.io in bulk over HTTPS.
Creating a Kinesis Stream
The first step in the pipeline is to create a new data stream in Kinesis. To do this, open the Kinesis console and hit the Create Kinesis stream button.
Give the stream a name and enter the number of shards you think you need. For the purpose of this article, starting with one shard will suffice but you can use the provided shard calculator to come up with a more adequate number to suit the expected data flow.
When done, click the Create Kinesis stream button at the bottom of the page.
Create an IAM role
First, let’s create a new IAM role allowing Kinesis to execute our Lamba.
On the Roles page in the IAM console, click the Create Role button.
Select Lambda from the available entities, and on the next page assign the AWSLambdaKinesisExecutionRole permissions policy to the new role.
Complete the process, give the role a memorable name – you will need this role in the next step.
Create a Lambda shipper
Next, we will create a new Lambda function to collect the GuardDuty events from CloudWatch and ship them to Logz.io via Kinesis.
First, let’s create the Lambda function.
In the AWS Lambda console, choose to create a function from scratch and use the following configurations for the function:
- Runtime – select Python 2.7
- Role – select the role you created above.
After clicking the Create Function button, your function is created and your next step is to upload the code itself.
In your terminal, clone the GitHub repo containing the Lambda:
git clone https://github.com/logzio/logzio_aws_serverless.git
Execute the commands below to access the folder and zip the shipper:
mkdir dist; cp -r ../shipper dist/ && cp src/lambda_function.py dist/ && cd dist/ && zip logzio-kinesis shipper/*
In the Function Code section, back in the Lambda console, open the Code entry type drop-down menu and select Upload a .zip file.
Upload the zipped shipper, and hit the save button in the top right corner.
Next, in the Environment variables section, add the following variables:
- FORMAT – json
- TOKEN – Your Logz.io token. It can be found in your Logz.io app account settings.
URL – enter the Logz.io listener URL. This depends on the region your account is deployed in. For US, use https://listner.logz.io:8071. For the EU, https://listner-eu.logz.io:8071 (to determine which region you are in, simply take a look at the login URL – app.logz.io means the U.S, app-eu.logz.io means the EU.
Save the function again.
Create CloudWatch rule
Next, we need to create a CloudWatch rule to ship the GuardDuty data into the Kinesis stream we created.
In the CloudWatch console, select Rules from the menu on the left, and then the Create
Under Event Source, select GuardDuty from the Service Name drop-down menu. In the Targets section, select the Kinesis stream you created in the previous step.
Click Configure Details to move to the next step, which is entering a name and creating the rule.
Define the Lambda trigger
To finish building our data pipeline, we need to create a new trigger for the Lambda function.
From the list of available triggers on the left, select Kinesis and select the Kinesis stream you created in the previous step, and click Add. The rule is added.
Save the function to apply the new design configurations.
A good way to test the pipeline is using GuardDuty’s sample findings feature. This can be used in the Settings section in the GuardDuty console.
Within a few minutes, you will begin to see GuardDuty events in Logz.io.
Data shipped to Logz.io is stored in your operations accounts. To identify threats and security analysis, switch over to the Security Analytics interface using the menu in the top-left corner of the page.
Building a GuardDuty security dashboard
Logz.io Security Analytics is based on Kibana, and as such gives you all the freedom you need to query the data and visualize it.
Querying can be done on the Research page, where you will find the familiar Kibana Discover page.For example, you can quickly find GuardDuty events with a high severity (severity level 7 and above). These events indicate that an EC2 instance or a set of IAM user credentials is compromised and is actively being used for unauthorized purposes:
type:guardduty AND detail.severity:[7 TO *]
Things get more interesting, of course, when we start using Kibana’s visualization capabilities. Kibana provides various options for visualizing data, and it is really up to you to slice and dice the data in a way that allows you to monitor it effectively.
Here are a few examples of visualizations that you can develop on top of GuardDuty data.
Using a line chart visualization, we can see events found by GuardDuty, broken down by their severity:
Threats per region
We can use another line chart visualization to view threat per AWS region in which they were identified:
A data table visualization can be created to display a simple but detailed list of the different threats identified by GuardDuty and shipped into Logz.io.
Threat severity by account
To try and identify a particular problematic account, we can create a bar chart visualization that shows each account and its threat severity.
Threats by security group
A pie chart gives us an overview of the threats by security group:
Adding these visualizations, and others, into one comprehensive dashboard, gives you a security dashboard of your AWS environment:
Summing it up
GuardDuty correlates between events taking place in your AWS environment and enriches these events to report on suspicious behavior. A small-to-medium sized deployment on AWS will generate thousands of these events and so a more centralized approach is necessary.
Logz.io Security Analytics provides built-in dashboards for GuardDuty, such as the general overview dashboard shown above as well as dashboards focusing on EC2 and VPC events. Integrating with Logz.io is straightforward with the use of the Lambda function above and with the help of Kibana’s analysis features, you’ll be able to investigate and analyze security incidents more efficiently.
In the next post, we will demonstrate using Logz.io correlation rules to identify and alert on a sequence of events happening in your AWS environment.