Network Security Monitoring with Suricata, Logz.io and the ELK Stack

By: Daniel Berman

Suricata is an open source threat detection system. Initially released by the Open Information Security Foundation (OISF) in 2010, Suricata can act both as an intrusion detection system (IDS), and intrusion prevention system (IPS), or be used for network security monitoring.

Installing Suricata

Our first step, is to set up Suricata.

Suricata can be installed on a variety of distributions using binary packages or compiled from source files. We’ll be installing Suricata on Ubuntu 16.04, and full installation instructions are available here.

Start with installing recommended dependencies:

apt-get install libpcre3 libpcre3-dbg libpcre3-dev build-essential libpcap-dev   \
                libyaml-0-2 libyaml-dev pkg-config zlib1g zlib1g-dev \
                make libmagic-dev

Next, define the PPA for installing latest stable release:

sudo add-apt-repository ppa:oisf/suricata-stable

Update your system and install Suricata with:

sudo apt-get update
sudo apt-get install suricata

Next, we’re going to install Suricata-Update — a tool for updating and managing Suricata rules. Suricate-Update is packaged with Suricata, so to use it simply use:

sudo suricata-update

The Emerging Threats Open ruleset is downloaded to /var/lib/suricata/rules/, and this is the command to use to update your rules.

Last but not least, start Suricata:

sudo service suricata start

Shipping Suricata data to the ELK Stack

After a minute or two, Suricata will begin to generate events in a file called ‘eve.json’ located at /var/log/suricata/eve.json.

All the data generated by Suricata, whether rule-based alerts, DNS or HTTP logs, will be sent into this file. You can configure what data is shipped into this file, define different log outputs per data type, and more, in the Suricata configuration file (/etc/suricata/suricata.yaml).

All the log types in the ‘eve.json’ file share a common structure:

{"timestamp":"2009-11-24T21:27:09.534255","event_type":"TYPE", ...tuple... ,"TYPE":{ 
... type specific content ... }}

Our next step is to ship this data into the ELK Stack for analysis. We will outline two methods — using Filebeat to ship to Logz.io and using a combination of Filebeat and Logstash for vanila ELK deployments.

Using Logz.io

If you’re a Logz.io user, all you have to do is install Filebeat and configure it to forward the suricata_ews.log file to Logz.io. Processing and parsing will be applied automatically.

To install Filebeat, use:

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
sudo apt-get install apt-transport-https

echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a 
/etc/apt/sources.list.d/elastic-6.x.list

sudo apt-get update && sudo apt-get install filebeat

Before configuring Filebeat, download an SSL certificate for encryption:

wget 
https://raw.githubusercontent.com/logzio/public-certificates/master/COMODORSADomainValidationSecureServerCA.crt

sudo mkdir -p /etc/pki/tls/certs

sudo cp COMODORSADomainValidationSecureServerCA.crt /etc/pki/tls/certs/

Next, open up the Filebeat configuration file at /etc/filebeat/filebeat.yml, and enter the following configuration:

filebeat.inputs:

- type: log
  paths:
    - /var/log/suricata/eve.json
  fields:
    logzio_codec: json
    token: <yourLogzioToken>
    type: suricata
  fields_under_root: true
  encoding: utf-8
  ignore_older: 3h

registry_file: /var/lib/filebeat/registry

output:
  logstash:
    hosts: ["listener.logz.io:5015"]  
    ssl:
      certificate_authorities: ['/etc/pki/tls/certs/COMODORSADomainValidationSecureServerCA.crt']

Be sure to enter your Logz.io account token in the relevant field. You can find your token in the Logz.io UI, on the General page of your settings.

Save the configuration file and start Filebeat:

sudo service filebeat start

Wait a minute or two, you will begin to see the Suricata data in Logz.io:

Using Logstash

If you’re using your own ELK deployment you will want to add Logstash into the pipeline for processing the data.

First though, our Filebeat configuration file will look as follows:

filebeat:
 prospectors:
  - input_type: log
    paths:
     - "/var/log/suricata/eve.json"
    json.keys_under_root: true
    json.overwrite_keys: true
    fields:
      application: suricata

output:
 logstash:
  # The Logstash hosts
   hosts: ["localhost:5044"]

Next up, Logstash. This is a critical step in the process as we need to ensure the Suricata logs are broken up properly for easier analysis.

Since we will be using the geoip filter plugin, we need to first install it:

cd /usr/share/logstash
sudo bin/logstash-plugin install logstash-filter-geoip

Create a configuration file:

sudo vim /etc/logstash/conf.d/suricata.conf

The configuration defines beats as the input, Elasticsearch as the output and uses a variety of filter plugins to process the logs:

input {
  beats { 
    port => 5044
    codec => "json_lines"
  }
}

filter {
  if [application] == "suricata" {
    date {
      match => [ "timestamp", "ISO8601" ]
    }
    ruby {
      code => "if event['event_type'] == 'fileinfo'; event['fileinfo']['type']=event['fileinfo']['magic'].to_s.split(',')[0]; end;" 
    }
  }

  if [src_ip]  {
    geoip {
      source => "src_ip" 
      target => "geoip" 
      add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
      add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
    }
    mutate {
      convert => [ "[geoip][coordinates]", "float" ]
    }
    if ![geoip.ip] {
      if [dest_ip]  {
        geoip {
          source => "dest_ip" 
          target => "geoip" 
          add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
          add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
        }
        mutate {
          convert => [ "[geoip][coordinates]", "float" ]
        }
      }
    }
  }
}

output { 
  elasticsearch {
    hosts => ["localhost:9200"]
  }
}

Start Logstash and Filebeat with:

sudo service logstash start
sudo service filebeat start

Within a minute or two, you should see a new Logstash index created in Elasticsearch:

curl -X GET "localhost:9200/_cat/indices?v"

health status index                     uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .kibana_1                 Eb79G5FESxiadHz8WCoE9w   1   0        178            8    280.3kb        280.3kb
yellow open   logstash-2019.01.28       _53z4wLXSLiiW5v-OMSqRg   5   1       1157            0      2.3mb          2.3mb

In Kibana, you can then define the new index pattern to start your analysis:

Analyzing the data

So – what are we looking for? What can be done with Suricata logs?

The parsing applied to the logs parses out some important fields which we can use for monitoring our network for suspicious behavior. Below are a few examples of visualizations that can be built in Kibana for detecting malicious activity.

Suricata Alerts

Suricata rules (/etc/suricata/rules) will trigger alerts and send logs to Logz.io should the conditions defined in those rules be met. To understand what the actual rules mean, I would recommend you refer to Suricata docs.

In the bar chart below, we are monitoring the different alert categories Suricata is logging over time:

To monitor alert signatures, we could build a Data Table visualization:

Traffic Geographic Distribution

We can monitor the geographical distribution of traffic coming into our system using Kibana’s Coordinate Map visualization. To do this we will make use of the geo enhancement applied to the src_ip field in Suricata logs.

Suricata Event Types

The event_type field indicates the Suricata log type. With the help of a pie chart visualization, we can see a breakdown of the top log types recorded in the system:

Line chart visualizations are great to monitor trends over time. In the visualization below, we’re looking at the top Suricata log types over time:

Once you’ve got all your visualizations lined up, you can add them into one comprehensive dashboard:

This dashboard is available in ELK Apps — Logz.io’s library of Kibana searches, alerts, visualizations and dashboards, together with other Suricata dashboards. Simply search for “Suricata” and install any of the listed dashboards in one click.

Endnotes

Suricata is a great open source option for monitoring your network for malicious activity. Still, it is just one layer in what must be a more comprehensive system deployed, rather than a complete solution for security issues.

Firewalls, endpoint monitoring, access monitoring, and more — these are additional layers that need to be put in place and integrated into your security architecture. Of course, a multi-layered system generates a huge volume of data which requires a security analytics platform that can ingest all the data, process it and give you the tools to effectively analyze it.

The integration with Logz.io provides you with aggregation and analytics capabilities necessary to make the most out of the logs and events that Suricata generates. We will be enhancing our Suricata integration with additional rules and dashboards in our Security Analytics platform so stay tuned!

Observability at scale,
powered by open source

See Plans

Completely free for 14 days, no strings attached.

Start Free Trial

Schedule Demo