Helping to protect IT environments from cyber attacks and comply with tightening compliance standards, SIEM systems are becoming the cornerstone for security paradigms implemented by a growing number of organizations.
In a previous post, we explained what a SIEM system actually is — why organizations require it to start with, the components it is comprised of, and how it helps mitigate attacks. One of the conclusions reached in that article was that SIEM is not actually a single tool in itself, but is instead comprised of multiple monitoring and analysis components.
There are proprietary platforms that do offer an all-in-one SIEM solution, such as LogRhythm, QRadar, and ArcSight. These solutions can become rather expensive, especially in the long run and in larger organizations, and so more and more companies are on the search for an open source SIEM platform.
But is there an open source platform that includes all the basic SIEM ingredients?
The simple answer is -— no. That’s why Logz.io Cloud SIEM exists, our proprietary endeavor to amalgamate open source options to create a defense-prioritized security solution for observability, largely based off the ELK Stack (more on that below).
There is no all-in-one perfect open source SIEM system. Existing solutions either lack core SIEM capabilities, such as event correlation and reporting or require combining with other tools. As always, though, there are some good contenders, and in this article, we take a look at six of these platforms.
We will follow up this article with a similar analysis of proprietary tools.
The open source version of AlienVault’s Unified Security Management (USM) offering, OSSIM is probably one of the more popular open source SIEM platforms. OSSIM includes key SIEM components, namely event collection, processing and normalization, and most importantly — event correlation.
OSSIM combines native log storage and correlation capabilities with numerous open source projects in order to build a complete SIEM. The list of open source projects included in OSSIM includes: FProbe, Munin, Nagios, NFSen/NFDump, OpenVAS, OSSEC, PRADS, Snort, Suricata and TCPTrack.
The inclusion of OpenVAS is of particular interest, as OpenVAS is used both for vulnerability assessment by correlating IDS logs with vulnerability scanner results.
As one would expect, the open source OSSIM is not as feature rich as its commercial “older brother”. Both solutions work fine for small deployments, but OSSIM users experience significant performance issues at scale, ultimately driving them towards the commercial offering. Log management capabilities in the open source version of OSSIM, for example, are virtually non-existent.
2. The ELK Stack
The ELK stack is arguably the most popular open-source tool used today as a building block in a SIEM system. A building block — yes. A complete SIEM system — no, since there is plenty of room for debate about whether or not the ELK Stack qualifies as an “all in one” SIEM system.
The ELK stack consists of the open-source products Elasticsearch, Logstash, Kibana and the Beats family of log shippers. Though it’s important to note that Elasticsearch and Kibana will be under SSPL licenses as of January 14, 2021.
Logstash is a log aggregator that can collect and process data from almost any data source. It can filter, process, correlate and generally enhance any log data that it collects. Elasticsearch is the storage engine and one of the best solutions in its field for storing and indexing time-series data. Kibana is the visualization layer in the stack and an extremely powerful one at that. Beats include a variety of light-weight log shippers that are responsible for collecting the data and shipping it into the stack via Logstash.
Logstash uses a wide array of input plugins to collect logs. However, it can also accept input from more purpose-built solutions like OSSEC or Snort (see below). Combined, the ELK Stack’s log processing, storage and visualization capabilities are functionally unmatched. For the purposes of SIEM however, the ELK Stack — at least in its raw open source format, is missing some key components.
First and foremost, there is no built-in reporting or alerting capability. This is a known pain point not only for users trying to use the stack for security but also for more common use cases — IT operations for example. Alerting can be added by using the X-Pack, a commercial product by Elastic, or by adding an adding open source security add-ons.
There are also no built-in security rules that can be used. This makes the stack a bit more costly to handle, both in terms of resources and operational costs.
OSSEC is a popular open source Host Intrusion Detection System (HIDS) that works with various operating systems, including Linux, Windows, MacOS, Solaris, as well as OpenBSD and FreeBSD.
OSSEC itself is broken into two main components: the manager (or server), responsible for collecting the log data from the different data sources, and the agents — applications that are responsible for collecting and processing the logs and making them easier to analyze.
The OSSEC project itself does not include a visualization layer. There was a UI which was deprecated, and instead, the recommendation is to use external visualization tools such as Kibana and Grafana.
OSSEC directly monitors a number of parameters on a host. This includes log files, file integrity, rootkit detection, and Windows registry monitoring. OSSEC can perform log analysis from other network services, including most of the popular open source FTP, mail, DNS, database, web, firewall, and network-based IDS solutions. OSSEC can also analyze logs from a number of commercial network services and security solutions.
OSSEC has a number of alerting options and can be used as part of automated intrusion detection or active response solutions. OSSEC has a primitive log storage engine. By default, log messages from host agents are not retained. Once analyzed, OSSEC deletes these logs unless the <logall> option is included in the OSSEC manager’s ossec.conf file. If this option is enabled, OSSEC stores the incoming logs from agents in a text file that is rotated daily.
Whether or not OSSEC can be counted as an “all in one” SIEM system is debatable. OSSEC definitely does the hard work involved in implementing a SIEM system: it collects data and analyzes it, but lacks some of the core log management and analysis components required. It’s worth pointing out that the OSSEC project has been forked by other HIDS solutions (e.g. Wazuh) that extend OSSEC functionality and make it a more complete SIEM option.
Wazuh is a HIDS solution forked from OSSEC. It describes itself as an “enterprise-ready security monitoring solution” that is fully compliant and instilled with both incident response capabilities and integrity monitoring. Wazuh’s creators contend OSSEC had not seen enough updates prior to 2015, when Wazuh was first released.
Wazuh intends to add scalability to OSSEC. That includes support for Puppet, Chief, Docket and Ansible. Cluster support and multithread support, plus anti-flooding capabilities, control for increased performance for sustainable horizontal scaling. It also integrates with Suricata or Owhl project for NIDS, other databases. It has modules and decoders for both AWS and Microsoft Azure. On the ELK Stack side, it is fully compliant via the Wazuh Kibana plugin and data enrichment via a GeoIP Logstash module.
Their documentation includes links to upgrading servers and agents to migrate from OSSEC to Wazuh. As of update 3.7.1, more tracing information is included within debugging mode logs. In 3.7.2, Wazuh fixed issues related to its Logcollector module, now discarding lines with binary characters.
5. Apache Metron
Evolving from Cisco’s OpenSOC platform and first released in 2016, Apache Metron is a relatively new player in the industry and another example of a security framework that combines multiple open source projects into one platform.
From an architectural perspective, Metron relies on other Apache projects for collecting, streaming and processing security data. Apache Nifi and Metron probes collect data from security data sources which is then pushed into separate Apache Kafka topics. Events are subsequently parsed and normalized into standard JSON and then enriched and in some cases labeled. Alerts can be triggered if certain event types are identified. For visualization, Kibana is used (albeit an outdated version)
For storage, events are indexed and persisted in Apache Hadoop and either Elasticsearch or Solr based on the organization’s preferences. On top of this data, Metron provides an interface for centralizing the analysis of the data with alert summaries and enriched data.
One of Metron’s strongest features is it’s pluggable and extensible architecture. Bro, pycapa and fastcapa sensors for example, can be used to ship specific data into Metron. Using Stellar, a simple DSL, users can write their own functions for transforming collected data. An extensive REST API allows users to interact with Metron, so users can for example programmatically manage alerts.
Being relatively young, Metron still lacks in some aspects. Metron can only be installed on a limited number of operating systems and environments though it does support automation scenarios with Ansible and installation via Docker (Mac and Windows only). The UI is a bit immature and does not support authentication for example.
SIEMonster is another young SIEM player but an extremely popular one as well, with over 100,000 downloads in just two years. SIEMonster is based on open source technology and is available for free and as a paid solution (Premium and MSSP multi-tenancy).
While SIEMonster uses its own “monster” terminology to name the different SIEM functions within the system (e.g. Kraken), the underlying components are well known open source technologies. The ELK Stack is used for collection (Filebeat and Logstash), processing, storage and visualization of the security data collected. RabbitMQ is used for queuing. SearchGuard is used for encryption and authentication on top of Elasticsearch and ElastAlert for alerting. A fork of OSSEC Wazuh for HIDS. The list goes on.
From a functionality perspective, SIEMonster includes all the goodies an analyst could wish for, each accessed via a main menu — the Kibana UI for searching and visualizing data, a MineMeld UI for threat intelligence, Alerts for creating and managing event-based notifications. Additional integrated open source tools are DRADIS, OpenAudit, and FIR.
SIEMonster can be deployed on the cloud using Docker containers, meaning easier portability across systems, but also on VMs and bare metal (Mac, Ubuntu, CentOS, and Debian). Documentation is extensive, though an online version is missing.
Similar to OSSIM, Prelude is a SIEM framework that unifies various other open source tools. And like OSSIM, it is also an open source version of the commercial tool by the same name. Prelude aims to fill the roles that tools like OSSEC and Snort leave out.
Prelude accepts logs and events from multiple sources and stores them all in a single location using the Intrusion Detection Message Exchange Format (IDMEF). It provides filtering, correlation, alerting, analysis, and visualization capabilities.
Again, like OSSIM, the open source version of Prelude is significantly limited when compared to the commercial offering in all of these capabilities which is probably why it is not very popular. Quoting the official documentation: “Prelude OSS is aimed for evaluation, research and test purpose on very small environments. Please note that Prelude OSS performances are way lower than the Prelude SIEM edition.”
SecurityOnion is a free Linux distribution (distro) for intrusion detection and network (NSM) and enterprise security monitoring (ESM). It piggybacks off other open-source projects like the ELK Stack, OSSEC, Snort (more on that below), Suricata and others. It was developed by Doug Burks and released in 2008, who later launched Security Onion Solutions in 2014.
It provides both host-based and network-based intrusion detection systems (IDS), as well as full packet capture (FPC) via netsniff-ng for catching events such as data exfiltration, malware, phishing emails, and other exploits on networks (other open-source options for FPC include GUI-based TCPDUMP and command-line interface Wireshark).
For network-based IDS, it provides user the choice of Snort or Suricata (more on those below); for host-based IDS (a.k.a., HIDS), it offers Wazuh.
A portmanteau for Mozilla Defense (and perhaps more importantly, a tribute to rapper-activist Mos Def), the company famous for Firefox built this security incident and response automation tool from other open-source tools. It was first released in 2014.
Each service in its architecture runs in a Docker container. Mozilla describes it as a SIEM add-on that runs on top of Elasticsearch for logging and Python for writing new rules.
Snort is a network intrusion detection system (NIDS) designed for Windows and Linux. This distinguishes it from other host-based systems like OSSEC. With that in mind, Snort is not necessarily an alternative to OSSEC or other SIEMs but a possible addendum.
Snort gets its name from being a packet sniffer that will ‘sniff’ out security threats to networks. It detects and reports attack methods, thereby sending an alert to syslog or through another channel. It conducts real-time traffic analysis along with logs. It is designed to detect a long list of different attack vectors that includes OS fingerprinting, DDOS, CGI, SMB probes, buffer overflows and stealth port scans. It uses OpenAppID to detect applications.
Its creator, Martin Roesch, assembled Sourcefire to manage the software for its hundreds of thousands of users. Sourcefire was acquired by Cisco in 2013, but Snort retains its open-source origins (while Cisco has gone on to develop commercial alternatives based on the original software).
It’s most recent release, 18.104.22.168, came in October 2019. Some of its shortcomings might be addressed by Snort 3.0 (currently in beta), including its lack of multithreading.
Snort is often compared to and might serve as an alternative to Suricata.
A common alternative to Snort, it has cut into the former’s user base as a common intrusion detection system (IDS), PCAP processing, intrusion prevention and network monitoring. It is owned by the Open Information Security Foundation (OISF). Distinctly, it is built of the Lia scripting language whose developers market as a small, fast and embeddable language.
It maintains integrations in YAML and JSON for other databases like Elasticsearch and Splunk.
It uses many of the same rules as Snort, but with some differences. Instead of OpenAppID, it can use application-layer detection to identify HTTP and SSH traffic.
As a newer tool, it is also more adept at modern computing issues. It supports multithreading natively instead of Snort’s running of multiple single-thread instances.
No “one ring to rule them all”
A complete SIEM solution includes the ability to collect information from various data sources, retain that information for an extended period of time, correlate between different events, create correlation rules or alerts, analyze the data and monitor it with visualizations and dashboards.
Answering a lot of these requirements, it is no coincidence that the ELK Stack is used by many of the open source SIEM systems listed in this article. OSSEC Wazuh, SIEMonster, Metron —all have ELK beneath the hood. But taken on its own, ELK lacks some key SIEM components, such as correlation rules and incident management.
That’s part of the reason we saw to build Logz.io Cloud SIEM, to create a single pane through which to observe the security-relevant events in your logs. Mostly based off the ELK Stack, it draws from wisdom seen in many solutions. It blends data from endpoint protections like Sophos and ESET, firewalls like SonicWall and Stormshield, MSSPs, and others.
Based on the analysis above, the simple conclusion is that there are no clear winners to the title “an all-in-one open source SIEM solution”. When implementing a SIEM system based on the solutions above, you will most likely find yourself limited as far as functionality is concerned or combining with additional open source tools.
Open source tools used for SIEM are versatile and powerful. But, they require a great deal of expertise, and above all — time to deploy properly. It is for this reason that commercial offerings still dominate the SIEM landscape, even when open-source tools lie at the core of those commercial offerings.
Having 80% of your SIEM solution handled for you is better than having to do it all by yourself. Commercial solutions handle installation, basic configuration, and provide filters, correlation configurations, and visualization designs for the most common use cases. Don’t underestimate the value of these commercial features: there are a seemingly unlimited number of things to monitor in today’s datacenters, and none of us have time to manually configure applications to watch them all.