10 Open Source SIEM Tools
Helping to protect IT environments from cyber attacks and comply with tightening compliance standards, SIEM systems are becoming the cornerstone for security paradigms implemented by a growing number of organizations.
Previously, we explained what a SIEM system is — why organizations require it to start with, the components it is composed of, and how it helps mitigate attacks. One of the conclusions reached in that article was that SIEM is not actually a single tool in itself, but instead comprised multiple monitoring and analysis components.
There are proprietary platforms that do offer an all-in-one SIEM solution, such as Splunk, LogRhythm, and AlienVault. These solutions can become rather expensive, especially in the long run and in larger organizations, and so more and more companies are on the search for an open source SIEM platform.
But is there an open source platform that includes all the basic SIEM ingredients? The simple answer is no.
There’s an old saying about Linux: “Linux is only free if you don’t value your time.” The same can be said about open source SIEM tools. Open source means “no price tag” but it also means “heavy time investment.”
This is why we created Logz.io Cloud SIEM. We want it to be the best of both worlds where we do all the management for you, such as full implementation, parsing as a service, creation of threat content and more. We do it so you don’t have to.
We amalgamate open source options to create a defense-prioritized security solution for observability, largely based on OpenSearch (more on that below).
There is no all-in-one perfect open source SIEM system. This is also not a list of open source SIEMs, because there is no one complete open source SIEM. Existing solutions either lack core SIEM capabilities, such as event correlation and reporting, or require combining with other tools. But if you decide that you want to take on the extensive project of building your own SIEM from the ground up with open source, here are the components we think you should use.
1. OpenSearch
OpenSearch is an open source software project launched in 2021 as a fork of the Elasticsearch and Kibana projects, with development led by Amazon Web Services. The project includes a database (also named OpenSearch) and frontend visualization and analytics called OpenSearch Dashboards.
In January 2021, Elastic, the company behind the Elastic Stack (made up of the Elasticsearch, Kibana, Beats, and Logstash projects, and often known as the ELK Stack or Elastic Stack) – announced that it would shift to a dual licensing structure based on the Server Side Public License (SSPL) and the Elastic License – neither of which has been recognized as an open source license by the Open Source Initiative (OSI). In response, Logz.io partnered with Amazon and other industry leaders to create an open-source alternative to the newly closed-source ELK stack, and OpenSearch was born.
Similar to the ELK stack, some organizations use OpenSearch as a homegrown logging repository for security data. This doesn’t qualify it as a SIEM, per se, because any core SIEM features like security detections, content, analytics, and threat enrichment need to be hand-rolled. But for organizations that are heavy on technical talent and light on cash, it can be a cost-effective and future-proofed approach.
2. The ELK Stack
This blog was originally posted when the ELK stack was still fully-open source. That is no longer the case, as we noted above. You can still use the free edition of the ELK stack. You may still use it for compatibility with legacy systems, or another reason. But while it’s no longer a true open source option, we considered it important to keep here given its pervasiveness in this space.
The ELK stack was arguably the most popular open-source tool used as a building block in a SIEM system before it stopped being truly open source. A building block — yes. A complete SIEM system — no, since there is plenty of room for debate about whether or not the ELK Stack qualifies as an “all in one” SIEM system.
The ELK stack consists of Elasticsearch, Logstash, Kibana and the Beats family of log shippers. Elasticsearch and Kibana are under SSPL licenses as of January 14, 2021.
Logstash is a log aggregator that can collect and process data from almost any data source. It can filter, process, correlate and generally enhance any log data that it collects. Elasticsearch is the storage engine and one of the best solutions in its field for storing and indexing time-series data. Kibana is the visualization layer in the stack and an extremely powerful one at that. Beats include a variety of light-weight log shippers that are responsible for collecting the data and shipping it into the stack via Logstash.
Logstash uses a wide array of input plugins to collect logs. However, it can also accept input from more purpose-built solutions like OSSEC or Snort (see below). Combined, the ELK Stack’s log processing, storage and visualization capabilities are functionally unmatched. For the purposes of SIEM however, the ELK Stack — at least in the free version offered by Elastic, is missing some key components.
First and foremost, there is no built-in reporting or alerting capability. This is a known pain point not only for users trying to use the stack for security but also for more common use cases — IT operations for example. Alerting can be added by using the X-Pack, a commercial product by Elastic, or by adding open source security add-ons.
There are also no built-in security rules that can be used. This makes the stack a bit more costly to handle, both in terms of resources and operational costs.
3. OSSEC
OSSEC is a popular open source Host Intrusion Detection System (HIDS) that works with various operating systems, including Linux, Windows, MacOS, Solaris, as well as OpenBSD and FreeBSD.
OSSEC itself is broken into two main components: the manager (or server), responsible for collecting the log data from the different data sources, and the agents — applications that are responsible for collecting and processing the logs and making them easier to analyze.
The OSSEC project itself does not include a visualization layer. There was a UI which was deprecated, and instead, the recommendation is to use external visualization tools such as Kibana and Grafana.
OSSEC directly monitors a number of parameters on a host. This includes log files, file integrity, rootkit detection, and Windows registry monitoring. OSSEC can perform log analysis from other network services, including most of the popular open source FTP, mail, DNS, database, web, firewall, and network-based IDS solutions. OSSEC can also analyze logs from a number of commercial network services and security solutions.
OSSEC has a number of alerting options and can be used as part of automated intrusion detection or active response solutions. OSSEC has a primitive log storage engine. By default, log messages from host agents are not retained. Once analyzed, OSSEC deletes these logs unless the <logall> option is included in the OSSEC manager’s ossec.conf file. If this option is enabled, OSSEC stores the incoming logs from agents in a text file that is rotated daily.
Whether or not OSSEC can be counted as an “all in one” SIEM system is debatable. OSSEC definitely does the hard work involved in implementing a SIEM system: it collects data and analyzes it, but lacks some of the core log management and analysis components required. It’s worth pointing out that the OSSEC project has been forked by other HIDS solutions (e.g. Wazuh) that extend OSSEC functionality and make it a more complete SIEM option.
4. Snort
Snort is a network intrusion detection system (NIDS) designed for Windows and Linux. This distinguishes it from other host-based systems like OSSEC. With that in mind, Snort is not necessarily an alternative to OSSEC or other SIEMs but a possible addendum.
Snort gets its name from being a packet sniffer that will ‘sniff’ out security threats to networks. It detects and reports attack methods, thereby sending an alert to syslog or through another channel. It conducts real-time traffic analysis along with logs. It is designed to detect a long list of different attack vectors that includes OS fingerprinting, DDOS, CGI, SMB probes, buffer overflows and stealth port scans. It uses OpenAppID to detect applications.
Its creator, Martin Roesch, assembled Sourcefire to manage the software for its hundreds of thousands of users. Sourcefire was acquired by Cisco in 2013, but Snort retains its open-source origins (while Cisco has gone on to develop commercial alternatives based on the original software).
Its most recent major-version release, Snort 3.0, came out in January 2021, and addressed many of the shortcomings in Snort 2.x, including lack of multithreading. Snort is often compared to and might serve as an alternative to Suricata.
5. Suricata
A common alternative to Snort, it has cut into the former’s user base as a common intrusion detection system (IDS), PCAP processing, intrusion prevention and network monitoring. It is owned by the Open Information Security Foundation (OISF). Distinctly, it is built on the Lua scripting language, a small, fast and embeddable language.
It maintains integrations in YAML and JSON for other databases like Elasticsearch and Splunk.
It uses many of the same rules as Snort, but with some differences. Instead of OpenAppID, it can use application-layer detection to identify HTTP and SSH traffic.
As a newer tool, it is also more adept at modern computing issues. It supports multithreading natively instead of Snort’s running of multiple single-thread instances.
6. SecurityOnion
SecurityOnion is a free Linux distribution (distro) for intrusion detection and enterprise security monitoring (ESM). It piggybacks off other open-source projects like the ELK Stack, OpenSearch, OSSEC, Snort, Suricata and others. It was developed by Doug Burks and released in 2008, who later launched Security Onion Solutions in 2014.
It provides both host-based and network-based intrusion detection systems (IDS), as well as full packet capture (FPC) via netsniff-ng for catching events such as data exfiltration, malware, phishing emails, and other exploits on networks (other open-source options for FPC include GUI-based TCPDUMP and command-line interface Wireshark).
For network-based IDS, it provides users the choice of Snort or Suricata; for host-based IDS (a.k.a., HIDS), it offers Wazuh.
7. MozDef
A portmanteau for Mozilla Defense (and perhaps more importantly, a tribute to rapper-activist Mos Def), the company famous for Firefox built this security incident and response automation tool from other open-source tools. It was first released in 2014.
Each service in its architecture runs in a Docker container. Mozilla describes it as a SIEM add-on that runs on top of Elasticsearch for logging and Python for writing new rules.
According to the MozDef docs, they can integrate with a number of log shippers while exporting JSON to HTTP(S) or rabbit-mq. It can also tie in with GuardDuty and AWS CloudTrail. In addition to those mentioned above, they list the following open-source tools as its basis: Nginx, Meteor, MongoDB, VERIS (from Verizon), and several Python or JavaScript-relevant tools.
8. OSSIM
The open source version of AlienVault’s Unified Security Management (USM) offering, OSSIM is probably one of the more popular open source SIEM platforms. OSSIM includes key SIEM components, namely event collection, processing and normalization.
OSSIM combines native log storage and correlation capabilities with numerous open source projects in order to build a complete SIEM. The list of open source projects included in OSSIM includes: FProbe, Munin, Nagios, NFSen/NFDump, OpenVAS, OSSEC, PRADS, Snort, Suricata and TCPTrack. Of course, this means greater management overhead to maintain the SIEM, as every open-source project you add to the pile will require its own maintenance.
The inclusion of OpenVAS is of particular interest, as OpenVAS is also used for vulnerability assessment by correlating IDS logs with vulnerability scanner results.
As one would expect, the open source OSSIM is not as feature rich as its commercial alternative, and both suffer from significant scaling problems, even in modestly-sized environments. Log management capabilities in the open source version of OSSIM are virtually non-existent.
9. Prelude
Similar to OSSIM, Prelude is a SIEM framework that unifies various other open source tools. And like OSSIM, it is also an open source version of the commercial tool by the same name. Prelude aims to fill the roles that tools like OSSEC and Snort leave out.
Prelude accepts logs and events from multiple sources and stores them all in a single location using the Intrusion Detection Message Exchange Format (IDMEF). It provides filtering, correlation, alerting, analysis, and visualization capabilities.
Like OSSIM, the open source version of Prelude is significantly limited when compared to the commercial offering in all of these capabilities which is probably why it is not very popular. Quoting the official documentation: “Prelude OSS is aimed for evaluation, research and test purpose on very small environments. Please note that Prelude OSS performances are way lower than the Prelude SIEM edition.”
10. Apache Metron
Evolving from Cisco’s OpenSOC platform and first released in 2016, Apache Metron is a data lake and not an open source SIEM tool per se, but we wanted to mention it here. It’s another example of a security framework that combines multiple open source projects into one platform.
From an architectural perspective, Metron relies on other Apache projects for collecting, streaming and processing security data. Apache Nifi and Metron probes collect data from security data sources which is then pushed into separate Apache Kafka topics. Events are subsequently parsed and normalized into standard JSON and then enriched and in some cases labeled. Alerts can be triggered if certain event types are identified. For visualization, Metron deployments commonly use Kibana.
For storage, events are indexed and persisted in Apache Hadoop and either Elasticsearch or Solr based on the organization’s preferences. On top of this data, Metron provides an interface for centralizing the analysis of the data with alert summaries and enriched data.
One of Metron’s strongest features is its pluggable and extensible architecture. Bro, pycapa and fastcapa sensors for example, can be used to ship specific data into Metron. Using Stellar, a domain-specific language, users can write their own functions for transforming collected data. An extensive REST API allows users to interact with Metron, so users can for example programmatically manage alerts.
Metron can only be installed on a limited number of operating systems and environments though it does support automation scenarios with Ansible and installation via Docker (Mac and Windows only). The UI is a bit immature and does not support authentication for example.
No “one ring to rule them all”
A complete SIEM solution includes the ability to collect information from various data sources, retain that information for an extended period of time, and – more importantly – correlate between different events, create correlation rules, analyze the data and monitor it with visualizations and dashboards.
That’s part of the reason we saw to build Logz.io Cloud SIEM: to leverage OpenSearch technology as a platform on which to build. Logz.io Cloud SIEM extends our scalable, fully-managed data collection platform with a custom dynamic correlation and alerting engine, threat intelligence enrichment, out-of-the-box security content, and advanced features like dynamic lookup tables.
Based on the analysis above, the simple conclusion is that there are no clear winners to the title “an all-in-one open source SIEM solution.” When implementing a SIEM system based on the solutions above, you will most likely find yourself limited as far as functionality is concerned or combined with additional open source tools.
Open source tools used for SIEM are versatile and powerful. But, they require a great deal of expertise, and above all — time to deploy properly. It is for this reason that commercial offerings still dominate the SIEM landscape, even when open-source tools lie at the core of those commercial offerings.