These 15 Tech Companies Chose the ELK Stack Over Proprietary Logging Software

As part of the widespread movement towards open source software, the ELK Stack is rapidly moving from a niche platform to the most common log management platform in the world.

A recent Black Duck Software study surveyed 1,300 individuals ranging from business analysts to CEOs to software engineers on the future of open source. Of the respondents, 78% reported that either part or all of their companies’ operations run on open-source software. In addition, 66% stated that their companies use open source in developing software for customers.

The ELK (Elasticsearch, Logstash, and Kibana) Stack is an example of the trend towards open source that has disrupted commercial proprietary markets (including, in that example, Splunk). ELK was created in 2010 and has already been adopted by well-known organizations such as LinkedIn, Netflix, and Stack Overflow. ELK supports many different log management and analysis use cases including typical IT operations, customer support, website traffic, business intelligence, security events, and user behavior.

To give you a better understanding of the widespread adoption of ELK in various industries, I have compiled the following list that shows how various companies are using ELK right now.

Netflix

As the Amazon Web Services poster child born in the cloud, Netflix heavily relies on ELK within various use cases to monitor and analyze customer service operations and security logs. The company chose Elasticsearch for its automatic sharding and replication, flexible schema, nice extension model, and ecosystem with many plugins. Netflix’s use of Elasticsearch to store, index, and search documents have grown from a couple of isolated deployments to more than fifteen clusters comprised of nearly 800 nodes that are centrally managed by a cloud database engineering team.

Stack Overflow

According to Stack Overflow’s VP of Engineering, the company’s websites sit on top of a monolithic architecture that uses C# and MSSQL. However, Stack Overflow uses Elasticsearch as a means to support full-text search capabilities. Furthermore, each Elasticsearch box also has 300 GBs also using SSDs, the search box is powered by a REST interface. Stack Overflow is using Elasticsearch because it performs better on SSDs and that Lucene.net could not handle the company’s workflows as a result of locking issues.

LinkedIn

LinkedIn has a well-known ELK adoption story. The business-focused social network uses ELK to monitor performance and security. The IT team integrates ELK with Kafka to support their load in real time. Their ELK operations include more than 100 clusters across more than twenty teams and six data centers.

Fujitsu (OpenStack Cloud)

Fujitsu has an interesting way of monitoring its private OpenStack cloud. It uses ELK to analyze the logs of Monasca (OpenStack’s open source monitoring project). The company states that “the technical basis of our work consists of Elasticsearch, Logstash, and Kibana.”

Accenture

Accenture is one of the largest IT consulting service companies in the world, so it is only natural for the company to lead ELK implementation projects. In that linked presentation, company staffer Alexander Szalonnas states that the company prefers the ELK Stack to Splunk because it is open source, has a simple web interface, and can use plugins to extend its functionality.

Tripwire

Tripwire is a worldwide SIEM (Security Information Event Management) leader, and Big Data analytics is critical for vendors such as the company. Tripwire uses ELK to support information packet log analysis.

Medium

Medium is one of the most popular modern blog-publishing platforms. Every month, its stack supports 25 million unique readers as well as tens of thousands of published posts each week. According to Medium’s engineering team, ELK is used to debug production issues. The company also uses the Elasticsearch, Logstash, and Kibana stack to detect DynamoDB hotspots.

Swat.io

Swat is a popular social management system that supports large enterprise marketing teams. According to its operations engineer, Swat uses ELK to log the AWS S3 storage that stores its sites’ traffic activity. This helps to control and forecast the growing cloud costs that are driven by new user demands.

IFTTT

IFTTT is a free web-based service that allows users to create chains of simple conditional statements. The IFTTT operations team uses Elasticsearch for real-time monitoring and receiving alerts on API events. They also use Kibana to visualize worker processes and see API performance in real-time.

HipChat

HipChat is an Atlassian tool that is well-known for its internal and private enterprise chat services. According to High Scalability, HipChat’s chat services doubles its user base every few months. The company’s use of Elasticsearch as a search backend allows for horizontal scalability, which supports HipChat’s overall growth. HipChat states that they moved to Elasticsearch because it can process any amount of data, it is multi-tenant, and it can handle a node loss transparently with sharding and replication.

Five Additional Online Services

The 10 best-known ELK use cases above represent only a tiny few of the millions of other users that exist. For some variety in use cases, here are a few more interesting examples that we have come across in our research: Ideato, MixRadio, Vinted, Auth0, and Rancher.

Looking for more? Just look up “ELK” or “Elasticsearch” on sites such as Medium, High Scalability, SlideShare, and our complete guide to the ELK Stack to read even more real-life stories and see how ELK can support your own specific use case.