[Note: See our prior post that announced the Logz.io integration with Slack and Webhooks.]
Slack is one of the most popular online collaboration and messaging apps. It is practically taking over the world and, in many cases, replacing a lot of e-mail. Why, you may ask? Well, it is very simple to use, and it actually brings back memories of IRC — yes, that old way of communication from many years ago. But Slack is IRC on steroids.
So, why use Elasticsearch and Slack together? If a lot of communications are moving through Slack, would it not be great to analyze all of the data going through that medium? In this guide, we will show how to make use of all of this information by connecting your Slack rooms and all of the content inside them to the ELK Stack, which is comprised of Elasticsearch, Logstash, and Kibana.
There are two ways to integrate Slack and ELK. The first and more robust way is to leverage the IRC feature in Slack. This will assure that all of the data from Slack is streaming to your ELK installation, but this will require you to install and configure Logstash. The second method is to leverage the outgoing webhooks from Slack. This will limit the tracking to specific channels or messages, but you can integrate it with ELK without the need to run anything else.
The First Method: Enable IRC in Your Slack Environment
There are a number of steps needed to make this work.
First, Slack provides an IRC and XMPP service gateway that will allow you to access Slack through other means in addition to the Slack client. Your team owner will first need to enable team-wide gateway access at my.slack.com/admin/settings in the Gateways section under the Permissions tab.
Note: The gateway is disabled by default because enabling this option allows third-party applications (over which Slack has no control) to access your channel and data. It is highly recommended to enable only the gateways that you need and then confirm and evaluate the security of the clients that you will be using to connect to these channels.
You will find the option to enable the IRC gateway under your Slack Team Settings at the bottom of the Permissions tab.
Note: This should be enabled for your own team. All links to a team name in this post are fictitious.
Enable the IRC gateway, allow non-SSL IRC connections, and save the settings:
Once that has been enabled, go to the Gateway Settings page. You will find the information that is needed to connect to Slack:
To allow Logstash to scrape the information from the Slack channel(s), you will need to define an IRC input plugin. The following configuration will allow you to connect to your Slack team and your defined channels:
# The host and channels settings are required by Logstash
host => "my-team.irc.slack.com"
channels => [ "#logstash", "#random" ]
# The following settings are required by Slack
nick => "myuser"
user => "muser"
password => "my-team.JIWR4gtBI12TRXDNW29X9"
real => "Logstash Integration"
secure => true
# It's a good idea to add the type for Elasticsearch
type => "slack"
Of course, the output Elasticsearch plugin configuration should be set up as well:
hosts => ["elk.mydomain.com:9200"]
You will need to restart your Logstash daemon for these changes to take effect:
service logstash restart
Testing Your Configuration
To see if your configuration is correct, log into Slack and start sending messages:
In your Kibana logs, you will see that chat messages are now available:
As you can see, the message that was received has been indexed correctly with the proper structure:
The Second Method: Use Slack Outgoing Webhooks
Another way to forward messages from a Slack channel is to use the built-in Outgoing Webhook feature in Slack. Outgoing Webhooks allow you to listen for triggers in Slack chat messages — such as specific words or all chat in a specific channel — and when a message matching the appropriate filter is found, Slack will forward the relevant data to external URL(s) in real-time.
Outgoing Webhooks will only be triggered when one or both of the following conditions are met:
- The message is on the specified channel
- The message begins with one of the defined trigger word(s)
External Webhooks can be found in the settings of your Channel under Custom Integrations > Outgoing Webhooks. Choose “Add Configuration.”
Then, click “Add Outgoing Webhooks Integration”:
With this, you can specify a desired channel and keywords and have the webhook send the appropriate messages to your ELK Stack.
In this case, we are going to configure messages from the #webhook channel:
To send the messages, you will need to direct the traffic to a web server that can forward the messages to ELK.
Next, complete the information and customization of the Webhook:
Finally, click “Save Settings.”
Now, it is time to test your configuration.
We can see that a Webhook was added to the channel (Slack is kind enough to tell us):
When you send a message to the channel, it will be forwarded to your ELK server.
You will notice that format of the received message is not the same as the previous method — it will require some additional work to recognize all of the fields and the messages that are sent from Slack:
Slack-ELK Stack Integration Use Cases
There are numerous benefits of integrating Slack and the ELK Stack. Here are two of them.
Data Mining & Trend Analysis
A chat room is exactly what it sounds like — a place where people come to talk. It’s valuable to understand what is actually happening in your rooms.
Some basic use cases:
- Who is the most active user? That would be a simple query to find who has sent the most messages. You could then reward them with recognition for their contributions.
- When is the channel active? This would give you insight into in which timezones your channels are active or when people are actually involved. This could be used to align your support efforts based on the times when people are actually there and need responses.
- What are people actually talking about? This would require you to scan the messages and determine the most common words or phrases in your channels. This business intelligence could lead to new opportunities or identify pain points that people are experiencing.
ChatOps is a collaboration model that brings people, tools, process, and automation together into a transparent workflow. This flow connects the work needed, the work happening, and the work is done in a persistent location staffed by the people, bots, and related tools — all in a single chat room.
Many companies are already using Slack for ChatOps, and analyzing that information with ELK can help you to understand your business better.
Some basic uses:
- Analyzing when in the day/week/month your code commits are happening. If you configure a webhook to post a commit message to Slack, then you will be able to analyze this data upon code-checking
- Analyzing if your build time has changed over time. By sending a message to Slack at the start and end of your build process, you can measure and compare how your builds are performing over time and see — if needed — what is causing things to slow down.
For more information on ChatOps, I will refer you to this InfoWorld column by Logz.io co-founder and CEO Tomer Levy.
A Final Note
The amount of information in the world is growing exponentially, so we have to find new ways to understand, keep up with, and maximize the value of this data.
Chat platforms are becoming an integral part of day-to-day work. The ELK Stack can analyze the data within these channels to help your business to perform better. The examples above show some of the ways to use such an integration so that your everyday tools, can make your job not only more interesting but also more enjoyable.