A Beginner’s Guide to Elasticsearch Plugins

Elasticsearch plugins are used to extend the basic Elasticsearch functionality in various, specific ways. There are types, for example, that add security functionality, discovery mechanisms, and analysis capabilities to Elasticsearch.

This post provides some basic tips and tricks on working with Elasticsearch plugins from how to install them to management best practices.

Installing Elasticsearch Plugins

Regardless of what functionalities they add, Elasticsearch plugins belong to either of the following two categories: core plugins or community plugins. The former is supplied as part of the Elasticsearch package and are maintained by the Elastic team while the latter is developed by the community and are thus separate entities with their own versioning and development cycles.

Installing core plugins is simple. In the example below, I’m going to install the X-Pack plugin. X-Pack extends Elasticsearch by plugging in Shield, Watcher, and Marvel — three plugins that prior to Elasticsearch 5.x were separate entities, each requiring separate installation and setup (commands apply to DEB/RPM installation):

cd /usr/share/elasticsearch

sudo bin/elasticsearch-plugin install x-pack

Please note that in the case of the X-Pack Elasticsearch plugin, you will need to compliment this step by installing the X-Pack plugin for Kibana as well (if it’s X-Pack you’re interested in, Elastic’s docs detail everything you need to know about setting it up).

Plugins must be installed on every node in the cluster, and each node must be restarted after installation.

Community plugins are a bit different as each of them has different installation instructions.

Some community plugins are installed the same way as core plugins but require additional Elasticsearch configuration steps.

Other community plugins are meant for integration with third-party tools. The New Relic plugin, for example, helps to monitor Elasticsearch instances using New Relic. There are countless of similar plugins and integrations out there, and how or if you use them depends, of course, on your needs.

Managing the Plugins

Here are some additional commands to use to manage your Elasticsearch plugins (commands apply to DEB/RPM installation):

Listing all loaded plugins

cd /usr/share/elasticsearch

sudo bin/elasticsearch-plugin list

Removing a plugin

cd /usr/share/elasticsearch

sudo bin/elasticsearch-plugin remove [plugin]

Add mandatory dependencies for plugins

Elasticsearch also allows you to define a dependency for a specific plugin, ensuring that a node will not start if the plugin is missing. To add the dependency, add the following setting to the ‘elasticsearch.yml’ file:

plugin.mandatory:[plugin]

Elasticsearch Site Plugins

A number of community plugins add graphical front-ends to Elasticsearch.

Called “site plugins,” these extensions allow users to write JavaScript applications that can be used for monitoring and managing your Elasticsearch clusters and indices — an aspect that many in the community feel is missing in Elasticsearch as bundled out-of-the-box.

The most popular site plugins are Head, Bigdesk, HQ and Hopf. All provide the same functionality more or less — providing a nice overview of your Elasticsearch cluster, indices and documents. Some include additional utilities such as a data browser for building Elasticsearch queries.

It’s important to note that as of Elasticsearch 5.x, these site plugins are no longer supported due to security issues but there are workarounds for some of the plugins. (For reference, you can see our introduction to the ELK Stack 5.0 as well as our complete guide to the ELK Stack.)

Endnotes

As good as it is, Elasticsearch does not provide a full out-of-the-box experience for production cases, and plugins can be great ways to compensate for missing functionalities.

X-Pack is a good example of how plugins can be used to complement Elasticsearch by adding security and alerting layers to the basic package — but remember that unless you’re using the free tier, X-Pack comes with a cost.

In addition to X-Pack, there is also the EC2 Discovery plugin. The plugin basically uses the AWS API to discover other machines in a cluster. In large AWS deployments, this plugin is recommended instead of using multicast or manually configuring the master nodes for each data node.