openstack and elk stack

OpenStack, which started in 2010 as a collaboration between Rackspace Hosting and NASA, is an open-source platform for cloud computing that is usually deployed as infrastructure as a service. As the project has grown, so has the number of users. Notable ones include PayPal, CERN, Intel, Sony, and Wikimedia Labs.

Why Deploy ELK on OpenStack?

There are many reasons to install the ELK Stack in an OpenStack environment, and they are generally related to three components within OpenStack itself.

Cinder (volume storage)

OpenStack has a volume storage service named Cinder. When you take advantage of this service, adding a new volume to your ELK Stack is simple. All you need to do is create a new volume and attach it to an instance. The data is persistent, so if you lose a single instance, the data will not be lost and will be able to be recovered and attached to another instance instead.

Neutron LBaaS (load balancing as a service)

Load balancing is a natural way to scale your environment by putting load balancers in front of your Logstash and Kibana front ends.

Heat (orchestration)

Heat is the orchestration engine for OpenStack. With Heat, you can automatically deploy almost all of the components that are needed for an ELK cluster. Heat has a lot of advantages such as the fact that it is a native tool that hooks into all of OpenStack’s APIs — which is great when you want to utilize OpenStack’s built-in functionality.

But just remember: Heat does not provide much-added benefit to the software installations inside provisioned instances. Software installations can and should be managed by a configuration management suite of tools.

The Step-By-Step Guide to ELK on OpenStack

Below is a basic example of how to deploy the ELK stack in a single machine that is attached to a Cinder volume (which can be used for data persistence or higher throughput as long as the underlying infrastructure has that option).

In this example, we will use an Ubuntu 14.04 server as our base image, assuming that the following already exists in your OpenStack environment:

  • Image-name – ubuntu_14.04
  • Flavor – x1
  • Network – network_id
  • Keypair – my_keypair
  • Security group – SSH (22) and ping (ICMP) ingress are allowed, outgress all ports are allowed

You will have to have the correct credentials to interact with OpenStack services, which are usually kept in environment files that look something like the following:

The file has a username, tenant name, password, and URL endpoint for Keystone, which will allow you to interact with OpenStack.

Source the file:

Create a new volume:

Boot an image and attach the volume to the instance:

Once the instance is deployed and up and running, SSH into the newly deployed instance:

Update the software on the instance:

Mount the additional volume under /opt.

Install Open JDK:

Install Elasticsearch:

Test that Elasticsearch is available by running the following:

You should get a result similar to the one below:

Install Logstash:

Redirect your system logs to Logstash:

Create the file /etc/logstash/conf.d/10-syslog.conf to pipe all of your logs into Logstash:

Add the Logstash user to the adm group:

Restart the Logstash service:

Install Kibana:

Move everything to /opt (which is the Cinder volume):

Now try to access your Kibana instance by opening your browser and going to: http://YOUR_ELASTIC_IP:5601

It will fail.

The reason being is that the security group rules do not allow traffic to this port.

Create the appropriate security group and its rules:

Now you should be able to access Kibana to see the logs. First, configure an index pattern, then you can start browsing your logs.

Summary

In general, the amount of information in data centers and the cloud is constantly rising. As a result, it is increasingly difficult and cumbersome to manage, collect, organize, and analyze everything that is logged in systems. Therefore, you should store your data in a central location such as an ELK Stack for security purposes, analytics, data mining, and more.

Making use of the information, tools, and deployment example above, as well as the additional tools that are available to you in OpenStack, will allow you to navigate the vast amount of information in the cloud.

Easily Configure and Ship Logs with Logz.io ELK as a Service.