Setting Up Application Performance Monitoring with the ELK Stack and Logz.io

application performaance

Application Performance Monitoring, aka APM, is one of the most common methods used by engineers today to measure the availability, response times and behavior of applications and services.

There are a variety of APM solutions in the market but if you’re familiar with the ELK Stack or are a Logz.io user, this article describes using a relatively new open source-based solution — Elastic APM.

What is Elastic APM?

As its name implies, Elastic APM is an application performance monitoring system which is built on top of the Elastic Stack (Elasticsearch, Logstash, Kibana, Beats). Similar to other APM solutions that you may have heard of, Elastic APM allows you to track key performance-related information such as requests, responses, database transactions, errors, etc.

The Elastic APM solution is comprised of 4 main building blocks, all open source — Elasticsearch for data storage and indexing, Kibana for analyzing and visualizing the data (the APM page in Kibana is provided as part of an X-Pack basic license), and two APM-specific components — the APM server and the APM agent.

APM agents are responsible for collecting the performance data and sending it to the APM server. The different agents are instrumented as a library in our applications. The APM server is responsible for receiving the data, creating documents from it and sending the data forth into Elasticsearch for storage

In this tutorial, I will describe how to set up Elastic APM for a basic node.js app on a single Ubuntu 16.04 machine on AWS. We’ll take a look at getting the data pipeline up and running, some of the visualization features, and for Logz.io users – how to ship the APM data into your Logz.io accounts.

You will need Elasticsearch and Kibana set up beforehand — please refer to our ELK tutorial for instructions.

Installing the APM Server

As explained above, the APM server collects the performance data tracked by the agents and forwards it to Elasticsearch.

To install the APM server using apt, we will use:

sudo apt-get install apm-server

And, to start the server:

sudo service apm-server start

We can query Elasticsearch to make sure a new apm-server index is created:

curl -XGET 'localhost:9200/_cat/indices?v&pretty'


health status index                           uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .kibana                         7CdGpSpuTDqSpzDGfV5XmQ   1   0         17            1     58.8kb         58.8kb
yellow open   apm-6.4.3-onboarding-2018.11.12 4XSxUrWnQryD9yRCwPmCRQ   1   1          1            0      5.9kb

Installing the APM agent and instrumenting our node.js app

Our next step is to create a simple demo app, install the APM agent for node.js and instrument it in our app’s index.js file.

First, let’s create the application folder and create a package.json file for tracking dependencies.

sudo mkdir /demoapp
cd /demoapp
npm init

This last command will result in a series of questions and we can just press enter to use the default settings

To install express and add it to our package.json, we’ll use:

npm install express --save-dev

While we’re at it, let’s also install the APM agent:

npm install elastic-apm-node --save

Next, we will create the app’s main file which we call index.js (sometimes this file is called server.js or app.js):

sudo vim index.js

We need to start the APM agent before requiring any other modules in our node.js app, so at the very top of our new file we’ll insert the following snippet:

var apm = require('elastic-apm-node').start({
  serviceName: 'demoapp',
  serverUrl: 'http://localhost:8200'
})

We then paste some basic ‘hello world’ express code:

const express = require('express')
const app = express()
app.get('/', (req, res) => {
 res.send('HEY!')
})
app.listen(3000, () => console.log('Server running on port 3000'))

To start the server, we will use:

node index.js

The output we get shows our node server running as expected:

Server running on port 3000

To simulate some transactions, we can browse to port 3000 on the server we installed the app:

hey

Analyzing in Kibana

So, we’ve installed all the building blocks of our APM solution. It’s time for analyzing the data and visualizing it in Kibana. You can do this just like you’d analyze any other data in Kibana — using querying and visualizations. The APM page described below is another option but is provided with the X-Pack basic license.  

First thing is first though, we need to define the new index in Kibana. Querying Elasticsearch to list indices, we can see that two apm-* indices have been created:

curl -XGET 'localhost:9200/_cat/indices?v&pretty'

health status index                            uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   apm-6.4.3-transaction-2018.11.13 ikIjni6qSDmgix2K9zC8vQ   1   1          5            0       37kb           37kb
yellow open   apm-6.4.3-onboarding-2018.11.13  -boQQic_S2ePPPtJ3HP83w   1   1          1            0      5.9kb          5.9kb
green  open   .kibana                          m4IxzDFrT9CkiB8EqpD3Kw   1   0         17            1     49.3kb         49.3kb

There are two ways of loading these indices in Kibana — one is via the Index Patterns tab on the Management page in Kibana.

create index pattern

The other way is using the APM page where clicking on Setup Instructions displays a tutorial detailing some of the instructions provided above. This option is a good way to verify that the APM server and agent are working as expected.

At the very bottom of the page there is an option to Load Kibana objects. Using this option will load the index patterns, visualizations, and dashboards.

agents status

Once you’ve defined the index patterns, you will see your transactions data as well as exceptions (if you have any) in the discover tab.

apm

On the APM page, you will see the ‘demoapp’ service we defined in the index.js file listed under Services.

Clicking it open up a dashboard displaying details on requests made to the app (response times and requests per minute).

demo app

You can click on a request to get more details on the request, the response, as well as system and user related info. Again, this APM page is provided as part of a basic license of X-Pack but you can, of course, build your own detailed dashboard in Kibana.

Shipping APM data into Logz.io

For Logz.io users, shipping the performance data above into a Logz.io account is easy and requires adjusting the APM server configuration file at /etc/apm-server/apm-server.yml.

First, we’re going to download the required SSL certificate:

wget 
https://raw.githubusercontent.com/logzio/public-certificates/master/COMODORSADomain
ValidationSecureServerCA.crt

We will then copy it into the correct location:

sudo mkdir -p /etc/pki/tls/certs
sudo cp COMODORSADomainValidationSecureServerCA.crt 
/etc/pki/tls/certs/

We can now edit the APM server configuration file:

sudo vim /etc/apm-server/apm-server.yml

We will then add the following snippets (detailed instructions are available within Logz.io, under Log Shipping → Beats):

 

fields:
  logzio_codec: json
  token:<yourAccountToken>
fields_under_root: true

output.logstash:
  enabled: true
  hosts: ["listener.logz.io:5015"]
  ssl.certificate_authorities: ['/etc/pki/tls/certs/COMODORSADomainValidationSecureServerCA.crt']

Be sure to keep to YAML syntax and comment out the default Elasticsearch output. Your account token can be found under Settings → General.

Restarting the APM server, you should begin to see the transactions appear in Logz.io:

logzio

Again, using Kibana’s analysis features you can create a bunch of visualizations for monitoring the performance of your apps and services. Here’s a simple data table visualization that gives you a breakdown of service transactions with info on response times:

response times

Summing it up

I only touched upon the core functionality of Elastic APM with a simple node.js app. Monitoring the performance of a full-fledged application, with multiple services, is an entirely different story of course, but Elastic APM should be considered a good open source alternative to other commercial APM tools in the market.

Those accustomed to working with Elasticsearch, Kibana and Beats will find it especially easy to install and handle. Setup is easy and the default Kibana objects make it easy to get started. To those new to the ELK Stack, however, I would first familiarize myself with some basic concepts before diving into Elastic APM.

Logz.io users — we will describe how to use this integration to perform APM in Logz.io in a future article.

Easily monitor your environment with Logz.io's ELK as a service.
Artboard Created with Sketch.
×

Turn machine data into actionable insights with ELK as a Service

By submitting this form, you are accepting our Terms of Use and our Privacy Policy

×

DevOps News and Tips to your inbox

We write about DevOps. Log Analytics, Elasticsearch and much more!

By submitting this form, you are accepting our Terms of Use and our Privacy Policy