application performaance

Application Performance Monitoring, aka APM, is one of the most common methods used by engineers today to measure the availability, response times and behavior of applications and services.

There are a variety of APM solutions in the market but if you’re familiar with the ELK Stack or are a Logz.io user, this article describes using a relatively new open source-based solution — Elastic APM.

What is Elastic APM?

As its name implies, Elastic APM is an application performance monitoring system which is built on top of the Elastic Stack (Elasticsearch, Logstash, Kibana, Beats). Similar to other APM solutions that you may have heard of, Elastic APM allows you to track key performance-related information such as requests, responses, database transactions, errors, etc.

The Elastic APM solution is comprised of 4 main building blocks, all open source — Elasticsearch for data storage and indexing, Kibana for analyzing and visualizing the data (the APM page in Kibana is provided as part of an X-Pack basic license), and two APM-specific components — the APM server and the APM agent.

APM agents are responsible for collecting the performance data and sending it to the APM server. The different agents are instrumented as a library in our applications. The APM server is responsible for receiving the data, creating documents from it and sending the data forth into Elasticsearch for storage

In this tutorial, I will describe how to set up Elastic APM for a basic node.js app on a single Ubuntu 16.04 machine on AWS. We’ll take a look at getting the data pipeline up and running, some of the visualization features, and for Logz.io users – how to ship the APM data into your Logz.io accounts.

You will need Elasticsearch and Kibana set up beforehand — please refer to our ELK guide for instructions.

Installing the APM Server

As explained above, the APM server collects the performance data tracked by the agents and forwards it to Elasticsearch.

To install the APM server using apt, we will use:

And, to start the server:

We can query Elasticsearch to make sure a new apm-server index is created:

Installing the APM agent and instrumenting our node.js app

Our next step is to create a simple demo app, install the APM agent for node.js and instrument it in our app’s index.js file.

First, let’s create the application folder and create a package.json file for tracking dependencies.

This last command will result in a series of questions and we can just press enter to use the default settings

To install express and add it to our package.json, we’ll use:

While we’re at it, let’s also install the APM agent:

Next, we will create the app’s main file which we call index.js (sometimes this file is called server.js or app.js):

We need to start the APM agent before requiring any other modules in our node.js app, so at the very top of our new file we’ll insert the following snippet:

We then paste some basic ‘hello world’ express code:

To start the server, we will use:

The output we get shows our node server running as expected:

To simulate some transactions, we can browse to port 3000 on the server we installed the app:

hey

Analyzing in Kibana

So, we’ve installed all the building blocks of our APM solution. It’s time for analyzing the data and visualizing it in Kibana. You can do this just like you’d analyze any other data in Kibana — using querying and visualizations. The APM page described below is another option but is provided with the X-Pack basic license.  

First thing is first though, we need to define the new index in Kibana. Querying Elasticsearch to list indices, we can see that two apm-* indices have been created:

There are two ways of loading these indices in Kibana — one is via the Index Patterns tab on the Management page in Kibana.

create index pattern

The other way is using the APM page where clicking on Setup Instructions displays a tutorial detailing some of the instructions provided above. This option is a good way to verify that the APM server and agent are working as expected.

At the very bottom of the page there is an option to Load Kibana objects. Using this option will load the index patterns, visualizations, and dashboards.

agents status

Once you’ve defined the index patterns, you will see your transactions data as well as exceptions (if you have any) in the discover tab.

apm

On the APM page, you will see the ‘demoapp’ service we defined in the index.js file listed under Services.

Clicking it open up a dashboard displaying details on requests made to the app (response times and requests per minute).

demo app

You can click on a request to get more details on the request, the response, as well as system and user related info. Again, this APM page is provided as part of a basic license of X-Pack but you can, of course, build your own detailed dashboard in Kibana.

Shipping APM data into Logz.io

For Logz.io users, shipping the performance data above into a Logz.io account is easy and requires adjusting the APM server configuration file at /etc/apm-server/apm-server.yml.

First, we’re going to download the required SSL certificate:

We will then copy it into the correct location:

We can now edit the APM server configuration file:

We will then add the following snippets (detailed instructions are available within Logz.io, under Log Shipping → Beats):

 

Be sure to keep to YAML syntax and comment out the default Elasticsearch output. Your account token can be found under Settings → General.

Restarting the APM server, you should begin to see the transactions appear in Logz.io:

logzio

Again, using Kibana’s analysis features you can create a bunch of visualizations for monitoring the performance of your apps and services. Here’s a simple data table visualization that gives you a breakdown of service transactions with info on response times:

response times

Summing it up

I only touched upon the core functionality of Elastic APM with a simple node.js app. Monitoring the performance of a full-fledged application, with multiple services, is an entirely different story of course, but Elastic APM should be considered a good open source alternative to other commercial APM tools in the market.

Those accustomed to working with Elasticsearch, Kibana and Beats will find it especially easy to install and handle. Setup is easy and the default Kibana objects make it easy to get started. To those new to the ELK Stack, however, I would first familiarize myself with some basic concepts before diving into Elastic APM.

Logz.io users — we will describe how to use this integration to perform APM in Logz.io in a future article.

Easily monitor your environment with Logz.io's ELK as a service.