jenkins build monitoring

Jenkins is an extremely popular, open-source, continuous integration tool used for running tests, building code, and subsequently pushing to staging and then production. 

In a previous post, I outlined instructions for collecting, analyzing and visualizing Jenkins system logs. Jenkins system logs can be useful to monitor the general health of a Jenkins setup, especially in the case of a multi-node environment and I highly recommend exploring this option as outlined in the above-mentioned article, but in this piece, I will be focusing on Jenkins build logs.

Jenkins build logs contain a complete record of an execution’s output, including the build name, number, execution time, result, and more. If your pipeline is broken, this data can provide a wealth of information to help troubleshoot the root cause. Jenkins supports console logging but in case of a large amount of running jobs, it becomes difficult to keep track of all the activity–so collecting all this data and shipping it into the ELK Stack can help to give you more visibility.

Installing Jenkins

As a first step, and for those just getting started, let’s review how to install and set up a single Jenkins server. If you already have Jenkins up and running, skip right to the next step.

Jenkins can be installed in a variety of different ways, depending on your operating system and environment. In this case, I’ll be installing Jenkins using Ubuntu packages. I also recommend checking out system requirements before beginning the process.

Start by adding the repository key:

Next, add the package repository address to your ‘sources.list’:

Run update so you can use the new repository:

To install Jenkins and its dependencies (this includes Java 8 also required for running Elasticsearch), use:

Start the Jenkins server using:

To open Jenkins, open your browser and enter the following URL:

You will then be required to enter an initial admin password available in the default installation directory. For Linux:

Follow the rest of the setup steps (installing plugins and creating a new admin user), and you should be all set and ready to go.

welcome to jenkins

Integrating with the ELK Stack

The integration with the  ELK Stack, either your own deployment or Logz.io (as demonstrated below) is done using a fork of a Jenkins plugin called logstash-plugin. The next step describes how to download, build, and install this plugin.

First, clone the plugin:

proxy configuration

Next, use maven to build it:

The building process takes a while, so be patient. Tip – if you don’t have Maven installed, you can use this Docker to run the build:

The end-result of this process is a logstash.hpi file located within the plugin directory at: logstash-plugin/target

Open Jenkins, and open the Advanced tab on the Manage Jenkins –> Manage Plugins page.

proxy configuration

Upload the logstash.hpi file in the Upload Plugin section. Jenkins will display a success message when done.

list

Select the Restart Jenkins checkbox to apply the changes.

Once Jenkins finished to reinstall, open the Manage Jenkins → Configure System page.

A section called Logstash appears in the middle of the page. Select the Enable sending logs to an indexer checkbox to open the configurations.

jenkins location

If you’re shipping to your own ELK deployment, enter the IP of your Elasticsearch instance and any authentication details if necessary. To ship to Logz.io, open the Indexer type drop-down, and select Logz.io.

Enter the following details:

  • Logz.io host – enter the URL of the Logz.io listener. If you are in the EU region insert https://listnereu.logz.io:8071. Otherwise, use https://listner.logz.io:8071. You can tell which region you are in by checking the login url. If your environment says app.logz.io then you are in the US, if it says app-eu.logz.io then you are in the EU.
  • Logz.io key – Your Logz.io token. It can be found in your Logz.io app account settings.

logstash

Click Save to apply the configurations.

Verifying the pipeline

Now that we have the plugin installed and configured, it’s time to test that the integration with Logz.io is working and that build logs are actually indexed properly.

To test the pipeline, I will create a simple item in Jenkins that executes a bash script.

I have two options of sending the build logs to Logz.io, either line by line as the logs are generated by the build or in bulk, post-build.

To send the logs to Logz.io line by line, simply select the Send console log to Logstash checkbox in the item’s General section. In this case, however, I’m going to send the logs in bulk post-execution.  

In the Post Build Actions section at the end of the configurations, open the Add post-build action in the drop-down menu, and select Send console log to Logstash. You can then configure how many lines to send to Logz.io. To send all the data, enter ‘-1’.

post build actions

That’s it. Save the configuration, run your build and within a few seconds you should be seeing build console logs in Logz.io.

build console

Analyzing Jenkins build logs in Kibana

Now that the logging pipeline is up and running, it’s time to look into the data with some simple analysis operations in Kibana.

I like adding some fields to the main display area in Kibana to get some visibility into the data. Adding, for example, the ‘buildNum’, ‘projectName’ and ‘result’ fields helps to give us some context.

context

We can use a field-level Kibana query to look for specific builds, say failed builds:

failed builds

Things get more interesting when visualizing the data. Using Kibana’s different visualizations you can create a series of simple charts and metric visualizations to get you a nice overview of your Jenkins builds.

Let’s start with a simple metric visualization showing the number of failed vs. successful builds:

number of failures

Or, for example, you can create a pie chart displaying a breakdown of the build results:

pie

Or, for example, you can create a pie chart displaying a breakdown of the build results:

line

Once you’ve got your visualizations lined up, you can add them into one comprehensive dashboard.

dashboard

Bt the way, this dashboard is available in Logz.io’s dashboard and visualization library, ELK Apps so if you’re shipping you can hit the ground running by installing this dashboard instead of building your own from scratch.

Endnotes

If your Jenkins build pipelines are busy, visibility becomes an issue. In a microservices environment, with multiple Jenkins jobs running continuously, monitoring and troubleshooting failed builds is a challenge.

The benefit of using a centralized logging system is the ability to not only collect and store the data in one single location but use best-in-class analysis and visualization tools to drill down to the root cause of failed builds. The Logstash plugin used here is an easy way to integrate a Jenkins deployment with the ELK Stack to enjoy these benefits.

Get important insights from your Jenkins build logs with Logz.io.