Heroku log analysis with elk

Heroku is a cloud PaaS that allows developers to create and deploy applications without having to worry about all the underlying infrastructure. Heroku supports applications written in Ruby, Python, node.js, PHP, Java, Scala, and Go and packs scores of third-party add-ons and complementary services.

The rich and ever-growing ecosystem, scalability, and overall simplicity of Heroku are all reasons why the platform is rapidly gaining popularity with companies of all sizes and independent developers alike.

Heroku’s logging system is extremely robust, aggregating application, system, and API logs into one central stream and allowing you to hook into this stream using either add-on logging services or logging hooks called “drains.”

Following our recent announcement of the Logz.io add-on for Heroku, we will now describe in this tutorial how to ship and analyze Heroku application logs using the enterprise-grade ELK Stack provided by Logz.io — our predictive, cloud-based log management platform that’s built on top of ELK.

Prerequisites

To follow this tutorial, make sure that your environment has the following requirements:

Step 1: Getting ready

Let’s first set up our environment using the Heroku CLI and creating our sample PHP application. If any of the following components are already installed, skip to the next step.

Installing PHP

The application used in this article is a PHP application. If you intend on using a different application or already have PHP installed (Mac OS X comes with a built-in PHP stack), skip this step:

Installing Git

We’ll need Git to clone our sample application and then deploy it to Heroku:

Installing the Heroku Toolbelt

The Heroku Toolbelt allows you to use the Heroku CLI, which can be used in turn for managing your applications and added tools:

Step 2: Creating a new application

Our next step is to create a new application that will generate some logging data that we can ship and analyze. I chose to demonstrate the process using a PHP application, but you can, of course, use any application that you like.

First, log into Heroku with:

The Heroku toolbelt CLI tool will now be installed. This might take a few minutes.

You will be required to enter your account credentials — the e-mail that you used to register to Heroku and the password you defined.

We’re now going to clone a new application from GitHub. For the purpose of convenience, I’m going to use Heroku’s “Getting Started with PHP application” guide:

To create the application, cd into the folder and run:

Note, you can leave the app parameter out, and Heroku will pick a random name for your application for you.

Next, deploy your application using the git push command:

Your application is now deployed. To make sure an instance is running, run:

Visit the app at the URL displayed at the end of the deployment, for example:

https://logz.ioioapp.herokuapp.com/

getting started with php on heroku

Step 3: Analyzing Heroku logs

We’ve created and deployed a new PHP application on Heroku. Now, let’s see how to handle logging.

In Heroku, all logs created by your app and Heroku components are aggregated and collected into a single channel by a log delivery system called Logplex, which can be accessed via any of the logging add-on providers supported by Heroku or a custom log drain.

Using logs –tail

The easiest way to take a look at your Heroku events is to use the following log command:

Refresh your app in the browser to see the fresh logs added.

Using the ELK Stack

These logs will multiply and get more complicated as you develop your application, gaining real-time and actionable insights will become a business problem in no time.

The ELK Stack (an acronym for Elasticsearch, Logstash, and Kibana) is the world’s most popular open source log analytics platform and can now be installed on your Heroku application using the Logz.io add-on.

Logz.io provides the ELK Stack as an easy, secure and scalable service on the cloud, with a bunch of enterprise-grade features such as archiving, alerts, security and more. To send logs to Logz.io, we’re going to install our add-on, which is currently in Beta mode and completely free of charge.

You can do this via the Heroku dashboard if you like, but we’re going to use the following command in CLI:

Logz.io is added, and is now displayed in the Heroku dashboard under your application:

heroku add-on installed

It’s time to access your ELK stack — to do this, simply click Logz.io in the add-ons list and a new window is opened with your first Heroku logs displayed in Kibana:

heroku log analysis

Once your pipeline into the Logz.io ELK is established, you can start to query the data, create visualizations and dashboards. For an idea of how the ELK Stack can be used for log analysis, check out this video.

Use Logz.io to monitor and analyze your Heroku logs