Integrating OpsGenie with the Logz.io ELK Stack

opsgenie integration logz.io

OpsGenie is an incident management platform that helps DevOps teams efficiently manage the alerts being triggered in their environments.

Integrating with Logz.io, OpsGenie can also act as a dispatcher for alerts triggered by log messages and can help in determining the right team members to notify based on on-call schedules, notifying them using email, text messages (SMS), phone calls, and mobile push notifications along with escalating alerts until the alert is finally taken care of.

Adding a New Integration in OpsGenie

To integrate OpsGenie with Logz.io’s alerting mechanism, you first need a webhook URL. So, before you begin to create a new alert and an OpsGenie endpoint in Logz.io, retrieve that URL from within OpsGenie.

Important note! If you want to customize the notification sent from Logz.io in any way, you will need to use OpsGenie’s default API.

In OpsGenie, open the Integrations → Add New Integrations page and then scroll down to select the Logz.io integration.

add new integration

The default configurations here will suffice in this case, but on this page and the Advanced Settings page you can fine tune the integration to suit your needs and preferences.

For example, you can select which team — and who on that team — will receive the alerts from Logz.io, and you can configure how the alert is displayed in OpsGenie. You can also disable the integration or suppress notifications.

The important piece for the integration is, of course, the webhook URL, which you need to remember for the next step. It will look something like this:

https://api.opsgenie.com/v1/json/logzio?apiKey=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

Save the new integration to see it listed under Configured Integrations.

Creating a New Endpoint in Logz.io

You can now use the OpsGenie webhook URL to create a new endpoint in Logz.io.

All the endpoints in Logz.io are managed on the Alerts → Alert Endpoints page. To add a new endpoint, just click New Endpoint.

add new endpoint

Configuring the new endpoint is simple.

After selecting Custom from the Type drop-down menu, fill in the following parameters:

  • Name and description — a descriptive name and description for the new endpoint
  • URL — the webhook URL you retrieved from OpsGenie
  • Method — select the POST method
  • Headers — not necessary in our case
  • Body — not necessary in our case

Save the new endpoint. It will be displayed on the Alert Endpoints page.

Creating a New Log-based Alert in Logz.io

The last and final step is to test the new integration. To do this, we will create a query-based alert in Logz.io that will use the new endpoint to send it off to OpsGenie for incident management.

In the case of this tutorial, we are monitoring Apache access logs and analyzing 4xx and 5xx response errors:

type:apache_access AND response:[400 TO *]

apache access response codes

Clicking Create Alert opens up the Create New Alert wizard, where the Kibana query is already loaded. (But you can edit the query further if you like.)

Complete the conditions that will trigger an alert. (For example, you can set a threshold that, if passed, will trigger an alert.)

create new alert

In the next step, enter a name and description for the alert and set the severity level. While the description is optional, I highly recommend you give a meaningful description — this will help you manage your alerts in OpsGenie.

Last but not least, for the third and final step, enter how you want to be notified. You can enter an email address, but in this case, select the new custom endpoint that we created in the previous step.

alert endpoint

Complete the wizard by saving the new alert. It is added to the Alert Definitions page, and the Logz.io alert engine will check once a minute whether the threshold that you configured has been surpassed.

Once triggered (and if all was configured correctly), you will see your alert in the Alerts page in OpsGenie.

Important note! If you want to customize the notification sent from Logz.io in any way, you will need to use OpsGenie’s default API.

alert triggered

Open it to see more details. You’ll notice that Logz.io only provides up to five samples of the event to avoid too much-alerting noise. Of course, if you want to take a look at all the incidents you can use Kibana.

Using OpsGenie, you can now manage the event — address it, assign it to other team members, and so forth. OpsGenie also provides additional tagging and notation option that can help you manage the incident more effectively.

Summary

Log-based alerting provides the benefit of being notified, in near real-time, when a specific event is taking place in your environment. Because logs represent raw output for processes, they are often also more accurate as a base for alerts compared to monitoring tools.

However, combining the ELK Stack and incident management platforms such as OpsGenie does not excuse you from building an alerting mechanism carefully. Otherwise, you will most likely be fighting alert fatigue instead of the critical events that will impact your business. More about this in a future post.

Logz.io is Proud to Offer a Variety of Integrations with Many of your Favorite DevOps Tools!

Artboard Created with Sketch.

Leave a Reply

Your email address will not be published. Required fields are marked *

×

Turn machine data into actionable insights with ELK as a Service

By submitting this form, you are accepting our Terms of Use and our Privacy Policy

×

DevOps News and Tips to your inbox

We write about DevOps. Log Analytics, Elasticsearch and much more!

By submitting this form, you are accepting our Terms of Use and our Privacy Policy
× Book time with us at re:Invent here! Book