Monitoring your Web Application on Apache with Logz.io

These days, more and more web applications are developed and refined to keep the customer engagement at the highest level possible. It is crucial to provide a smooth experience to the customer hence monitoring is of paramount importance. One key factor in that is monitoring the web server we use. In this article, we will explore Logz.io features by monitoring an Apache Web Server.

We will be using Ubuntu 20.04 LTS as a host in the examples and outputs.

Monitoring Web Servers for Web Applications

Web servers are the backbone of our web applications. Naturally, there are numerous metrics we can monitor here: CPU, memory usage, connections per second, etc. Additionally, we can gather from the logs browser type, country, operating system, HTTP verb used, and so on. All these valuable bits of data about our web server and customers need recording, analysis, and action when necessary. For this crucial process, we need to use an excellent monitoring stack.

Monitoring with Logz.io

There are numerous solutions for shipping, storing, visualizing, and analyzing these metrics and other information. One of the best solutions is the ELK Stack for logging, complemented by Prometheus for metrics and and alerts. Now, you can add Jaeger for distributed tracing into the mix. While Elasticsearch holds our most valuable information, Logstash is responsible for shipping and transformation of that data, and Kibana is for visualizing that logging data.

We will use Filebeat as our log shipper, and Prometheus as our metric collector and aggregator. 

Web Application Served by Apache

Apache is one of the most used web servers in the world, with extreme flexibility and a varied feature set. In this article, we will use Apache as the web server. Default settings for Apache are sufficient for our case. We will explore the flow information that we can digest from the system and Apache, including logs and various metrics:

• Apache
  • Access Logs
  • Error Logs
  • Metrics from Apache Server Status endpoint
• System
  • System Log Files
  • System Metrics

We will use two exporters for Prometheus, the Apache Exporter and Node Exporter. Prometheus will aggregate all the metrics and ship to our Logz.io Prometheus instance.

Installing Apache

Installing Apache web server from apt repositories is straight forward,  with apt install -y apache2 command.

Installing Filebeat

Filebeat is a log shipper agent, it monitors the log inputs defined and ships them to Logstash or Elasticsearch.

First, download Filebeat to your server:

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.10.0-amd64.deb

Install the package with the dpkg command:

sudo dpkg -i filebeat-7.10.0-amd64.deb

We can see the installation steps in the output as Filebeat successfully installs.

Now, we need to download the CA certificate. We will be using curl here:

Selecting previously unselected package filebeat.
(Reading database ... 81586 files and directories currently installed.)
Preparing to unpack filebeat-7.10.0-amd64.deb ...
Unpacking filebeat (7.10.0) ...
Setting up filebeat (7.10.0) ...
Processing triggers for ureadahead (0.100.0-19.1) ...
Processing triggers for systemd (229-4ubuntu21.29) ...

Now, we need to download the CA certificate. We will be using curl here:

sudo curl https://raw.githubusercontent.com/logzio/public-certificates/master/AAACertificateServices.crt --create-dirs -o /etc/pki/tls/certs/COMODORSADomainValidationSecureServerCA.crt

Edit the Filebeat configuration file with your favorite editor,. Here, for simplicity, we will use Nano:

sudo nano /etc/filebeat/filebeat.yml

Configuring Filebeat

Replace your Logs.io token below and paste into the editor.

filebeat.inputs:
- type: log
  paths:
  - /var/log/apache2/access.log
  fields:
    logzio_codec: plain
    token: xxxxxxxxxxxxxxxxxxxxxxxx
    type: apache_access
  fields_under_root: true
  encoding: utf-8
  ignore_older: 3h
- type: log
  paths:
  - /var/log/apache2/error.log
  fields:
    logzio_codec: plain
    token: xxxxxxxxxxxxxxxxxxxxxxxx
    type: apache_error
  fields_under_root: true
  encoding: utf-8
  ignore_older: 3h
  
filebeat.registry.path: /var/lib/filebeat
processors:
- rename:
    fields:
    - from: "agent"
      to: "filebeat_agent"
    ignore_missing: true
- rename:
    fields:
    - from: "log.file.path"
      to: "source"
    ignore_missing: true
​
output.logstash:
  hosts: ["listener-eu.logz.io:5015"]
  ssl:
    certificate_authorities: ['/etc/pki/tls/certs/COMODORSADomainValidationSecureServerCA.crt']

This configuration defines two inputs: Apache access logs and Apache error logs. Additionally, we have one output, which is Logstash in Logz.io.

To enable system module

sudo cd /etc/filebeat/modules.d

Change name of the default system.yml.disabled to system.yml.

sudo mv system.yml.disabled system.yml

Restart Filebeat for changes to take effect.

sudo systemctl restart filebeat

TL;DR Filebeat

Installing Prometheus

Installing Prometheus is straightforward using apt packages.

sudo apt install -y prometheus

An output similar to below is displayed.

Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  prometheus
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
...

Installing Prometheus Node Exporter

Prometheus Node Exporter collects metrics about your system and exposes these to Prometheus server.

Again, we use apt command to get the package installed:

sudo apt install -y prometheus-node-exporter

A similar output is displayed.

Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  prometheus-node-exporter-collectors
The following NEW packages will be installed:
  prometheus-node-exporter prometheus-node-exporter-collectors
...

Installing Prometheus Apache Exporter

Now we need to install an exporter to gather Apache metrics and allow Prometheus to scrape them.

sudo apt install -y prometheus-apache-exporter

Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  prometheus-apache-exporter
...

Next, let’s check our Prometheus listeners.

netstat -tulpen | grep prometheus

Prometheus Apache exporter is listening on 9117 as expected and Node Exporter listens on 9100.

tcp6       0      0 :::9117                 :::*                    LISTEN      114        11459019   61135/prometheus-ap 
tcp6       0      0 :::9090                 :::*                    LISTEN      114        11469554   63857/prometheus    
tcp6       0      0 :::9100                 :::*                    LISTEN      114        11489288   68358/prometheus-no 

TL;DR Prometheus Installation

Configuring Prometheus

First, we need to configure Prometheus to scrape the metrics from our exporters. For this, we define three jobs under the scrape_configs stanza: one for Apache exporter, one for Node exporter, and lastly one for Prometheus itself.

Additionally, we define remote_write to send metrics to Logz.io Prometheus-as-a-Service instances. After this, we will be able to graph our metrics in the Metrics UI.

Edit the Prometheus config file at /etc/prometheus/prometheus.yml:

global:
  scrape_interval:     15s
  evaluation_interval: 15s 
  
  external_labels:
      p8s_logzio_name: 'app_server_apache'
​
alerting:
  alertmanagers:
  - static_configs:
    - targets: ['localhost:9093']
​
rule_files:
​
scrape_configs:
  - job_name: 'prometheus'
​
    scrape_interval: 5s
    scrape_timeout: 5s
​
    static_configs:
      - targets: ['localhost:9090']
​
  - job_name: node
    static_configs:
      - targets: ['localhost:9100']
​
  - job_name: apache
    static_configs:
      - targets: ['localhost:9117']
​
remote_write:
 - url: https://listener.logz.io:8053
   bearer_token: xxxxxxxxxxxxxxxxxxxxxxxx
   remote_timeout: 30s
   queue_config:
     batch_send_deadline: 5s  
     max_shards: 10  
     min_shards: 1
     max_samples_per_send: 500 
     capacity: 10000

Visualizing Metrics with Logz.io

When we click the “Metrics” button, we are welcomed with a default dashboard, listing shortcuts to other pre-built dashboards and panels.

As you can see, Logz.io receives our Apache and system metrics, detailed in the “Doc Count Per Module” and “Doc Count Per Metrics” panels.

System Metrics

The built-in “System Metrics” dashboard aids us in monitoring our server in great detail. The dashboard includes graphs for CPU, memory, disk and network metrics.

Apache Metrics

We can visualize Apache server metrics both in Kibana and our Metrics UI, but first we will explore ELK Apps for Apache.

ELK apps are curated by Logz.io and include visualizations, dashboards and alerts. They offer best practices and some great ideas as well.

Click the “ELK Apps” button and type in “Apache” as a keyword Select Apache Access and hit Install.

After installation, go to “Kibana” and click “Dashboards” on the lefthand side menu (button has a graph icon). Click “Apache Access.” The subsequent dashboard organizes  several Apache access metrics.

This particular graph represents Apache access type by browser.

Another graph that “Apache Access” app generated for us is “Apache Access By OS..

Of course, we also have the flexibility to generate our own metric dashboards. Here, we can see scraped data from the Apache Status endpoint for use in further visuals.

Using these metrics we generate a dashboard quite easily. This custom dash makes use of the following metrics; “Apache Sent KB”, “Apache CPU Load”, “Apache Scoreboard”, “Apache Connections” and “Apache Accesses Total” metrics for this custom dashboard.

Live Tail of Logs

On “Live Tail” press play button (press ctrl+alt+s) to start the live tail and after a short time we will be able to see our Apache access, Apache errors and system logs. (for brevity following shows only Apache logs in real time).

Here we are filtering the live tail logs with the keyword “http”. This filtering ability is quite useful on troubleshooting live scenarios.

Logz.io Exceptions

Another great feature of Logz.io is that of “Exceptions.” Here, we can find a lot of important information gathered together. One of the messages catches our eye “error: maximum authentication attempts exceeded for root”, this shows us someone tried to authorize as root unsuccessfully.

Another great feature of Logz.io is that of “Exceptions.” Formally called the Insights Engine, it uses machine learning and analyzes the data in your logs from different perspectives.

You can see if there is any spike in received exceptions and whether there are new kinds of exceptions or not. Correlating exceptions finds the needles in haystacks by analyzing thousands of exceptions and then drilling down into them. Exceptions then lists them according to their context. Any exceptions that haven’t been seen before are highlighted, so information is easy to find. Also, you can ignore known and harmless exceptions as well.

Here, we can find a lot of important information gathered together. One of the messages catches our eye: error: maximum authentication attempts exceeded for root, this shows us someone tried to authorize as root unsuccessfully.

Conclusion

We have explored Apache and system observability, both logs and metrics, in great detail.  We have found correlations between system logs and system metrics via the Insights module and have also explored the Logz.io metrics UI by both re-using existing dashboards and also creating a dashboard for ourselves.

We were able to see our Apache web server’s CPU load, detailed information about connections, and visitor information like browser data, GEOIP (via IP Geoip databases), and other crucial data points for monitoring web servers.

Of course, there is another side to all this: the application itself.  For application monitoring, you can use the provided libraries to send custom metrics to Logz.io instances.

Another feature we haven’t touched is application tracing. Powered with this, you can observe the inner workings of an application, where improvements might be made. But that doesn’t mean we won’t cover it. Stay tuned.