These days, more and more web applications are developed and refined to keep the customer engagement at the highest level possible. It is crucial to provide a smooth experience to the customer hence monitoring is of paramount importance. One key factor in that is monitoring the web server we use. In this article, we will explore Logz.io features by monitoring an Apache Web Server.
We will be using Ubuntu 20.04 LTS as a host in the examples and outputs.
Monitoring Web Servers for Web Applications
Web servers are the backbone of our web applications. Naturally, there are numerous metrics we can monitor here: CPU, memory usage, connections per second, etc. Additionally, we can gather from the logs browser type, country, operating system, HTTP verb used, and so on. All these valuable bits of data about our web server and customers need recording, analysis, and action when necessary. For this crucial process, we need to use an excellent monitoring stack.
Monitoring with Logz.io
There are numerous solutions for shipping, storing, visualizing, and analyzing these metrics and other information. One of the best solutions is the ELK Stack for logging, complemented by Prometheus for metrics and Open Source grafana for metric visualization and alerts. Now, you can add Jaeger for distributed tracing into the mix. While Elasticsearch holds our most valuable information, Logstash is responsible for shipping and transformation of that data, and Kibana is for visualizing that logging data.
We will use Filebeat as our log shipper, and Prometheus as our metric collector and aggregator.
Web Application Served by Apache
Apache is one of the most used web servers in the world, with extreme flexibility and a varied feature set. In this article, we will use Apache as the web server. Default settings for Apache are sufficient for our case. We will explore the flow information that we can digest from the system and Apache, including logs and various metrics:
- Access Logs
- Error Logs
- Metrics from Apache Server Status endpoint
- System Log Files
- System Metrics
We will use two exporters for Prometheus, the Apache Exporter and Node Exporter. Prometheus will aggregate all the metrics and ship to our Logz.io Prometheus instance.
Installing Apache web server from apt repositories is straight forward, with
apt install -y apache2 command.
Filebeat is a log shipper agent, it monitors the log inputs defined and ships them to Logstash or Elasticsearch.
First, download Filebeat to your server:
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.10.0-amd64.deb
Install the package with the
sudo dpkg -i filebeat-7.10.0-amd64.deb
We can see the installation steps in the output as Filebeat successfully installs.
Now, we need to download the CA certificate. We will be using curl here:
Selecting previously unselected package filebeat. (Reading database ... 81586 files and directories currently installed.) Preparing to unpack filebeat-7.10.0-amd64.deb ... Unpacking filebeat (7.10.0) ... Setting up filebeat (7.10.0) ... Processing triggers for ureadahead (0.100.0-19.1) ... Processing triggers for systemd (229-4ubuntu21.29) ...
Now, we need to download the CA certificate. We will be using curl here:
sudo curl https://raw.githubusercontent.com/logzio/public-certificates/master/AAACertificateServices.crt --create-dirs -o /etc/pki/tls/certs/COMODORSADomainValidationSecureServerCA.crt
Edit the Filebeat configuration file with your favorite editor,. Here, for simplicity, we will use Nano:
sudo nano /etc/filebeat/filebeat.yml
Replace your Logs.io token below and paste into the editor.
filebeat.inputs: - type: log paths: - /var/log/apache2/access.log fields: logzio_codec: plain token: xxxxxxxxxxxxxxxxxxxxxxxx type: apache_access fields_under_root: true encoding: utf-8 ignore_older: 3h - type: log paths: - /var/log/apache2/error.log fields: logzio_codec: plain token: xxxxxxxxxxxxxxxxxxxxxxxx type: apache_error fields_under_root: true encoding: utf-8 ignore_older: 3h filebeat.registry.path: /var/lib/filebeat processors: - rename: fields: - from: "agent" to: "filebeat_agent" ignore_missing: true - rename: fields: - from: "log.file.path" to: "source" ignore_missing: true output.logstash: hosts: ["listener-eu.logz.io:5015"] ssl: certificate_authorities: ['/etc/pki/tls/certs/COMODORSADomainValidationSecureServerCA.crt']
This configuration defines two inputs: Apache access logs and Apache error logs. Additionally, we have one output, which is Logstash in Logz.io.
To enable system module
sudo cd /etc/filebeat/modules.d
Change name of the default
sudo mv system.yml.disabled system.yml
Restart Filebeat for changes to take effect.
sudo systemctl restart filebeat
Installing Prometheus is straightforward using apt packages.
sudo apt install -y prometheus
An output similar to below is displayed.
Reading package lists... Done Building dependency tree Reading state information... Done The following NEW packages will be installed: prometheus 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded. ...
Installing Prometheus Node Exporter
Prometheus Node Exporter collects metrics about your system and exposes these to Prometheus server.
Again, we use
apt command to get the package installed:
sudo apt install -y prometheus-node-exporter
A similar output is displayed.
Reading package lists... Done Building dependency tree Reading state information... Done The following additional packages will be installed: prometheus-node-exporter-collectors The following NEW packages will be installed: prometheus-node-exporter prometheus-node-exporter-collectors ...
Installing Prometheus Apache Exporter
Now we need to install an exporter to gather Apache metrics and allow Prometheus to scrape them.
sudo apt install -y prometheus-apache-exporter
Reading package lists... Done Building dependency tree Reading state information... Done The following NEW packages will be installed: prometheus-apache-exporter ...
Next, let’s check our Prometheus listeners.
netstat -tulpen | grep prometheus
Prometheus Apache exporter is listening on 9117 as expected and Node Exporter listens on 9100.
tcp6 0 0 :::9117 :::* LISTEN 114 11459019 61135/prometheus-ap tcp6 0 0 :::9090 :::* LISTEN 114 11469554 63857/prometheus tcp6 0 0 :::9100 :::* LISTEN 114 11489288 68358/prometheus-no
TL;DR Prometheus Installation
First, we need to configure Prometheus to scrape the metrics from our exporters. For this, we define three jobs under the
scrape_configs stanza: one for Apache exporter, one for Node exporter, and lastly one for Prometheus itself.
Additionally, we define
remote_write to send metrics to Logz.io Prometheus-as-a-Service instances. After this, we will be able to graph our metrics in Grafana.
Edit the Prometheus config file at
global: scrape_interval: 15s evaluation_interval: 15s external_labels: p8s_logzio_name: 'app_server_apache' alerting: alertmanagers: - static_configs: - targets: ['localhost:9093'] rule_files: scrape_configs: - job_name: 'prometheus' scrape_interval: 5s scrape_timeout: 5s static_configs: - targets: ['localhost:9090'] - job_name: node static_configs: - targets: ['localhost:9100'] - job_name: apache static_configs: - targets: ['localhost:9117'] remote_write: - url: https://listener.logz.io:8053 bearer_token: xxxxxxxxxxxxxxxxxxxxxxxx remote_timeout: 30s queue_config: batch_send_deadline: 5s max_shards: 10 min_shards: 1 max_samples_per_send: 500 capacity: 10000
Visualizing Metrics with Logz.io
When we click the “Metrics” button, we are welcomed with a default dashboard, listing shortcuts to other pre-built dashboards and panels.
As you can see, Logz.io receives our Apache and system metrics, detailed in the “Doc Count Per Module” and “Doc Count Per Metrics” panels.
The built-in “System Metrics” dashboard aids us in monitoring our server in great detail. The dashboard includes graphs for CPU, memory, disk and network metrics.
We can visualize Apache server metrics both in Grafana and Kibana, but first we will explore ELK Apps for Apache.
ELK apps are curated by Logz.io and include visualizations, dashboards and alerts. They offer best practices and some great ideas as well.
Click the “ELK Apps” button and type in “Apache” as a keyword Select Apache Access and hit Install.
After installation, go to “Kibana” and click “Dashboards” on the lefthand side menu (button has a graph icon). Click “Apache Access.” The subsequent dashboard organizes several Apache access metrics.
This particular graph represents Apache access type by browser.
Another graph that “Apache Access” app generated for us is “Apache Access By OS..
Of course, we also have the flexibility to generate our own metric dashboards. Here, we can see scraped data from the Apache Status endpoint for use in further visuals.
Using these metrics we generate a dashboard in grafana quite easily. This custom dash makes use of the following metrics; “Apache Sent KB”, “Apache CPU Load”, “Apache Scoreboard”, “Apache Connections” and “Apache Accesses Total” metrics for this custom dashboard.
Live Tail of Logs
On “Live Tail” press play button (press ctrl+alt+s) to start the live tail and after a short time we will be able to see our Apache access, Apache errors and system logs. (for brevity following shows only Apache logs in real time).
Here we are filtering the live tail logs with the keyword “http”. This filtering ability is quite useful on troubleshooting live scenarios.
Another great feature of Logz.io is that of “Exceptions.” Here, we can find a lot of important information gathered together. One of the messages catches our eye “error: maximum authentication attempts exceeded for root”, this shows us someone tried to authorize as root unsuccessfully.
Another great feature of Logz.io is that of “Exceptions.” Formally called the Insights Engine, it uses machine learning and analyzes the data in your logs from different perspectives.
You can see if there is any spike in received exceptions and whether there are new kinds of exceptions or not. Correlating exceptions finds the needles in haystacks by analyzing thousands of exceptions and then drilling down into them. Exceptions then lists them according to their context. Any exceptions that haven’t been seen before are highlighted, so information is easy to find. Also, you can ignore known and harmless exceptions as well.
Here, we can find a lot of important information gathered together. One of the messages catches our eye:
error: maximum authentication attempts exceeded for root, this shows us someone tried to authorize as root unsuccessfully.
We have explored Apache and system observability, both logs and metrics, in great detail. We have found correlations between system logs and system metrics via the Insights module and have also explored Grafana by both re-using existing dashboards and also creating a dashboard for ourselves.
We were able to see our Apache web server’s CPU load, detailed information about connections, and visitor information like browser data, GEOIP (via IP Geoip databases), and other crucial data points for monitoring web servers.
Of course, there is another side to all this: the application itself. For application monitoring, you can use the provided libraries to send custom metrics to Logz.io instances.
Another feature we haven’t touched is application tracing. Powered with this, you can observe the inner workings of an application, where improvements might be made. But that doesn’t mean we won’t cover it. Stay tuned.