Grafana Tutorial: Getting Started with Installation & Configuration

Grafana is one of the most popular open source UI for metrics and infrastructure monitoring today. Capable of ingesting metrics from the most popular time series databases, it’s an indispensable tool in modern DevOps. This Grafana tutorial will go over installation, configuration, queries, and initial metrics shipping.

Gafana is the equivalent of what Kibana is for logs (for more, see Grafana vs. Kibana). For the most popular time series DBs for metrics, particularly Prometheus, there is built-in support. For others, there are officially supported plugins, such as for Nagios or Zabbix.

Let’s get started.

Installing Grafana

This Grafana tutorial mainly focuses on installation and configuration on macOS or Ubuntu. In this case, I installed it on an Ubuntu EC2 instance on AWS. Go with the following Bash commands in the AWS CLI:

sudo apt-get install -y apt-transport-https
sudo apt-get install -y software-properties-common wget
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -

This repository covers the latest stable release:

echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list

Next, make sure to update apt-get.

sudo apt-get update

Once you go through that, you can finally install Grafana:

sudo apt-get install grafana

You will get a prompt asking if you’re “sure you want to continue? [Y/n].” Hit that Ywith absolute confidence.

Install the .deb package

Now move to install the .deb package.

[This can work on Mac as well, if you want to be tough and not use Homebrew (don’t be tough if you’re like me; just use Homebrew and get it over with. If you don’t need to do the extra work when you’re at home with three kids in quarantine and you think you have 10 minutes between having to change Netflix shows or getting the kids another snack or wondering why the baby is suddenly crawling on the ceiling, then avoid it. Otherwise, go with the .deb if you really want to. Anyway, this is an Ubuntu tutorial so good luck).]

First, add a user for Fontconfig1.

sudo apt-get install -y adduser libfontconfig1

Then, use wget to download the appropriate .deb file.

wget https://dl.grafana.com/oss/release/grafana_7.1.1_amd64.deb

Or for Debian, use the dpkg command—which I like to call the Democratic People’s Korea-G command—to install.

sudo dpkg -i grafana_7.1.1_amd64.deb

Now, start the server with systemd:

sudo systemctl daemon-reload
sudo systemctl start grafana-server
sudo systemctl status grafana-server

Or init.d:

sudo service grafana-server start
sudo service grafana-server status

To initially login when you go to localhost:3000, enter the default USER: admin and PASS: admin. You will then be prompted to change the default password once logged in. You can also set user and password in grafana.ini (more on this file below).

Grafana CLI

Grafana CLI (or grafana-cli) comes with the installation of Grafana. Many of these commands need sudo privileges. Syntax will follow this model:

grafana-cli --command arguments

When you enter grafana-cli -h, you will get a brief list of commands for the CLI.

COMMANDS:

plugins  Manage plugins for grafana
admin    Grafana admin commands
help, h  Shows a list of commands or help for one command

GLOBAL OPTIONS:

--pluginsDir value       path to the grafana plugin directory (default: "/usr/local/var/lib/grafana/plugins") [$GF_PLUGIN_DIR]
--repo value             url to the plugin repository (default: "https://grafana.com/api/plugins") [$GF_PLUGIN_REPO]
--pluginUrl value        Full url to the plugin zip file instead of downloading the plugin from grafana.com/api [$GF_PLUGIN_URL]
--insecure               Skip TLS verification (insecure)
--debug, -d              enable debug logging
--configOverrides value  configuration options to override defaults as a string. e.g. cfg:default.paths.log=/dev/null
--homepath value         path to grafana install/home path, defaults to working directory
--config value           path to config file
--help, -h               show help
--version, -v            print the version

Grafana Configuration

There are basic configurations to activate in Grafana, as well as more advanced or choice ones. Its configurations are handled in a couple of main .ini files.

It is important to note however not to alter the defaults.ini file. Any changes you need to implement can override defaults from the other files you will work in. Depending on how you installed Grafana or the package, you will need to alter the grafana.ini file and the custom.ini file. To uncomment lines, remove the semicolon from the beginning of each line you want to uncomment.

Linux (deb or rpm)

Locate the file at /etc/grafana/grafana.ini and create a new file at /etc/grafana/custom.ini.

macOS

Find the grafana.ini file already in place at /usr/local/etc/grafana/grafana.ini. Then create /conf/custom.ini inside the /conf/ folder (the same directory as (the same directory as /conf/defaults.ini).

Grafana Logs Path and Logging

The default path for logs in macOS will be /usr/local/var/log/grafana/grafana.log and by default will be saved for 24 hours. You can change these settings in the grafana.ini file by altering the path that reads /var/log/grafana (don’t forget to uncomment by deleting ; at the beginning of the relevant line).

Alternatively, change the default logs path from the command line:

cfg:default.paths.log
./grafana-server --config /custom/config.ini --homepath /custom/homepath cfg:default.paths.logs=/custom/path

For server logs, go to the Server section of grafana.ini. Then, reset router_logging to true to log any HTTP requests to the main log.

To log all SQL calls and their execution times, go the Database section of grafana.ini and set log_queries to true.

To enable data proxy logs, go to the Data Proxy section and make sure logging = true.

There are also multiple settings in grafana.ini under the Logging section for [log], [log.console], [log.file], and [log.syslog]. A sample version of this section is available below:

#################################### Logging ##########################

[log]
# Either "console", "file", "syslog". Default is console and  file
# Use space to separate multiple modes, e.g. "console file"
;mode = console file

# Either "debug", "info", "warn", "error", "critical", default is "info"
;level = info

# optional settings to set different levels for specific loggers. Ex filters = sqlstore:debug
;filters =

# For "console" mode only
[log.console]
;level =

# log line format, valid options are text, console and json
;format = console

# For "file" mode only
[log.file]
;level =

# log line format, valid options are text, console and json
;format = text

# This enables automated log rotate(switch of following options), default is true
;log_rotate = true

# Max line number of single file, default is 1000000
;max_lines = 1000000

# Max size shift of single file, default is 28 means 1 << 28, 256MB
;max_size_shift = 28

# Segment log daily, default is true
;daily_rotate = true

# Expired days of log file(delete after max days), default is 7
;max_days = 7

[log.syslog]
;level =

# log line format, valid options are text, console and json
;format = text

# Syslog network type and address. This can be udp, tcp, or unix. If left blank, the default unix endpoints will be used.
;network =
;address =

# Syslog facility. user, daemon and local0 through local7 are valid.
;facility =

# Syslog tag. By default, the process' argv[0] is used.
;tag =

Grafana Queries

Grafana supports as many as 26 queries per panel. Grafana also depends on the relevant query language for its data source. That could be Prometheus’s PromQL, Elasticsearch’s Query DSL, PostgreSQL’s SQL variant, InfluxQL for InfluxDB, etc.

In the query selection tab of Grafana’s UI, you can choose which data source to use. It will display whichever sources you already linked to your iteration of Grafana. You can even select the Mixed option to query from multiple sources (and hence, in multiple QLs) at the same time.

Query options include interval, time shift, relative time, minimum interval (min interval), and max data points.

Shipping Metrics from Prometheus to Grafana

Grafana retains built-in support for Prometheus. We’ll start there.

In this case, you need to configure prometheus.yml (if using a MacOS, this is either at /usr/local/etc/prometheus/prometheus.yml or /usr/local/Cellar/prometheus/prometheus.yml). You will want to make sure that targets are selected for Prometheus and the Prometheus Node Explorer.

Add the Prometheus datasource. Add localhost:9090 for Prometheus and localhost:9100 for Prometheus Node Explorer.

Confirm it’s working by opting to test it below. Then it should give you the okay.

Go to your Prometheus Dashboard on in the Grafana UI at localhost:3000. Then make adjustments as necessary.

Exporting From Other Sources

This brief Grafana tutorial and intro to key concepts was meant to give you a glimpse of the tool’s abilities. Grafana’s built-in integrations make popular metric source exports like Prometheus a snap. You can also export from sources like Graphite and Metricbeat in this way. Officially supported plugins exist for other metrics sources like Zabbix or Nagios. Check out our other tutorials for further Grafana examples and monitoring configuration examples.