Grafana is one of the most popular open source UI for metrics and infrastructure monitoring today. Capable of ingesting metrics from the most popular time series databases, it’s an indispensable tool in modern DevOps. This Grafana tutorial will go over installation, configuration, queries, and initial metrics shipping.
Gafana is the equivalent of what Kibana is for logs (for more, see Grafana vs. Kibana). For the most popular time series DBs for metrics, particularly Prometheus, there is built-in support. For others, there are officially supported plugins, such as for Nagios or Zabbix.
Let’s get started.
This Grafana tutorial mainly focuses on installation and configuration on macOS or Ubuntu. In this case, I installed it on an Ubuntu EC2 instance on AWS. Go with the following Bash commands in the AWS CLI:
sudo apt-get install -y apt-transport-https sudo apt-get install -y software-properties-common wget wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
This repository covers the latest stable release:
echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
Next, make sure to update apt-get.
sudo apt-get update
Once you go through that, you can finally install Grafana:
sudo apt-get install grafana
You will get a prompt asking if you’re “
sure you want to continue? [Y/n].” Hit that
Ywith absolute confidence.
Install the .deb package
Now move to install the
[This can work on Mac as well, if you want to be tough and not use Homebrew (don’t be tough if you’re like me; just use Homebrew and get it over with. If you don’t need to do the extra work when you’re at home with three kids in quarantine and you think you have 10 minutes between having to change Netflix shows or getting the kids another snack or wondering why the baby is suddenly crawling on the ceiling, then avoid it. Otherwise, go with the .deb if you really want to. Anyway, this is an Ubuntu tutorial so good luck).]
First, add a user for
sudo apt-get install -y adduser libfontconfig1
wget to download the appropriate
Or for Debian, use the
dpkg command—which I like to call the Democratic People’s Korea-G command—to install.
sudo dpkg -i grafana_7.1.1_amd64.deb
Now, start the server with systemd:
sudo systemctl daemon-reload sudo systemctl start grafana-server sudo systemctl status grafana-server
sudo service grafana-server start sudo service grafana-server status
To initially login when you go to localhost:3000, enter the default USER: admin and PASS: admin. You will then be prompted to change the default password once logged in. You can also set user and password in
grafana.ini (more on this file below).
Grafana CLI (or
grafana-cli) comes with the installation of Grafana. Many of these commands need sudo privileges. Syntax will follow this model:
grafana-cli --command arguments
When you enter
grafana-cli -h, you will get a brief list of commands for the CLI.
COMMANDS: plugins Manage plugins for grafana admin Grafana admin commands help, h Shows a list of commands or help for one command GLOBAL OPTIONS: --pluginsDir value path to the grafana plugin directory (default: "/usr/local/var/lib/grafana/plugins") [$GF_PLUGIN_DIR] --repo value url to the plugin repository (default: "https://grafana.com/api/plugins") [$GF_PLUGIN_REPO] --pluginUrl value Full url to the plugin zip file instead of downloading the plugin from grafana.com/api [$GF_PLUGIN_URL] --insecure Skip TLS verification (insecure) --debug, -d enable debug logging --configOverrides value configuration options to override defaults as a string. e.g. cfg:default.paths.log=/dev/null --homepath value path to grafana install/home path, defaults to working directory --config value path to config file --help, -h show help --version, -v print the version
There are basic configurations to activate in Grafana, as well as more advanced or choice ones. Its configurations are handled in a couple of main .ini files.
It is important to note however not to alter the defaults.ini file. Any changes you need to implement can override defaults from the other files you will work in. Depending on how you installed Grafana or the package, you will need to alter the
grafana.ini file and the custom.ini file. To uncomment lines, remove the semicolon from the beginning of each line you want to uncomment.
Linux (deb or rpm)
Locate the file at
/etc/grafana/grafana.ini and create a new file at
grafana.ini file already in place at
/usr/local/etc/grafana/grafana.ini. Then create
/conf/custom.ini inside the
/conf/ folder (the same directory as (the same directory as
Grafana Logs Path and Logging
The default path for logs in macOS will be
/usr/local/var/log/grafana/grafana.log and by default will be saved for 24 hours. You can change these settings in the
grafana.ini file by altering the path that reads
/var/log/grafana (don’t forget to uncomment by deleting
; at the beginning of the relevant line).
Alternatively, change the default logs path from the command line:
cfg:default.paths.log ./grafana-server --config /custom/config.ini --homepath /custom/homepath cfg:default.paths.logs=/custom/path
For server logs, go to the Server section of
grafana.ini. Then, reset
true to log any HTTP requests to the main log.
To log all SQL calls and their execution times, go the Database section of
grafana.ini and set
log_queries to true.
To enable data proxy logs, go to the Data Proxy section and make sure logging
There are also multiple settings in
grafana.ini under the Logging section for
[log], [log.console], [log.file], and
[log.syslog]. A sample version of this section is available below:
#################################### Logging ########################## [log] # Either "console", "file", "syslog". Default is console and file # Use space to separate multiple modes, e.g. "console file" ;mode = console file # Either "debug", "info", "warn", "error", "critical", default is "info" ;level = info # optional settings to set different levels for specific loggers. Ex filters = sqlstore:debug ;filters = # For "console" mode only [log.console] ;level = # log line format, valid options are text, console and json ;format = console # For "file" mode only [log.file] ;level = # log line format, valid options are text, console and json ;format = text # This enables automated log rotate(switch of following options), default is true ;log_rotate = true # Max line number of single file, default is 1000000 ;max_lines = 1000000 # Max size shift of single file, default is 28 means 1 << 28, 256MB ;max_size_shift = 28 # Segment log daily, default is true ;daily_rotate = true # Expired days of log file(delete after max days), default is 7 ;max_days = 7 [log.syslog] ;level = # log line format, valid options are text, console and json ;format = text # Syslog network type and address. This can be udp, tcp, or unix. If left blank, the default unix endpoints will be used. ;network = ;address = # Syslog facility. user, daemon and local0 through local7 are valid. ;facility = # Syslog tag. By default, the process' argv is used. ;tag =
Grafana supports as many as 26 queries per panel. Grafana also depends on the relevant query language for its data source. That could be Prometheus’s PromQL, Elasticsearch’s Query DSL, PostgreSQL’s SQL variant, InfluxQL for InfluxDB, etc.
In the query selection tab of Grafana’s UI, you can choose which data source to use. It will display whichever sources you already linked to your iteration of Grafana. You can even select the Mixed option to query from multiple sources (and hence, in multiple QLs) at the same time.
Query options include interval, time shift, relative time, minimum interval (min interval), and max data points.
Shipping Metrics from Prometheus to Grafana
Grafana retains built-in support for Prometheus. We’ll start there.
In this case, you need to configure
prometheus.yml (if using a MacOS, this is either at
/usr/local/Cellar/prometheus/prometheus.yml). You will want to make sure that targets are selected for Prometheus and the Prometheus Node Explorer.
Add the Prometheus datasource. Add
localhost:9090 for Prometheus and
localhost:9100 for Prometheus Node Explorer.
Confirm it’s working by opting to test it below. Then it should give you the okay.
Go to your Prometheus Dashboard on in the Grafana UI at
localhost:3000. Then make adjustments as necessary.
Exporting From Other Sources
This brief Grafana tutorial and intro to key concepts was meant to give you a glimpse of the tool’s abilities. Grafana’s built-in integrations make popular metric source exports like Prometheus a snap. You can also export from sources like Graphite and Metricbeat in this way. Officially supported plugins exist for other metrics sources like Zabbix or Nagios. Check out our other tutorials for further Grafana examples and monitoring configuration examples.