Musings in YAML – Tips for Configuring Your Beats

The following scenario might sound familiar.

After carefully configuring the prospectors and output settings in your Filebeat configuration, you finally start the service only to open up Kibana and face an empty Discover page.

After hours of searching the documentation and forums online, you finally find your mistake — one of your prospectors was entered in the configuration file with bad indentation!

If only you would have been more attentive during those YAML lessons!

The truth is that there are no lessons teaching us YAML syntax. Elastic has improved the configurations tests that you can run, but some errors still pass these tests. It’s just one of those things we are expected to know. Sadly, when working with the ELK Stack and the Beats family of shippers especially, we learn by error.

To save you the time (and aggravation), here are a few guidelines on writing your YAML configuration files.

Don’t Invent the Wheel

There is no need to start from scratch. Default configuration settings are supplied in most, if not all, of the log shippers in the Beats family. Use these settings as your initial reference. In most cases, you will only need to comment or uncomment specific lines.

Some Beats, such as Filebeat, include full example configuration files (/etc/filebeat/filebeat.full.yml). These files include long lists all the available configuration options. Copy the content of these files into a text editor, carefully remove the unnecessary pieces, and paste the content back into the file after validating the formatting (more about validation later).

Spaces vs. Tabs

How the lines are indented is critical in your YAML file, and in the timeless “spaces vs. tabs” debate, YAML creators decided that tabs are forbidden:

“Tabs have been outlawed since they are treated differently by different editors and tools. And since indentation is so critical to proper interpretation of YAML, this issue is just too tricky to even attempt.”

So — rule number one — DO NOT use tabs when indenting your lines — only spaces. Otherwise, you will be reliving the scenario described above time and again.

You can use a code editor to verify if there are any abnormal whitespace characters in the configuration file. In Sublime, for example, simply paste the YAML into the editor and highlight the lines — you will see dots for every space before the line and blank spaces before the line for other whitespace characters. This will help you determine which lines have incorrect characters.

Preserve A Unified Structure

YAML configuration files for Beats are mostly built the same way, using two spaces for indentation.

Sections are separated by a long line of # (hash) symbols — as much as this sounds funny — don’t mistake these for commented out configuration lines.

The ‘-’ (dash) character is used for defining new elements — be sure to preserve their indentations and the hierarchies between sub-constructs.

Validate, Validate, and then Validate Again

The times in which we needed to pay for text editors have long gone. We now have a wide set of free, online tools that will help you validate your YAML formatting. I recommend YAML Lint, but there are plenty more parsing assistants out there.

Once validated, you can also run a configuration test. You can use Filebeat, for example, once more (you’ll need to run this command from the directory in which the binary is installed):

sudo filebeat -c filebeat.yml -configtest

If there are any additional errors, you will be notified.

Endnotes

It ain’t rocket science, but a small and simple mistake can make all the difference between a bad day and an even worse day.

If you are using Logz.io, we have made it easy to configure a Filebeat YAML with a dedicated wizard. Even if you are not a Logz.io user, it can come in handy. Go through the wizard and simply remove the fields specific to Logz.io.

Happy YAMLing!