Logstash has a number of helpful plugins. We’ve covered the mutate plugin in great detail here, as well as Logstash grok, but it was time to go over some of the others. Here, the Logstash Prune Filter will get its due attention. Its existence owes to the need to remove fields according to select blacklists or whitelists of field names and their associated values. Put more curtly, it prunes the excess branches (fields) in your garden (your data).
(Or, if you really like dried plums, I guess you can think of the Prune Filter as some sort of, um, filter that turns things into prunes? Whatever, back to the tut.)
If you’ve got other filters making new fields when they handle data, this organizes and cleans up those filters’ work.
Logstash Prune Filter Basics
The Logstash Prune Filter is a built-in feature. Verify it’s there with:
cd /usr/share/logstash/bin ./logstash-plugin list | grep -i prune
The output should come back:
logstash-filter-prune
Logstash Prune Filter Configuration Options
Logstash has three sections in its configuration file: inputs, filters, and outputs. Note, if you’re a newbie to Logstash, inputs were once called prospectors. In the filters section, add the appropriate prune filters.
filter { prune { blacklist_names => [ "[0-9]+", ] } } filter { prune { whitelist_names => [ "$_login", ] } }
There are five main prune filter configuration options:
Blacklist_names
Blacklist_values
whilelist_names
whitelist_values
interpolate
Similarly to the Logstash Mutate Filter, it also supports these seven other common config options:
add_field
add_tag
remove_field
remove_tag
id
periodic_flush
enable_metric
Prune Filter Examples
Commonly, you’ll be shipping to Logstash from the lightweight shipper Filebeat and the Filebeat port number 5044. The Logstash configuration then will look like this:
input { beats { port => "5044" } } filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } } prune { #We’ll fill this in } } output { lumberjack { host => "listener.logz.io" port => 5006 ssl_certificate => "/usr/share/logstash/keys/TrustExternalCARoot.crt" codec => json_lines }
You can add multiple configurations to a single Prune filter. For instance, you can add tags (and multiple tags at that) and remove old tags at the same time. Such as:
filter { prune { add_tag => [ “waldo_%{taggy_mctagface}”, “wally_%{taggy_mctagface}” ] remove_tag => [ "once_useful_tag", "now_useless_tag" ] } }
Or:
filter { prune { blacklist_names => [ "[0-9]+", ] whitelist_names => [ "$_login", ] } }
Or better yet, altogether:
filter { prune { add_tag => [ "waldo_%{taggy_mctagface}", "wally_%{taggy_mctagface}" ] remove_tag => [ "once_useful_tag", "now_useless_tag" ] blacklist_names => [ "[0-9]+", ] whitelist_names => [ "$_login", ] } }
Endnotes
The Prune Filter isn’t at the top of the list for Logstash users, but it’s still quick and handy. It makes it easy to reformat small pieces of data with easy syntax or common regex. It’s also a handy alternative to using a code-specific filter, such as the Logstash Ruby Filter. But most importantly, it’s a great last stopgap for, well, pruning excess fields created by other filters like the JSON Filter or KV Filter.