Using the Prune Filter in Logstash

Logstash has a number of helpful plugins. We’ve covered the mutate plugin in great detail here, as well as Logstash grok, but it was time to go over some of the others. Here, the Logstash Prune Filter will get its due attention. Its existence owes to the need to remove fields according to select blacklists or whitelists of field names and their associated values. Put more curtly, it prunes the excess branches (fields) in your garden (your data).

(Or, if you really like dried plums, I guess you can think of the Prune Filter as some sort of, um, filter that turns things into prunes? Whatever, back to the tut.)

If you’ve got other filters making new fields when they handle data, this organizes and cleans up those filters’ work.

Logstash Prune Filter Basics

The Logstash Prune Filter is a built-in feature. Verify it’s there with:

cd /usr/share/logstash/bin

./logstash-plugin list | grep -i prune

The output should come back:

logstash-filter-prune

Logstash Prune Filter Configuration Options

Logstash has three sections in its configuration file: inputs, filters, and outputs. Note, if you’re a newbie to Logstash, inputs were once called prospectors. In the filters section, add the appropriate prune filters.

filter {
      prune {
        blacklist_names => [ "[0-9]+", ]
      }
    }
filter {
      prune {
        whitelist_names => [ "$_login", ]
      }
    }

There are five main prune filter configuration options:

• Blacklist_names
• Blacklist_values
• whilelist_names
• whitelist_values
• interpolate

Similarly to the Logstash Mutate Filter, it also supports these seven other common config options:

• add_field
• add_tag
• remove_field
• remove_tag
• id
• periodic_flush
• enable_metric

Prune Filter Examples

Commonly, you’ll be shipping to Logstash from the lightweight shipper Filebeat and the Filebeat port number 5044. The Logstash configuration then will look like this:

input {
    beats { 
        port => "5044" 
    }
}
filter {
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
    prune {
      #We’ll fill this in
    }
}
output {
  lumberjack {
    host => "listener.logz.io"
    port => 5006
    ssl_certificate => "/usr/share/logstash/keys/TrustExternalCARoot.crt"
    codec => json_lines
 }

You can add multiple configurations to a single Prune filter. For instance, you can add tags (and multiple tags at that) and remove old tags at the same time. Such as:

filter {
    prune {
      add_tag => [ “waldo_%{taggy_mctagface}”, “wally_%{taggy_mctagface}” ]
      remove_tag => [ "once_useful_tag", "now_useless_tag" ]
    }
}

Or:

filter {
    prune {
        blacklist_names => [ "[0-9]+", ]
        whitelist_names => [ "$_login", ]
    }
}

Or better yet, altogether:

filter {
    prune {
      add_tag => [ "waldo_%{taggy_mctagface}", "wally_%{taggy_mctagface}" ]
      remove_tag => [ "once_useful_tag", "now_useless_tag" ]
      blacklist_names => [ "[0-9]+", ]
      whitelist_names => [ "$_login", ]
    }
}

Endnotes

The Prune Filter isn’t at the top of the list for Logstash users, but it’s still quick and handy. It makes it easy to reformat small pieces of data with easy syntax or common regex. It’s also a handy alternative to using a code-specific filter, such as the Logstash Ruby Filter. But most importantly, it’s a great last stopgap for, well, pruning excess fields created by other filters like the JSON Filter or KV Filter.