Shipping Multiline Logs with Filebeat

Multiline logs provide valuable information for developers when troubleshooting issues with applications. An example of this is the stack trace. A stack trace is a sequence of method calls that an application was in the middle of when an exception was thrown. The stack trace includes the line in question that encountered the error, as well as the error itself. An example of a Java stack trace can be seen here: 

Exception in thread "main" java.lang.NullPointerException
        at com.example.myproject.Book.getTitle(Book.java:16)
        at com.example.myproject.Author.getBookTitles(Author.java:25)
        at com.example.myproject.Bootstrap.main(Bootstrap.java:14)

When using a logging tool like the ELK stack, it can be difficult to identify and search for a stack trace without the right configuration in place. When shipping application logs with an open source light shipper like Filebeat, each line of a stack trace will be seen in Kibana as an individual document.

So, the stack trace above would be seen in Kibana as four separate documents. This makes it difficult to search and understand errors and exceptions within the stack trace as they are divorced from their context as a common event. When logging application logs with Filebeat, users can avoid this issue by adding configuration options in the filebeat.yml file.

You can configure the filebeat.yml input section filebeat.inputs to  add a few multiline configuration options to make sure that multiline logs, like stack traces, are sent as one complete document. Adding the configuration options below to the filebeat.yml input section will ensure that the Java stack trace referenced above will be sent as a single document. 

multiline.pattern: '^[[:space:]]'
multiline.negate: false
multiline.match: after

Multiline logging in Filebeat

mutliline.pattern – This configuration option defines the regular expression pattern to match. In the example above the regular expression is matching any line that begins with whitespace up to the previous line.

multiline.negate – This option defines if the pattern is negated. The default is false.

multiline.match – This option determines how Filebeat combines matching lines into an event. This option depends on the value for negate. In the example above, we set negate to false and match to after. This means that consecutive lines that match the pattern are attached to the previous line that does not match the pattern. 

Along with the multiline configuration options mentioned above, you can set options to flush the memory of a multiline message, set the maximum number of lines that can be included in a single event, and you can increase the timeout, which is set to 5 seconds by default.

Let’s take a look at an example using the multiline.flush_pattern. This configuration option with Filebeat is useful for multiline application logs that contain events that start and end with specific markers.

[2015-08-24 11:49:14,389] Start new event
[2015-08-24 11:49:14,395] Content of processing something
[2015-08-24 11:49:14,399] End event

If we want these lines to be seen in Kibana as a single document we would use the following multiline configuration options in the filebeat.yml:

multiline.pattern: ‘Start new event’
multiline.negate: true
multiline.match: after
multiline.flush_pattern: ‘End event’

From the configuration options above, when the pattern “Start new event” is seen and the following lines do not match the pattern, they will be appended to the previous line that does match the pattern. The flush_pattern option will then signal that the multiline event is over when a line is seen beginning with “End event.”

Multiline logging in Filebeat

Conclusion

Centralizing your application logs into a single location is an important first step to help troubleshoot any issues that arise with your application. Making sure that your logs are being ingested and displayed correctly within that tool can help companies decrease their mean time to resolution.