Announcing Logz.io’s New Data Parsing and Log Transformation Tool

*The new self-service parsing tool is not available for Community accounts.

We all know the importance of cataloging, organizing, and breaking down the data in your logs. That process, parsing, makes information easier to find and simplifies subsequent analysis. 

Now, with Logz.io’s upgraded self-parsing tool, custom parsing rules, and log organization is easier than ever.

What’s important is parsing that data out correctly. The better parsed, the easier to query. Some logs though won’t always be parallel with a standard structure because of differences in formatting or extra fields. This is certainly the case with application logs which might have unique formatting and structure.

It utilizes our API to transfer log parsing rules back to your Logz.io account. The new parsing feature is powered by Sawmill, an open source JSON transformation tool developed in-house at Logz.io. The process includes three steps: input a sample log, write the parsing rules, then finally push those rules to your account.

Open your Logz.io account. On the left sidebar, hover your mouse over Logs. Down the menu, under MANAGE YOUR DATA, select Data Parsing. This will take you to the new Data Parsing UI. You’ll see places to input rules, work on log samples, edit, and ultimately validate rule changes.

To get started, go to Editor setup on the left side of the page and click. There, input your account API token. From there, you will be prompted to select your account region location. 

After that, existing log types on your account will load to a drop-down list. Select the log type that you want to parse. If there is a set of rules that exist for that type, it will appear in the Parsing rules workspace on the main screen.

Additionally, you can also add new log types that have not yet been sent to Logz.io, then define their rules.

As you test and edit the rules in the Parsing Rules Workspace to the left side of the screen, the autocomplete feature in the editor will offer code suggestions (e.g., geoIp, grok, addTag, etc.), as seen below:

To see how these rules look when applied to the log sample, press Validate Your Rules to see your custom-formatted log. 

You can also use the Auto-format option to clean up your work.

Example: Grok Parsing (Syslog)

Insert the “steps” format, then “grok” inside of it. Make sure that the JSON attributes and values appearing in your rules are surrounded by double quotation marks, as well as using the anchors ^ at the beginning and $ at the end. 

{
    "steps": [
        {
            "grok": {
                "config": {
                    "field": "message",
                    "patterns": [
                        "^%{TIMESTAMP_ISO8601:time} %{LOGLEVEL:logLevel} %{GREEDYDATA:logMessage}$"
                    ]
                }
            }
        }
    ]
}

Submit & Validate

The new Logz.io parsing tool provides full access and flexibility to those users who want to take a hands-on approach to creating and editing their log parsing rules, without having to work with Support. For those users who want to continue to work with our Support team to make and edit those rules, access is easier with this new tool.

Altogether, Logz.io parsing streamlines rule creation with the use of features like Autocomplete. It allows you to customize rule sets according to log type, as well as test and validate them. And finally, we offer fast response time from our Support team who validate and ultimately implement your parsing rule sets to your account. 

Stay tuned for more updates from Logz.io regarding parsing and new features here on the blog. And try the new Logz.io Parser today.