What Do You Call an ELK with a Tail? Logz.io Live Tail!

We are excited to announce the general availability of Live Tail in Logz.io for tailing all the logs in your system, in real time, and from the same place within Kibana. This feature is now part and parcel of our comprehensive end-to-end ELK service and adds a much-required functionality to the open-source version of Kibana.

Easily analyzing and querying log data is one of the main reasons Kibana is so popular. But sometimes, the need (or urge) arises to tail -f a specific log file to see the stream of logs as they are being outputted by the relevant process.

For example, say you deployed new code into staging or even production. Using Live Tail, you will be able to see new errors as they are being logged in real-time without having to leave Kibana, open a terminal, and manually tail all the relevant files.

Let’s take a closer look at how Live Tail works and how to use it. We are officially releasing the feature at the AWS Summit in San Francisco today — so, if you are there, head on over to Booth #1080 for a demo! If you’re not there, keep reading for more information.

Just Hit Play

Live Tail is accessed from the main menu and turning the feature on is easy — all you have to do is enter a regex-based search in the designated field at the top of the page, and click the Play button (or just press Enter).

Logz.io opens up a connection and begins tailing all the logs being shipped into Logz.io.

To stop the tailing, hit the same button (now a Stop button).

Filtering the Logs

Of course, you don’t always want to tail ALL the log messages being logged in the system. To help you find the tree in the forest, Live Tail includes filtering options that you can use to pinpoint the data that you are interested in tailing.

In the Match field at the top of the page, you can filter the tail according to a pattern that you define. This can be as simple as a free text pattern or you can use REGEX to enter more advanced filtering patterns. Enter your filtering pattern and hit enter — Live Tail will initiate the tailing session with the predefined filter enabled.

In the example below, I am tailing only accepted traffic in my AWS VPC Flow logs:

Using the Ignore field, I can use an “Ignore” pattern to initiate a tailing session that will ignore logs containing specific filtering patterns.

Note: The information displayed in Live Tail represents the data included in the “message” field for all the logs being shipped into Logz.io. Advanced field-based filtering can be done on metadata not displayed in Live Tail. Contact our support team for more details.

Searching the Logs

Live Tail includes some helpful features to help you find and locate specific strings within the log messages.

Finding

Just like a text editor or browser, Live Tail’s Find feature allows you to locate — and subsequently browse between — specific text strings within the tail session. Click the search icon on the top-right side of the page, and search away!

Highlighting

Expand the main Live Tail bar to display some additional features.

Within the Add highlight field, you can enter any string you would like to focus on. This allows you to begin a tailing session with any predefined string you may be looking for. In a long stream of logs, this will help specific data stand out.

You can add as many highlights as you wish and easily remove them when necessary.

The Display Timestamp check-box allows you to add or remove the timestamp added by Logz.io to the messages. In case your messages already include a timestamp, use this option to have only one displayed.

Live Tail Display Modes

Live Tail can be displayed in Dark or Light mode — and you can switch between modes using the slider handle.

Scrolling and Clearing Sessions

As soon as your eye catches a piece of data that interests you, you will intuitively scroll up.

When this happens, the screen will freeze and allow you to take a look at this data but the Live tailing will continue. To help you scroll down to the last logged message, Live Tail includes a Scroll Down button which will lead you to the bottom of the tail.

Just like the clear command in your terminal, the Clear icon will clear your tailing session.

Keyboard Shortcuts

Conveniently, Live Tail supports a bunch of keyboard shortcuts to help you operate the feature more easily:

• CTRL + ALT (OPTION on Mac)+ S = Start a new session
• CTRL + ALT (OPTION on Mac) + Z = Stop the current session
• CTRL + ALT (OPTION on Mac) + X = Clear the session window
• CTRL + ALT (OPTION on Mac) + T = Toggle settings
• CTRL + ALT (OPTION on Mac) + A = Toggle scroll-to-bottom
• CTRL + ALT (OPTION on Mac) + H = Toggle light and dark themes

To get a list of the available shortcuts, press “SHIFT + ?”.

Entering the Matrix

The ELK Stack is designed for centralized logging — pulling and indexing data from various data sources, enhancing this data, and then performing various analysis operations for the purpose of troubleshooting and monitoring.

To present the data in near real-time, Live Tail enables you to tap into the stream of logs being shipped into the system, in their raw form, and before they are enhanced.

In a way, this endless stream of data resembles a “matrix-like” streaming of logs coming into the system. Live Tail enables you to view this stream and enter this matrix in real-time.

If you don’t believe me — access Live Tail and press CTRL + ALT (OPTION on Mac) + M.