Lessons Learned While Writing Our New Logback Appender + How You Can Get It!

logz.io appender

Here at Logz.io, we take logs seriously. (If not us, who will?)

We run mainly on Dockerized Java microservices, and until recently, we had another Docker container that collected logs from our microservices.

Then, as part of a move to “self-contained” containers in preparation for another future move to a Docker orchestration solution — more on that in a future blog post! — we created a robust appender that integrates seamlessly into our current Logback setup.

Guess what? You can go here on GitHub to get it too!

Here are some key features:

  • Persistency. Each log line is saved on your local file system in a persistent queue (with a very fast input/output — we checked) using the superb bigqueue library. So, if your application crashes and reboots, you won’t lose a single logline.
  • Robustness. We will make sure that each log that you send will end up in your Logz.io account and return an acknowledgment from our log receivers.
  • Security. All logs sent are encrypted via HTTPS.
  • Light network load. We aggregate your logs into bulks and send them together.
  • Out-of-the-box enrichment. You don’t need to care about simple groking. The logline will retain the timestamp, logger, thread, log level, message, and any other additional fields and environment variables that you have requested.
  • Shading. We know that you love your libraries just the way they are. So, this appender is distributed as an “Uber Jar” that shades everything we need inside so you won’t have to worry about it.

We encourage Logback and Logz.io users to use the appender. (The log4j version is coming soon!)

The Lessons I Learned

When I sat down to write the appender, I encountered some caveats that I wanted to share because I could not find much documentation on writing new appenders at all.

So, here are some useful tips if you ever find yourself in the same situation.

Select the proper class to implement

Your greatest source when starting your appender will be this Logback page. Almost all online searches for answers to questions on Logback appenders will end up there.

It’s a great resource on the options of the common appenders, but it lacks useful information when it comes to writing your own.

The beginning states that all appenders must implement the ch.qos.logback.core.Appender interface. But if you don’t want to waste time coding unnecessary stuff, you can extend ch.qos.logback.core.AppenderBase (which does most of the dirty work for you) or extend any of the other appenders if your appender shares some of the functionality with them.

Don’t code things you don’t have to.


When you start extending the AppenderBase class, you will notice that the one thing you basically need to do is to handle this:

abstract protected void append(E eventObject);

The thing that is not documented clearly is that eventObject can come in the form of ILoggingEvent, which allows you to access the members that you usually need right out of the box.

For example:

protected void append(ILoggingEvent loggingEvent) {
   String level = loggingEvent.getLevel().levelStr;
   String loggerName = loggingEvent.getLoggerName();
   String formattedMessage = loggingEvent.getFormattedMessage();
   String thread = loggingEvent.getThreadName();

Task Executor

The actual work of this appender is done outside of the “logback ecosystem” and the class that extends the AppenderBase just calls another generic class, so we didn’t want to use logback AsyncAppender and chose to implement our own threading mechanism instead. (AppenderBase calls another generic class so we can integrate more frameworks easily.)

I really love to use ScheduledExecutorService from java.util.concurrent instead of just implementing Runnable or any other third-party scheduling library, so I did.

Everything was just awesome until… the appender just did not stop. The main Java program finished its job, but the process was kept going on and on and on.

At first, we considered declaring that to be a feature — an appender that cares so much about logging that it just can’t stop sending them! (Even if there is nothing to send.)

But then we came to our senses and decided to read how Logback handles that issue in their own code. Here was the answer:


You get an executor service from Logback that shuts itself down when the appender dies. That way, we can close ourselves gracefully (while implementing a “no log left behind” policy).

Which leads me to my next point.

Shutdown hook

For the scheduled service to terminate, we needed to link the JVM termination hook with Logback. Oddly enough, that needed to be explicitly configured. Adding the following line under the <configuration> section of logback does exactly that:

<shutdownHook class="ch.qos.logback.core.hook.DelayingShutdownHook"/>

Now, Logback will listen to the JVM shutdown hook and terminate everything the way it should.


Can’t stress that enough.

It’s obviously important to test all code. But I wanted to mention testing here because this is not a trivial case — the appender, after all, is not a part of our log receivers. I needed to mock it in testing to make sure that I was sending the proper things to it.

In this case, I wrote a Jetty-based mock Logz.io log receiver that accepted logs in the same format as we have and allowed me to assert on it directly based on the specific logline that was sent during the test.

I also added some real-life “features” such as the abilities to get timeouts while sending logs and deliberate exceptions from the server — resulting in the connection being closed — so that we can make sure that no log goes to waste when the shit hits the fan in real life.

What Do You Think?

So, if you ever need to write an appender yourself, I hope that you will find useful tips here.

And in case you use Logz.io and Logback, this is the appender for you!

As always, we encourage feedback. You can drop us an e-mail, open an issue on GitHub, or — even better — issue a pull request!

Observability at scale, powered by open source


Consolidate Your AWS Data In One Place

Learn More
scaleup-logo Join our annual user conference! November 16, 2021