Elastic beats Beats Users with a Breaking Change

Last week Elastic.co started locking down its Beats OSS shippers such that they will not be able to send data to: 

• Elasticsearch 7.10 or earlier open source distros
• Non-Elastic distros of Elasticsearch

If you weren’t watching closely this might have slipped under your radar. Embedded within the Beats 7.13 minor release that was published over the weekend, a release note advised of a breaking change in which “Beats may not be sending data to some distributions of Elasticsearch”.

Breaking changes are not to be taken mildly in many open source projects. We avoid these at all costs to prevent problems for those integrating open source to other technologies. Elastic, however, has been more liberal with introducing breaking changes. This causes challenges for other projects which are using these open source technologies or integrating with them. 

Because of this, from their perspective Elastic finds it challenging to support these other technologies from a compatibility perspective. As a result, they have introduced this breaking change that checks for an Elastic licensed product. This specifically breaks open source technologies including previously open source Elastic solutions, forcing users to upgrade or downgrade different components. 

What Does this Breaking Change to Beats Mean?

I don’t want to get into the politics of the vendor move, the ethics of open source or the choice to insert such a breaking change in a minor release (there’s enough chatter on that over Twitter). 

Instead, I’d like to talk about what it means to you, to the user community:

In plain English this means that if you work with older versions of Elasticsearch or with other distros that aren’t “officially supported”, you’re now faced with one of two options:

1. Avoid Upgrades and Stay Behind

Avoid upgrades and stick with older versions of Beats (pre-7.13) to ensure that your logs and data keep flowing freely. This means however that you will not be able to stay current with the latest versions to benefit from bug fixes, enhancements, security vulnerability fixes and performance improvements.

2. Upgrade at your Own Risk

Upgrade to this minor release and take the risk that things stop working, either immediately or at any point down the road. 

It’s not theoretical. ELK users have already begun reporting things breaking for them upon upgrade, and have understandably complained about such a change made in a minor release.

Should you decide to upgrade to 7.13, I can give you one important piece of advice: 

Typically we treat upgrades of minor versions lightly on regression testing and other checks. In this case, although it’s released as a minor version, run extensive testing, at least as you would for a major release.

I use Logstash directly without Beats, so I’m off the hook right?

Well, not exactly. In fact, Elastic carried out a similar move with Logstash recently when it pushed license checks into the open source, to ensure it sends data only to licensed Elasticsearch. 

At Logz.io, we’ve started moving away from Logstash and Metricbeat already, but have kept recommending Filebeat as a good log shipper for many use cases. If you’re a Logz.io customer using Filebeat, then rest assured that the service keeps working even with Filebeat 7.13. But we cannot ignore the risk for our users. And it is unlikely to end here. 

We feel we can no longer wholeheartedly recommend Filebeat shipper for logs. 

Open Source Alternatives to the Rescue

Luckily the open source community has a rich and vibrant ecosystem of shippers for telemetry:

Log Collection and Aggregation Open Source Alternatives to Filebeat

For log data there are Fluentd and Fluent Bit. Fluentd is more flexible, with hundreds of plugins and the ability to aggregate multiple sources, while Fluent Bit is more performant but limited in functionality and focused more on log collection and forwarding. The Fluentd-Fluent Bit relationship somewhat resembles that of logstash-filebeat pair. Fluentd also recently added capabilities for service metric data. You can read more about Fluentd and Fluent Bit here.

We at Logz.io found Fluentd to be a good OSS for sending log data. It is a mature project that’s been widely adopted in the Kubernetes community (which replaced the ELK with EFK). As such, we provide integrations for both Kubernetes and Docker based on Fluentd. 

We provide an easy way to send logs with Fluentd using the fluent-plugin-logzio plugin.

If you run on Kubernetes, you can use a ready made Fluentd DaemonSet, which can also serve you well when running on a managed Kubernetes service such as AWS EKS, Azure AKS, and IBM IKS. 

If you want to collect logs from your Docker environment you can use our Docker log collector based on Fluentd, which also provides native integration with HTTP proxies. We will keep increasing these integrations to make sending your data even easier. 

Metrics Collection and Aggregation Open Source Alternatives to Metricbeat

For metric data there’s Telegraf, which offers over 200 pre-built integrations, including Elasticsearch output. Telegraf is also the metric collector OSS of choice in the TICK Stack (a popular OSS monitoring stack, including Telegraf-Influxdb-Chronograf-Kapacitor). Our blog compares Metricbeat and Telegraf.

Prometheus offers a pull based solution for metric collection, with Prometheus instance able to scrape metrics off a wide ecosystem or popular platforms, thanks to its rich suite of exporters. Collecting metrics in pull mode is especially useful for monitoring microservices. 

At Logz.io we adopt both Telegraf and Prometheus, and offer integrations for both, to serve metrics to our Prometheus-as-a-Service offering. 

OpenTelemetry: an Open Source, Unified, Vendor-Neutral Future for Telemetry Collection

In the longer term, the industry seems to converge around OpenTelemetry as the unified open source framework and standard for generating and collecting logs, metrics and traces. 

OpenTelemetry is a young project compared to Filebeat, Fluentd and others. Its tracing specification reached GA earlier this year, and is still pre-GA for logs and metrics, but it is advancing, backed by the CNCF community and all major vendors and cloud providers. 

With OpenTelemetry in GA, things will become simpler, unified, eliminating the risk of vendor locking. 

We at Logz.io already provide pre-built integrations with OpenTelemetry enabling metric and trace data shipping to our service, and we are constantly working to extend this offering.

We’re also participating in and contributing to OpenTelemetry to help make it the best open source for our users and for the community at large.

Endnotes

ELK has been a great open source stack, and a hugely popular one for observability. But unfortunately the ELK Stack open source appears to be slipping between our fingers, as some vendor-owned OSS end up doing.

Elastic has made another anti-#opensource step, making breaking changes to its #Beats shippers, so they will no longer send data to non-Elastic distros of #elasticsearch.
That's the result of a vendor owned open source. https://t.co/l71NcPmaKe pic.twitter.com/EIREkAp2Py

— 𝙳𝚘𝚝𝚊𝚗 𝙷𝚘𝚛𝚘𝚟𝚒𝚝𝚜 #CNCFAmbassador 🇮🇱🎗 (@horovits) May 30, 2021

Thankfully there’s a selection of open source alternatives offering a path forward, and I’m proud that my team at Logz.io remains committed to open source and to the community, and not just on the data shipping side.

Following the relicensing of Elasticsearch and Kibana from Apache 2.0 to non-OSS licensing, we’ve contributed to the new OpenSearch project, which offers a potential successor to Elasticsearch and Kibana. 

I strongly believe in staying true to the open source path and contributing to it to get the best tools out there. After all, open source is eating the world, so make sure you get onboard.