OpenSearch: The Open Source Successor of Elasticsearch

What an exciting OpenObservability Talks episode it was! On May 27, I hosted Kyle Davis, Senior Developer Advocate for OpenSearch at AWS, for a chat about the OpenSearch project, where it stands and where it’s heading. I wanted to share with you some interesting insights from our chat. You’re more than welcome to check out the full episode.

How it All Started

Fireworks started in the Elasticsearch community in January with an announcement by Elastic, when they announced their decision to relicense the Elasticsearch and Kibana open source projects, switching from the Apache2 OSS license to a dual license scheme with the non-OSS SSPL and proprietary Elastic licenses. 

That placed many services that rely on Elasticsearch in a legal predicament, as we discussed on a previous episode of OpenObservability Talks.  

Following the announcement, community members began looking for alternative open source projects, as well as discussing the option to fork Elasticsearch and Kibana. For our part, the team at Logz.io started discussing the forking option internally, and also opened discussions on the subject with other community members and software providers. 

In our podcast chat, Kyle shared how the AWS team was undergoing the same process at the time, and how they also decided to go down the fork path. Thanks to its size and agility, AWS quickly assembled the necessary engineers internally and almost immediately began the forking process in full. 

After several months of hard work led by AWS, OpenSearch Alpha was released in April 2021 under the Apache 2.0 open source license, which introduced the projects: 

OpenSearch as a fork of Elasticsearch 7.10.2, and OpenSearch Dashboards as a fork of Kibana 7.10.2 (the last Apache2-licensed versions of these projects before Elastic’s announcement). OpenSearch Beta 1.0 came out just a month later, and already included plugins. 

This wasn’t simply hitting the fork button (not by a long shot)

Getting the alpha to launch did not come about simply by “hitting the fork button.” Not by a long shot.

Rather, the engineers involved found themselves turning over numerous rocks and finding new things to dig into almost every day. They immediately realized that Elasticsearch and Kibana contained mixed code of different licenses in a single repo, with Apache2-licensed (open source) code intertwined with Elastic’s (non-open) X-pack code. 

The team found itself constantly pulling the proprietary code out of the open source, some of which required traversing the code line by line. Inside, they also found Elastic’s marketing and branding elements, pop-up messages, dial-home features, and a lot more. 

Not exactly the “fork it!” OSS mentality. 

These sorts of things kind of make me wonder if vendor-owned open source is an oxymoron.

What’s the Plan for OpenSearch?

The OpenSearch project is full-steam-ahead to get v1.0 to GA. The goal of OpenSearch 1.0 is to provide feature completeness with minimum differences from Elasticsearch and Kibana 7.10.2 (from which it was forked).

But there’s something bigger than feature coverage involved here. While OpenSearch 1.0 won’t bring a wealth of new capabilities, it brings a clean open source project. There will be no proprietary code mixed in, and no vendor-dial-home or marketing elements embedded within the open source repo. It’s just a clean Apache 2.0 codebase that you can use, reuse, modify, and even fork with ease. 

I’m not sure how much people outside the project or unfamiliar with open source projects appreciate that, but I know that those who’ve ever had to dig into Elasticsearch and Kibana codebase surely do.  

Beyond v1.0, things will get even more interesting once the community starts adding features and capabilities to make users’ lives easier. Following the semantic versioning conventions, no ‘breaking changes’ should be introduced before v2.0. That means that v1.0 should work pretty smoothly for Elasticsearch and Kibana v7.10.2 users.

Into Release Candidate and Beyond: OpenSearch Releases

At the time of writing, the Beta is out, with RC1 expected in a matter of days. Finally, GA is expected sometime within the following month. But what do these OpenSearch releases mean? And what’s expected next? Here’s a short overview:

• The Alpha version was the fork of Elasticsearch and Kibana 7.10.2 (Apache2-licensed), following a meticulous cleanup of the code for proprietary elements and dependencies, and setting both the build process and infrastructure in place. The Alpha was considered untested and feature-incomplete. It introduced the project name and a project that people could now compile (though not without some sweat).

• The Beta version enabled users to try OpenSearch out and let the team know if it worked. It added plugins from OpenDistro into the bundle and came with downloadable artifacts –Linux TAR and Docker images – which provided a smoother ‘getting started’ experience for users. The Beta was feature-complete, but untested. 

• The RC (release candidate) version is feature complete and tested, but not validated. Current plan is to have only one RC version before GA. According to the current roadmap, RC1 is currently planned for June 7.

• The GA (general availability) version will be feature complete, tested, and validated. GA is planned for July 12, according to the roadmap.

The OpenSearch Roadmap was also recently published on GitHub. There’s also a clear process for community members to add things to the roadmap and take part in shaping it. See the FAQ for guidelines on that.

Growing Adoption and Community

Looking at GitHub, we are currently 123 contributors-strong for the OpenSearch project, which is a pretty accurate number. There are also many external contributors (outside of GitHub) on AWS that worked on OpenSearch Beta 1.0.

There are also partners of the project, organizations showing firm commitment and active involvement – partners like Aiven, Bonzai Search, and of course Logz.io. We’re looking for more contributors and partners to get involved and contribute to the project (see the end of the post for more).

There are also early signs of companies testing out OpenSearch as an Elasticsearch replacement. Earlier this month, Wu Sheng shared on a post that Apache Skywalking has looked into moving from ES to OS as their storage database. It’s not surprising, as open source projects in particular fundamentally require an open source stack. It’s something that we’ll see more often on the heels of the RC and GA releases. 

It’s also worth noting that the project is working on a lean artifact that will provide just OpenSearch, without plugins, which would make it easier to embed within individual projects, rather than using it as a stand-alone tool.

Shared Governance and Open Source Foundations

AWS decided from the outset not to own OpenSearch but release it as a shared governance project. Admittedly, AWS built the Alpha version largely as an internal project (though we kept an open channel to help shape it), but the repos were released to the community along with the brand name at the Alpha release. 

Since then, people outside of AWS have started contributing, but with AWS members being the only ones approving code.This has raised an active discussion in the community about governance and moving that to a foundation, as no one wants to find themselves again under a benevolent dictatorship.

Kyle reaffirmed AWS’ commitment for shared governance. Kyle said that AWS prioritized releasing OpenSearch 1.0 while following the AWS principle of “It’s ok to be misunderstood for a period of time” (and clarifying later with the community). 

Now, the current stage is aimed more at involving people who can commit code and actively contribute. 

The next step is to bring non-Amazonians on board with permission to approve code, a contributorship path for which being expected in a matter of weeks.

After that, Amazon’s view is that the next stage should be to formalize the governance model and determine how decisions are made, how code is committed, how the roadmap is put together, and so on. 

After governance is formalized, AWS expects to start looking into where the project lives long-term:under an existing foundation, a new foundation, or otherwise. 

Kyle emphasized that approaching foundations needs the preceding steps of an active community, across many organizations, and governance in place. You can’t jump the line.

Kyle admitted that it’s going to take a while, as Amazon has some adjustments to make internally to properly adapt to the process.

How You Can Get Involved in OpenSearch

The project is out on GitHub, and now it’s your time to get involved.

How? Simple! Check out the GitHub repos of the project, open issues, comment on conversations, consider code contributions. In each repo you’ll find a contributing document with guidelines to help you. Also join the meetups and forum discussions. You can find all the information and links from the project website Opensearch.org.

And feel free to reach out to Kyle at @stockholmux or to me @horovits (on Twitter, GitHub or elsewhere).

It’s up to all of us to turn it into a community project that serves as a good OSS successor for Elasticsearch.

I’m looking forward to seeing many of you becoming active in the OpenSearch community!