From Spotify to Open Source: The Backstory of Backstage
Technology juggernauts–despite their larger staffs and budgets–still face the “cognitive load” for DevOps that many organizations deal with day-to-day.
That’s what led Spotify to build Backstage, which supports DevOps and platform engineering practices for the creation of developer portals. Eventually, Spotify made the decision to open source Backstage and donate it to the Cloud Native Computing Foundation (CNCF), and the project’s popularity has since skyrocketed, with over 21.5K stars and 4K forks on Github and over a reported thousand adopters to date.
On my recent episode of OpenObservability Talks I hosted Lee Mills, senior engineering manager at Spotify, to hear first hand about the early days of Backstage at Spotify, as well as the project’s fascinating journey to opensource, starting with one internal hack week. We also discussed surprising new verticals for Backstage, such as banking and healthcare.
The Need For Developer Productivity Drives Backstage’s Creation
Spotify built Backstage with the desire to increase developer productivity and enhance developer experience. The hypergrowth of Spotify and the lean engineering methods created many point solutions developed by different teams, oftentimes reinventing the wheel due to not knowing of similar utilities developed by other teams in the organization.
According to Lee, These were the original drivers for Spotify engineering to developer Backstage as an internal tool, to empower developers to take ownership. This echoes what I see in other large organizations in our industry.
Backstage provides three key functions: create, manage and explore. The “create” piece is provided with Software Templates that allow developers to quickly spool up services or components and build in company standards and golden paths. The “manage” piece is achieved with the Catalog, a central inventory that enables easy lookup of services throughout the organization, and the various plugins that can be used with that service and in that context. The “explore” piece is about discoverability, namely in-context search for relevant documentation, stackoverflow of questions, CI/CD pipelines or other resources.
For example, a typical engineer at Spotify may have some knowledge of how to properly configure Kubernetes clusters, but it’ll take a lot of time and the need to go back and refresh without the right knowledge. Instead, it’s better to consume reusable software templates and plugins maintained by the subject matter experts in Engineering.
“Within Backstage, we have a Kubernetes plugin, and that plugin is owned by the expert and by the team that manages Kubernetes at Spotify,” Lee says. “They surface information and tools that are easier for me to quickly jump into, to manage things, to investigate things, and so on. It takes away some of that context switching and removes some of that cognitive load so I can go and focus on my end users.”
Backstage provides these through a friendly UI and helpful metadata, which provides a superior developer experience to that we’ve grown used to as developers. As Lee puts it, “From an engineering perspective, [Backstage] gives me the things I need relatively easily, in a nice UI, that I can see and brings those things to the forefront a little bit and become part of my flow.”
Streamlining Platform Engineering with Backstage and Internal Developer Platforms
Backstage was developed within Spotify’s Platform Engineering domain (Platform Mission in Spotify terms), as a tool to support the rest of the Engineering Mission to deliver their needs and goals. For adopters, Lee believes Backstage elevates the need around Platform Engineering.
“Platform engineering, in many instances, is core to the things that we’re doing, but it’s not really visible,” he says. “Backstage simplifies it so you can have that conversation with your stakeholders about platform engineering and quickly evidence the velocity gain that you can get by using tools like Backstage.”
This is also what I see among platform engineers I meet. In fact, a new category of Internal Developer Portals (IDP) is emerging, and even catching the attention of analyst firms such as Gartner, to meet this need and facilitate collaboration and communication between developers within an organization. Backstage fits neatly into this new trend.
Backstage’s Journey to Open Source
Over time, Spotify realized that contributing Backstage to the open source community was the right way to help DevOps across industries, as well as to enhance Spotify’s own engineering efforts with that of the community. Lee shared an example of a Spotify Engineering requesting a feature to support dark mode view in Backstage. This request which had been floating around for some time but had never gotten priority with Spotify’s engineering team. Once this request was published to the open source community, it was picked up and implemented within 24 hours, and in a way that was consumable by Spotify internally as well.
This is, to me, the power of open source in a nutshell. Moreover, when a project is released into the open, it takes a life of its own, to the extent that new use cases and verticals emerge. Last month at KubeCon Europe I spoke with engineers from Lunar digital bank on how they used Backstage for compliance and audit use cases. Lee says this is not coincidental, and that Backstage has proven within the Banking sector, and shared interesting reasoning around that.
The developer portal is always going to be open source, and Lee says it’ll continue to grow and get enhancements:
“We are an audio company, but that doesn’t mean we can’t share the expertise that we’ve got,” Lee says. “We’re gonna grow this and we continue to see other players entering the space. We want to support engineers and reduce cognitive load. We’re enabling others who have these skillsets but haven’t had a way to bring it to market…we hope we can continue to grow and build the open source community and also enable this new market of bringing productivity engineering to life from people as well.”
Want to learn more? Check out the OpenObservability Talks latest episode: From Spotify to Open Source: The Backstory of Backstage.
