It’s no big secret that container orchestration is all the rage today. Being the topic of many articles and conferences, it can sometimes seem as though it is the ONLY topic worthy of discussion.
At Logz.io, we are now at the end of the process of migrating all of our containers into Kubernetes, and I would like to tell you the story of the process we went through when deciding which orchestration platform to use in the hopes of helping those of you who are still unsure of which tool to use or whether you need orchestration to start with.
What is orchestration and do I need it?
In my opinion, the first ground rule is that if you don’t know why you need orchestration you probably don’t. Container orchestration (and I’m purposely avoiding using the word Docker) is not for everyone and does not answer every need.
So what is orchestration?
Imagine you have 10 containers that serve different purposes. Using a bunch of instances and running these containers is pretty easy. When your application begins to grow and the number of containers you’ve deployed goes up to 100, pressure mounts but it’s still bearable. But when you find yourself managing thousands of containers, each with different versions, relationships and network configurations, things begin to get a bit crazy.
For companies using modern development techniques that heavily rely on containers, the challenge of scaling this type of architecture can be too much to handle.
And this is where orchestration comes into the picture.
The entire point of an orchestration infrastructure is to provide a simple way to “schedule” containers and let the underlying infrastructure do the rest.
It sounds like pure magic, but there is a lot of complicated software running this, and as extremely complicated software tends to be, everything works great until it doesn’t.
Narrowing down Container Orchestration Tools
There are five big names you will hear over and over again in the context of container orchestration: Kubernetes, Mesos (DC/OS), ECS, Swarm and Nomad. The process of deciding which of these tools to use will differ according to the company and individuals involved. Sometimes it will just boil down to personal preferences.
At Logz.io, we ended up with the two platforms named in the title of this article after a process of elimination.
- Swarm – and yes, this is a matter of opinion – was profiled as being too basic and simple for our needs. Good for testing but not really a tool we felt comfortable using in production.
- Amazon’s ECS has improved greatly since its initial release, but it still seems to be falling behind the other main players. Since we wanted a tool that was cloud agnostic, ECS was not really an option for us to start with.
- We felt Nomad was too young a project and not mature enough to be seriously considered but with all due fairness, it might deserve another evaluation in the future.
Which left us with two strong players — the ever growing in popularity and usage, Kubernetes, and the evolving DC/OS.
About DC/OS and Kubernetes
Just to make sure we are all referring to the same concepts, here is a short historical background and explanation to help clarify matters.
Mesos is a project by Apache that gives you the ability to run both containerized, and non-containerized workloads in a distributed manner.
It was initially written as a research project at Berkeley and was later adopted by Twitter as an answer to Google’s Borg (Kubernetes’ predecessor). To combat its high degree of complexity (Mesos is super complicated and hard to manage!), Mesosphere came into the picture to try and make Mesos into something regular human beings can use.
Mesosphere supplied the superb Marathon “plugin” to Mesos, which provides users with an easy way to manage container orchestration over Mesos.
In mid-2016, DC/OS (Data Center Operating System) — an open source project backed by Mesosphere — was introduced, which simplifies Mesos even further and allows you to deploy your own Mesos cluster, with Marathon, in a matter of minutes.
When referring to Mesos in this article, I am referring to DC/OS.
Kubernetes? Well, just in case you’ve lived on the moon for the past few years, Kubernetes is a container orchestration platform that was released by Google in mid-2014 and has since been contributed to the Cloud Native Computing Foundation.
Mesos vs. Kubernetes
The first thing to point out is that you can actually run Kubernetes on top of DC/OS and schedule containers with it instead of using Marathon. This implies the biggest difference of all — DC/OS, as it name suggests, is more similar to an operating system rather than an orchestration framework. You can run non-containerized, stateful workloads on it. Container scheduling is handled by Marathon.
Simplicity wise, Marathon’s general approach to APIs is straightforward in comparison to Kubernetes. Marathon aggregates APIs and provides a relatively small amount of API resources, whereas Kubernetes provides a larger variety of resources and is based on label selectors.
Third, there is an obvious difference in the level of popularity the two platforms enjoy. Why does this matter? For the obvious reasons — the size of the community-driven development and offering support. Kubernetes has almost 10x the commits and GitHub stars as Marathon.
DC/OS has a “Premium” subscription that opens up extra features, while Kubernetes is a completely open source.
Which brings me to the next bullet.
Why Kubernetes won
At Logz.io, I was a champion of DC/OS. I loved the simplicity of it, and the ability to run stateful workloads. I was fully ready to give up on some of Kubernetes’ strengths in favor of choosing DC/OS.
Then I discovered that a simple feature I needed to automate the deployment process is only included in the enterprise version. From that point on, and after talking to Mesosphere, we came to the realization that this might not be a one-time thing, and even if we overcome this specific hurdle, DC/OS is controlled by a commercial company, for better and for worse.
So began our Kubernetes project.
After a lengthy process of adapting our containers to the Kubernetes state of mind, and perhaps more significantly – after overcoming organizational and cultural challenges (a topic for an entirely different post), we are now managing hundreds of containers with Kubernetes.
This change has also facilitated more efficient Continuous Deployment by helping us shift all deployment responsibilities to our developers who now deploy new code multiple times a day. Bottom line – the move to container orchestration with Kubernetes has shortened the “Jira ticket -> Production” development cycle to 30 minutes.
Summing it up
So while orchestration platforms are one of the hottest technologies in town, it still doesn’t mean you actually need it, but in case you do – I hope I shed some light on the reason we chose Kubernetes over other existing solutions.
In a future blog post, I’ll dive deeper into the technical difficulties needed to safely move to Kubernetes, and the cultural changes we had to go through to make Logz.io continuously deployed.
I am not sure if I understand your reasons, so just to confirm. You selected Kubernetes over DC/OS because some functionalities are only available in Enteprise edition. If Mesosphere has different licensing like full source code available and focus on only providing enterprise support than you would choose DC/OS over Kubernetes or not?
At that point in time, yes.
I might have hit another wall of lacking DC/OS features, but this was the primary choice indeed. I feld like DC/OS gave me more freedom, and further simplicity
Are you considering running your Kubernetes on top of DC/OS in the future?
Currently is not on the roadmap, but its definitely not something i’m neglecting.
Another aspect I am missing here is possible combination of using both DCOS with Kubernetes. It is not about one or each other, but those 2 can smoothly cooperate and be deployed together.
Hi Archenroot. I thinks you re right when you say that those 2 can cooperate. What s your experience in ? My context is SMACK platform and microservices. Thanks
There is archived project and multiple statements from mesosphere about kubernetes on DCOS.
You can take a look at archived project Kubernetes on Mesos:
The core features are described as:
Features On Mesos
Kubernetes gains the following benefits when installed on Mesos:
– Node-Level Auto-Scaling – Kubernetes minion nodes are created automatically, up to the size of the provisioned Mesos cluster.
– Resource Sharing – Co-location of Kubernetes with other popular next-generation services on the same cluster (e.g. Hadoop, Spark, and Chronos, Cassandra, etc.). Resources are allocated to the frameworks based on fairness and can be claimed or passed on depending on framework load.
– Independence from special Network Infrastructure – Mesos can (but of course doesn’t have to) run on networks which cannot assign a routable IP to every container. The Kubernetes on Mesos endpoint controller is specially modified to allow pods to communicate with services in such an environment.
When we started, kubernetes on top of DC/OS was still in alpha.
It is indeed an interesting solution that we heavily considered, but just could not rely on in production
But how will you deploy your Kafka?
We deploy Kafka on bare EC2 Instances currently
I’d be interested to know which feature you needed to automate the deployment process is only included in the enterprise version?
Not sure if its still the case, but at that time – service accounts
I know I’m late to this party but I was wondering what solution you are using to host and deploy your kubernetes solution ? cloud instances ? own datacentre hypervisors ? bare metal ? nested virtual instance / virtual machines
Cloud instances currently