#DevOps Tools
#Kubernetes
#Service Mesh

What Are the Hardest Parts of Kubernetes to Learn?

By: Gedalyah Reback
What are the hardest parts of Kubernetes to learn?

Many enterprises have already adopted Kubernetes or have a Kubernetes migration plan in place, making it clear that the platform is here to stay. While it provides a lot of benefits to its users, to take advantage of them, you need to thoroughly learn Kubernetes and how it works in production. Typically, the most difficult aspects of Kubernetes are learned through experience solving real-world problems. This post will focus on providing resources to help you do exactly that, while also explaining some of the core concepts behind Kubernetes.

1. Cluster Administration

If you have to start somewhere, learn Kubernetes cluster concepts. The major cloud providers (Google, AWS, Azure, etc.) all provide Kubernetes clusters as a service. This is a great place to get started. With just a couple of clicks in their cloud consoles, you can create a cluster.

Furthermore, the cloud providers manage the cluster, performing routine tasks like certificate rotation and version patching. If, however, you have to set up your own Kubernetes clusters, you’ll need to manage everything yourself, making the situation a bit more complex.

Described below are a few of the scenarios you might come across when managing clusters on your own, as well as some resources that can help you become more proficient at this process.

Bootstrap Kubernetes from Scratch

While you can easily use a Kubernetes bootstrapping tool like kubeadm to set up clusters, it’s important to know how the cluster is bootstrapped from scratch. Accordingly, you’ll be able to tackle production issues that may arise later on. The following two sources explain this well:

Bootstrap Kubernetes Using kubeadm

Kubeadm will automatically perform the configuration you had to do on your own in the previous section. As a result, it’s a pretty popular tool to bootstrap Kubernetes.

There are multiple architectures to decide from, and, during your set up process, you’ll have to answer questions like, “Should the architecture be highly available?” and, “Is a single master cluster sufficient?” Below are a few guides that will help you make these decisions:

Server Patching

You still need to maintain clusters after they are configured. It’s recommended that you update Kubernetes versions as soon as new stable versions are released, and these updates need to be done incrementally. In other words, if your current version is 1.15, and the latest version is 1.18, you can’t update your servers directly to 1.18. The following guides will help you to update Kubernetes with minimum disruption to your running workloads:

Prevent Resource Overuse

When you deploy your workloads in Kubernetes clusters, there’s a chance you’ll end up overusing the cluster node resources, a situation which can eventually lead to system crashes.

You can minimize these risks by learning Kubernetes pod limits and resource quotas, processes that the following resources tackle in depth:

Backup and Restore

Every system should have a backup and restore plan in place, and Kubernetes is no exception. The resources listed below describe how to envision and set up an effective plan. Note that the persistence layer in Kubernetes is a key value store called etcd.

2. Networking

Kubernetes networking-related issues are common in misconfigured Kubernetes systems. Networking is a core layer in Kubernetes, and, early in the process of bootstrapping a cluster, you need to make a decision about which container networking interface (CNI) you want to use in your cluster.

Depending on the CNI you choose, you may gain access to additional features like network policy support and encryption over network. Network policy support may be a critical requirement for you if you want to control ingress and egress traffic through your namespaces or pods inside the Kubernetes cluster.

CNI Providers

Choosing the correct CNI depends on your security policies, performance targets, and scalability, as well as the hardware running in your data center. These resources can help you to select the best CNI for your use case:

Network Policies

Kubernetes allows you to configure network policies in the cluster network. These act as a virtual firewall inside the cluster and allow you to fine tune and control traffic between pods and namespaces in Kubernetes.

In order to do this, you’ll need to configure a network policy supported CNI. The most common CNI used in tutorials, Flannel, doesn’t support network policies. The resources below examine the details of working with network policies:

Service Discovery

Service discovery in Kubernetes works with the component coredns, which requires your CNI to be configured properly. Coredns provides other functionalities as well, allowing you to configure a wide array of DNS plugins to suit your specific requirements.

3. Kubernetes Storage

Kubernetes is a PaaS that allows you to run workloads as containers. More often than not, these workloads will need to persist their state.

K8s supports a wide range of storage drivers natively, and the option to use external drivers exists as well. When choosing a storage driver, you need to consider performance, volume access modes, availability and scalability.

Kubernetes Volume Access Modes

Kubernetes supports three volume access modes: ReadOnly, ReadWriteOnly, and ReadWriteMany. Be careful when choosing the volume drivers, since some may not support all three modes.

Many drivers don’t support ReadWriteMany. However, if ReadWriteMany mode is important to you, the most commonly used driver is NFS. More information about volume access modes can be found at these links:

Persistent Volume Backup and Restore

After you’ve configured storage for Kubernetes, you need to learn Kubernetes backup and restore protocol. There are multiple tools out there to support this, and some storage drivers already have backup systems implemented. Here are a few resources addressing possible solutions:

4. Learn Kubernetes Security

Whether your Kubernetes cluster is running on-prem or in the cloud, its security is of utmost importance. An ill-configured Kubernetes cluster will be vulnerable to attacks.

As a rule of thumb, it’s best to avoid exposing the Kubernetes API to the public in order to reduce the surface area of potential attacks.

Of course, there are other ways to attack a cluster and consequently to prevent those attacks, many of which we describe here.

Service Accounts

Service accounts are the resources associated with Kubernetes’ authentication mechanism. They let you log in and use the Kubernetes API. From there, you can create multiple roles with specific permissions in Kubernetes and bind these roles to service accounts.

The majority of developers use the default kubectl config file from kubeadm instead of creating a separate user and a separate config file for every kubectl user. However, this isn’t advisable since it might open up security vulnerabilities.

The resources below can help you navigate service accounts:

Avoiding Root Containers

It’s best to avoid running containers in root mode. You can avoid containers running as root using the security contexts available in Kubernetes and by using container sandboxing methodologies if you’re running a multi-tenant system in a single Kubernetes cluster. The resources below describe these processes:

User Federation

Many enterprises use Active Directory or some other tool for user management. When you set up Kubernetes clusters, you must federate users to these clusters and configure access using LDAP or SAML protocols. Here are a few guides that can help you get started with this process:

Ingress Traffic

Your Kubernetes cluster will almost always have a few services exposed to the internet. To have more control over your cluster’s ingress traffic, send it through an API gateway.

Secure Secrets

The secrets you store in Kubernetes are just base64 encrypted strings. Consequently, just about anyone can easily decrypt them. Unfortunately, you cannot commit configuration files to a repository without exposing your secrets to third parties.

The resources in this section provide best practices and tools for storing your sensitive data in Kubernetes securely.

Kubernetes Security Best Practices

A lot of security implementations are required to secure your Kubernetes cluster. This section’s resources explain best practices for Kubernetes clusters.

Remember to always verify and benchmark your Kubernetes cluster to make sure that it meets the security standards specified by your organization.

5. Kubernetes Observability

Logging and monitoring (metrics) complete the setup of your production-grade Kubernetes cluster.

Metrics, also known as monitoring (though the term can sometimes be found extending to other aspects of observability) allows you to observe and act upon cluster behavior over time. Concurrently, logging lets you debug running workloads and observe their status.

Monitoring

There are many monitoring stacks, but the most popular combine Prometheus with Grafana and/or the ELK Stack. Both provide similar features (though Grafana is designed with metrics in mind; Elastic with a focus on logs).

Fortunately, if you’re using a cloud-managed Kubernetes solution, your cloud vendor will have a dedicated monitoring stack available to you. For example, GKE uses Stackdriver.

Below are a few guides describing how to set up monitoring in Kubernetes:

Logging

Containers are stateless. Logs are streamed to standard out, and log files are saved in the container engine log folder in the host operating system.

However, looking for these logs manually is incredibly tedious, so logs are typically aggregated and pushed to a centralized logging provider where users can easily look into them. The following list of links should help you navigate the details of logging:

Tracing

In a microservice environment, where you have hundreds if not thousands of microservices running, debugging gets complicated. All your requests hop off these microservices to complete one business domain request in your application.

Tracing manages this complexity by monitoring and analyzing request payloads and request hops between your services, allowing you to get a clearer picture of how your services are running in your environment.

Conclusion

Getting started with Kubernetes is easy; doing things the right way requires practice. To master it fully, you need to have hands-on experience using it to solve real world problems.

Sometimes, you need a little guidance from an expert on where to start looking and how to get going. There are a lot of different opinions out there regarding how to best achieve the outcomes we discussed above. As such, we’ve tried to provide a collection of some of the best resources to help you sort through them. The resources in this post have been compiled in hopes of providing you with that initial direction.

It’s up to you to dive into these articles and build a Kubernetes strategy that suits your requirements and needs. With a little time and effort, you, too, can be a pro-Kubernetes administrator!

Get started for free

Completely free for 14 days, no strings attached.
Start Free Trial
Schedule Demo