How to Secure a Kubernetes Cluster

Kubernetes is one of the most advanced orchestration tools that currently exists in the software world. It provides out-of-the-box automation for environment maintenance and simplifies deployment and upgrade processes. It has different implementation types (on-premise, cloud-managed, hybrid, and more), multiple open-source supporting tools, and supports a wide range of configuration options. But the task to secure Kubernetes is gaining added urgency.

As Kubernetes continues to improve and develop, however, it is also becoming increasingly exposed to new security vulnerabilities and threats. Just last month, CNCF (Cloud Native Computing Foundation) performed a Kubernetes security audit survey and released its results to the public. The audit’s results were not promising and should raise a red flag for all Kubernetes users. According to the audit, attackers can take advantage of different Kubernetes vulnerabilities and several default configurations to execute malicious operations—such as running unpermitted, and sometimes unperceived, code—on the cluster content and gain access to other infrastructure components.

These vulnerabilities mean that securing a Kubernetes cluster has become mission-critical for DevOps teams. This article will review the different modules comprising a Kubernetes cluster and the different protection layers they require and will provide some best practices for applying security strategies.

Kubernetes pod protection

Every Kubernetes cluster consists of services and deployments which are represented as pods running Docker containers. Each of these containers is a separate OS that requires protection from both external attackers and internal malware. There are several attack vectors for infiltrating or misusing the container inside a pod, and there are different types of defenses that should be set up to prevent such breaches. 

The first and most common step for protecting a pod is using a minimalistic OS (such as CoreOS). This can help with preventing malware or unwanted processes from running inside the pod and block unaccredited software from being executed as part of the cluster. 

The next step is configuring the communication routes of each of these pods. This helps ensure that only approved ports and destinations are being used by the service. This way, a software that managed to run inside a container cannot affect any other service, spread to other pods, or establish communication with the outside world. 

Another method for creating segregation is using namespaces inside the cluster. Using namespaces for each of your applications or application modules defines them as logical groups. This helps control who can access each module or service and what permissions they have while also protecting workloads in case any of them were compromised. 

Extending coverage and adding monitoring for these communications can provide insights into these requests in realtime and provide all the necessary information for responding to and preventing misuse. Tools that provide a “live” map of all communications inside the cluster as well as into and from the Kubernetes cluster can facilitate this process (e.g. Twistlock or Alcide). 

Kubernetes pod monitoring should also be used for detection of new pods with unfamiliar names or the unauthorized scaling configuration of existing pods. These can indicate that data exfiltration or even mining efforts are occurring. Such events should generate alerts that equip the DevOps and security teams with all the information they need to resolve an incident.

Secure Kubernetes host

Host—or node—security in Kubernetes is a world of its own. Different clusters can be launched with different OSs on their nodes, and each OS is exposed to unique vulnerabilities. Relevant hardening operations should be performed on every new node that is spun up in order to secure Kubernetes nodes.

Whether they are running on a cloud-managed cluster, an on-premise deployment, or a hybrid solution, Kubernetes-based applications often use disks to store application state and for persistency. These disks must be protected with permission management to allow only the authorized services to access the stored information. They should also be protected with data encryption in case the data somehow ends up in unauthorized hands. 

Kubernetes’ vulnerabilities are constantly being produced and discovered, and DevOps teams must stay on top of the Kubernetes CVEs, making sure they understand their relevance and evaluate their effects on applications. 

Kubernetes’ etcd comes with support for automatic TLS. It also offers authentication through client certificates for both client-to-server as well as peer (server-to-server or cluster) communication.

In-cluster authentication and authorization is another vital aspect of Kubernetes security. Having only certified users accessing and managing resources such as the CPU, memory, and disk space is crucial for maintaining an efficient and well-administered infrastructure. Enabling RBAC helps restrict access to permitted users and allows for integration with external user management systems such as Active Directory, LDAP, SAML, Github, and more.

The most common procedure for increasing security levels and creating trustful Kubernetes nodes is restricting Linux capabilities to the exact usage of the services and utilizing Seccomp correctly to constantly perform system updates and install kernel patches.

Monitoring the Kubernetes nodes resources, measuring them in real time, and tracing the processes running on the VMs helps to create a higher level of protection. It exposes any unapproved operation or software and provides DevOps with the ability to instantly respond and apply a solution.

Kubernetes network protection

Up to this point, we’ve been discussing the internal risks that Kubernetes clusters are exposed to. Unfortunately, however, most of the threats to these clusters are external. Whether it’s a DDoS attack, hackers trying to infiltrate the cluster for a long-term eavesdropping attack, the exposure of information through a misconfigured firewall, or the usage of shared network, attackers tend to eventually find a loophole. They can then penetrate the cluster and the application, affecting the infrastructure’s credibility and sometimes even the company’s reputation. 

VPC networks, configured in the right manner, can provide a level of isolation that allows information to flow into the cluster—but not out of it—to a non-permitted target. HPA, one of Kubernetes’ strengths, must be configured in order to support planned and unplanned system  load. There are several methodologies for configuring the horizontal pod autoscaler. Engineers must always be aware of these possibilities and have the ability to choose the right method for the relevant state of each service. A high availability solution, where mission critical services have pods in different data centers around the globe, is a great option for securing against data-centered attacks and supplying a built-in disaster recovery solution.

Encryption in transit should be set up to make sure all requests—both inside and outside the cluster—are protected and only permitted components can read and understand them.

Security validations that are executed automatically can improve the authenticity of the infrastructure and network configuration. Monitoring the results of these executions and the real time status of the network will upgrade a team’s awareness of potential attacks as they occur. 

Kubernetes cluster auditing

Every activity, operation, deployment or configuration change should be audited and stored. This log should show all activities that users execute in and to the cluster, including what happened, who did it, and when it was done. 

Kubernetes audit logging tracks most of this information, and a simple integration with the cluster API should provide the ability to ship these logs to external logging and storage systems. It can also generate dashboards and suspicious activity alerts or reports that can be used during an incident investigation.

Summary

Protecting a Kubernetes cluster is not a simple job. Due to the variety of attack vectors that exist, the regular evolution of technologies, and the steady adoption rate of this tool, attackers find it tempting to infiltrate clusters and either extract data or use the resources for their own needs.

DevOps teams basing their backend on Kubernetes technology must stay on top of all threats and vulnerabilities that a cluster, and the Docker containers running inside it, are exposed to and have a runbook for responding for every type of incident. And using telemetry data, such as metrics and logs, to keep eyes on a cluster, is crucial for identifying security breaches or malicious activities quickly. This is, of course, a topic unto itself and requires a deeper look. I recommend reading up on Kubernetes logging to get started.