Tigera Logo

Guides: High Availability Kubernetes

High Availability Kubernetes Clusters: A Practical Guide

What Is the Role of High Availability in Kubernetes Cluster Deployment?

High availability (HA) in Kubernetes refers to designing clusters so that critical services remain accessible even in the presence of failures, such as node outages or network disruptions. In practice, this means setting up your Kubernetes environment to minimize downtime and maintain continuous operations for your applications and infrastructure components.

Kubernetes achieves HA primarily by replicating key control plane components, distributing workloads, and ensuring that both the control plane and application data are not single points of failure. This can include using redundant control plane nodes, and replicating the etcd data store that holds the cluster state. The focus is on resilience and quick recovery.

This is part of a series of articles about Kubernetes security.

In this article:

Importance of HA in Kubernetes

High availability in Kubernetes is crucial because many applications running in clusters are mission-critical. When these applications go down or perform poorly, the consequences can include lost revenue, reduced customer trust, and disruptions to business operations.

HA is not only important for production environments. Development and staging clusters also need to remain reliable to avoid delays across teams. For example, if a staging cluster becomes unavailable, development teams may be blocked from testing or deploying changes.

Because Kubernetes is a distributed system, it can fail in many ways—hardware issues, human error, misconfigurations, or cascading failures across loosely coupled microservices. Even planned events like upgrades or maintenance can temporarily affect availability if not handled properly.

By designing Kubernetes environments with HA in mind, teams can reduce the risk of these disruptions. Common approaches include eliminating single points of failure, using built-in Kubernetes mechanisms for redundancy and failover, and following design patterns that ensure applications remain resilient during outages or system changes.

Alongside availability, teams should pair these HA practices with strong container security controls, since an outage caused by a compromised workload is just as disruptive as one caused by hardware failure.

Two Architecture Options for High Availability in Kubernetes

Stacked vs. External etcd

In a stacked etcd architecture, the etcd database runs on the same nodes as the Kubernetes control plane components. This setup is easier to deploy and manage but increases the risk of a single point of failure—if a node goes down, both etcd and control plane services are affected. It’s suitable for smaller clusters or non-critical environments.

An external etcd architecture separates etcd from the control plane nodes. This decoupling enhances reliability and scalability by isolating the cluster’s key-value store from other components. External etcd clusters often use dedicated machines and quorum-based replication to ensure consistency and fault tolerance. While more complex to set up, this design is recommended for production-grade, high-availability clusters.

Multi-Zone or Multi-Region Deployments

Deploying Kubernetes clusters across multiple availability zones or regions provides protection against localized failures. In a multi-zone deployment, the control plane and worker nodes are distributed across several zones within the same region. This setup ensures that the cluster remains operational even if one zone experiences an outage.

Multi-region deployments go further by spreading the infrastructure across different geographic regions. While this enhances fault isolation and disaster recovery, it introduces complexity around data consistency, latency, and control plane coordination. Multi-region architectures often rely on global load balancers and external storage solutions to handle these challenges.

Tips from the Expert

In my experience, here are tips that can help better harden and optimize high availability (HA) Kubernetes cluster deployments beyond the content in the provided article:

  1. Implement quorum guard for etcd in external clusters:

    Deploy a quorum guard (or similar etcd quorum loss prevention mechanism) to avoid accidental etcd quorum loss during rolling updates or node failures. This ensures the cluster control plane remains operational even during maintenance windows.

  2. Use topology-aware scheduling for workload resilience:

    Enable and configure Kubernetes Topology Aware Hints (introduced in v1.21) to spread pods across failure domains (zones, racks) at the workload level, not just for the control plane. This prevents over-concentration of critical pods on a single failure domain.

  3. Automate etcd snapshot integrity checks:

    Don’t just take etcd backups—automate regular validation of the snapshots by restoring them to a test cluster and performing integrity checks on the data. This helps avoid discovering silent corruption only during disaster recovery.

  4. Harden node-to-node communication with mutual TLS and network policies:

    Use mTLS between kubelet, API server, and etcd, and deploy Kubernetes network policies or Cilium/Calico global policies to limit east-west traffic between control plane and worker nodes to essential ports only.

  5. Implement control plane audit policy rate limiting:

    If the control plane is under load, configure audit policy rate limiting to prevent audit logging from overwhelming etcd or disk IO during failure scenarios. This guards against cascading failures due to log floods.

Profile Picture

Peter Kelly

VP of Engineering

Peter Kelly is VP of Engineering at Tigera and Site Leader for Tigera's EMEA office in Cork, Ireland. He is responsible for all of Tigera’s Engineering teams and operations. Peter has two decades of experience in software development, including recently building control plane technology for open-source proxies at NGINX and later F5 Networks, where he held engineering leadership positions. Peter has a degree in Computer Science and a Masters in Advanced Software Engineering.

Tutorial: Creating Highly Available Clusters with kubeadm

To create a highly available Kubernetes cluster using kubeadm, you can follow one of two topologies: stacked etcd or external etcd. Both methods begin with preparing infrastructure and configuring a load balancer in front of the control plane nodes. The instructions below are adapted from the Kubernetes documentation.

1. Set Up Prerequisites

You need at least three control plane nodes and three worker nodes, each meeting kubeadm’s minimum system requirements. Ensure full network connectivity among all nodes, with kubeadm, kubelet, and a container runtime installed. You also need SSH access and superuser privileges on all nodes.
Create a load balancer for the kube-apiserver and configure it to distribute traffic across all control plane nodes. The load balancer must perform TCP health checks on port 6443.

2. Deploy a Cluster with Stacked etcd

With stacked etcd, each control plane node also runs an etcd member. Start by initializing the first control plane node:

sudo kubeadm init --control-plane-endpoint "LOAD_BALANCER_DNS:6443" --upload-certs

This command sets the cluster endpoint and uploads control plane certificates for other nodes to use. It returns join commands for both additional control plane and worker nodes.
On each additional control plane node, use the join command with the –control-plane and –certificate-key flags:

sudo kubeadm join :6443 --token --discovery-token-ca-cert-hash --control-plane --certificate-key

Apply a CNI plugin before proceeding, using a plugin compatible with your network configuration.

3. Deploy a Cluster with External etcd

In this topology, set up a separate etcd cluster first. Once etcd is ready, copy the CA and client certificates to the first control plane node. Then, create a kubeadm configuration file specifying the external etcd endpoints and credentials.
Run:

sudo kubeadm init --config kubeadm-config.yaml --upload-certs

This returns the same set of join commands for control plane and worker nodes. Join remaining control plane nodes one at a time using the provided command and apply a CNI plugin before deploying workloads.

4. Join Worker Nodes

Use the join command from the kubeadm init output to add worker nodes:

sudo kubeadm join :6443 --token --discovery-token-ca-cert-hash

This completes the highly available cluster setup. The cluster can now tolerate node failures and maintain control plane availability across zones or regions.

Best Practices for Implementing Kubernetes HA

1. Deploy Multiple Masters

The control plane is responsible for maintaining the desired state of the cluster, and its availability directly impacts cluster management operations. A minimum of three master nodes is recommended to maintain etcd quorum, which requires a majority to make decisions. This means the cluster can tolerate a node failure without losing the ability to manage workloads.

Master nodes should be placed across separate failure domains, such as different availability zones or racks. In addition to node-level redundancy, there needs to be a reliable way to route traffic to the control plane. This is typically achieved using a highly available load balancer that fronts the kube-apiserver instances on each master node.

The load balancer must support TCP-level health checks and failover, ensuring requests reach only healthy control plane nodes. It’s also critical to ensure that control plane components are managed as static pods or system services that restart automatically upon failure.

2. Use Resilient Networking Design

Any failure in the network path—from clients to the API server or between pods—can lead to degraded cluster functionality or complete unavailability. The HA networking design should include redundant physical or virtual interfaces, failover-capable routing, and resilient overlay networks.

Control plane access should go through a multi-node load balancer with TCP health checks to ensure traffic is always routed to a live API server. The CNI plugin should support dynamic reconfiguration and recovery after node or link failures.

DNS and service discovery must also be considered part of the HA network stack. If CoreDNS pods are not properly distributed or made redundant, name resolution failures can cascade across the cluster. Similarly, kube-proxy or service mesh components must be monitored and replicated for fault tolerance.

For ingress and north-south traffic, the Kubernetes Gateway API offers a more expressive, role-oriented model that can be configured with redundant gateway controllers to support HA traffic routing.

3. Continuously Validate Cluster Resilience

Resilience must be continuously validated through testing and monitoring. Use chaos engineering tools like Chaos Mesh, LitmusChaos, or kube-monkey to simulate failures such as node crashes, network partitions, or etcd disruptions. These tests help identify blind spots in failover handling, misconfigured pod disruption budgets, or insufficient redundancy.

They also validate whether workloads automatically recover and whether alerting systems correctly identify and report failures. Alongside failure testing, implement monitoring and observability practices. Key metrics to track include etcd health, kube-apiserver availability, pod restarts, and scheduler performance.

Alerting should cover conditions like quorum loss, high API latency, and pod eviction rates. Observability tools such as Prometheus, Grafana, and Fluent Bit help operators detect subtle degradation before they escalate into outages.

4. Back Up and Protect etcd Regularly

etcd is the single source of truth for the entire Kubernetes cluster state, making it a critical dependency for control plane availability. A failure or corruption in etcd can render the entire cluster unmanageable.

To reduce this risk, implement regular, automated backups of etcd data. Use the built-in etcdctl snapshot save command or schedule etcd backup jobs using Kubernetes CronJobs or external automation tools. Store backups in offsite or cloud storage to protect against local hardware failures.
In addition to backups, enable encryption of etcd data at rest and restrict access to etcd endpoints using strict network policies and TLS authentication. Periodically test the restoration process by spinning up a test cluster and restoring from backup, to ensure recovery procedures are reliable and documented.

5. Use Static Pods or Systemd Services for Control Plane Components

For self-managed Kubernetes clusters, deploy control plane components (kube-apiserver, kube-controller-manager, kube-scheduler) as static pods or systemd-managed services. Static pods run directly under the kubelet and are automatically restarted by the kubelet if they crash or are deleted, improving resilience.

By running these components as static pods, you ensure that they don’t depend on the Kubernetes API server itself for scheduling or management, which avoids circular failure scenarios during outages.
Alternatively, on some distributions or in more advanced setups, running control plane components as systemd services provides finer control over restart behavior and monitoring through native OS tooling.

Related content: Read our guide to Kubernetes monitoring

Supporting Kubernetes HA with Calico

Calico delivers high availability for Kubernetes by providing redundant network paths, multi-cluster connectivity, automated failover mechanisms, and integration with enterprise network infrastructure, ensuring resilient and reliable application deployments. Calico supports Kubernetes high availability (HA) through several key features and architectural options:

Dual Top-of-Rack (ToR) Peering

Calico Enterprise enables clusters to connect to two independent ToR switches per rack, providing active-active redundant network paths. If one switch or link fails, the cluster nodes remain connected via the other, ensuring continuous network connectivity and preventing service downtime. This setup automates BGP peering configuration between cluster nodes and ToR switches, reducing manual effort and operational complexity. Kubernetes alone does not provide this level of network redundancy; Calico fills this gap for critical applications that require HA at the network layer.

Cluster Mesh for Multi-Cluster HA

Calico Cluster Mesh extends Kubernetes networking across multiple clusters, enabling seamless service discovery and direct, low-latency communication between pods in different clusters. This is essential for high availability, disaster recovery, and shared services scenarios. If one cluster becomes unavailable, services can fail over to another cluster, maintaining application uptime. Cluster Mesh optimizes routing to avoid unnecessary network hops and does not require an additional management plane.

Egress Gateway High Availability

Calico supports load balancing egress traffic across multiple egress gateways. If one gateway fails, traffic is automatically rerouted to healthy gateways, ensuring uninterrupted outbound connectivity for pods. Health checks continuously monitor gateway status, further improving reliability.

On-Premises HA Network Design

Calico supports advanced on-premises network designs, such as native pod routing and dual ToR configurations, to ensure both node and pod network availability. These designs integrate with existing infrastructure and security processes, providing robust HA for enterprise environments.

Next steps:

X