Native Kubernetes cluster mesh with Calico

As Kubernetes continues to gain traction in the cloud-native ecosystem, the need for robust, scalable, and highly available cluster deployments has become more noticeable.

While a Kubernetes cluster can easily expand via additional nodes, the downside of such an approach is that you might have to spend a lot of time troubleshooting the underlying networking or managing and updating resources between clusters. On top of that, a multi-regional scenario or hyper-cloud environment might be off the limits depending on the limitations that a cloud provider or your Kubernetes distro might impose on your environment.

Calico Enterprise cluster mesh is a suite of features native to Kubernetes with a multi-layer design that connects two or more Kubernetes clusters and seamlessly shares resources between them. This post will explore cluster mesh, its benefits, and how it can enhance your Kubernetes environment.

Projects that provide cluster mesh

Multiple projects offer cluster mesh, and while they are all similar in basic principles, each has a different take on implementing this solution in an environment.

The following table is a brief overview of notable projects that offer cluster mesh:

* Cilium uses an identity-based approach injected into packets for its design; while it can be adjusted to accommodate more clusters, it taps into the local identities, limiting the number of local identities you can have in a cluster.
** Submariner can NAT traffic between two clusters, which helps with cases where you have low IP addresses. However, NAT will create problems, such as losing the source IP address.

Calico Enterprise cluster mesh

As previously described, cluster mesh is a multi-layer solution that seamlessly connects two or more clusters. On a fundamental level, cluster mesh starts with network interconnectivity between two or more cluster resources, allowing Pods and services to communicate between these clusters.

Calico Enterprise cluster mesh, however, doesn’t leave you after networking is established. It provides a suite of tools that allow you to easily achieve Federated Identities and federated services. If you are not familiar with these concepts, don’t worry. The coming sections of this blog will shed some light on each feature and its use cases.

Let’s review a scenario to understand the benefits of Calico Enterprise cluster mesh. Imagine that we have two clusters in different regions in a cloud environment. Each cluster has a deployment of your application running on it with services and other Kubernetes native resources that make your services available for your users.

Cluster mesh networking

Calico Enterprise Federation provides an Encapsulation layer based on VXLAN technology that can be utilized to establish networking in environments where BGP or IPIP is limited (For example, Azure).

The networking part can be established in different ways:

• The direct method is usually used when the underlying networking fabric can route traffic to the internal resources within the cluster.(Usually on-premises environments)
• The overlay method is used when the underlying networking fabric doesn’t know about the internal cluster CIDR. (Usually a cloud deployment)

For our scenario, we would need to use overlay because the underlying network infrastructure in AWS is unaware of our internal Kubernetes network, and it will drop any IP addresses that are not part of our AWS VPC. Calico cluster mesh overlay cloaks the traffic by encapsulating it inside VxLAN and sending it to the remote cluster.

Calico open-source multi-cluster networking is established via BGP. To learn more about these concepts, click here.

Cluster mesh federated identities and services

Cluster mesh allows you to expand your environment beyond what is possible with a single cluster. However, this expansion doesn’t necessarily allow you to use resources or refer to objects from local clusters. This notion creates a gap in what could be achieved with a regular cluster mesh setup.

Calico Enterprise federation builds on this underlying network connectivity to establish federated features such as load-balancing over multiple clusters, high availability between various regions (sites or cloud providers), and unified network policy selectors for enforcement across clusters. This ability is achieved by linking Calico components between two clusters to exchange information about the local setup.

To accomplish this, Typha, a key component of Calico Open Source and Enterprise, which acts as a caching layer between Calico and the Kubernetes API server establishes a link between various clusters in your mesh, and it aggregates the information in all clusters, allowing you to use references and objects from remote clusters in local policies or services.

Federated Identities

Federated identities represent the relationships between endpoints and services across different clusters. These relationships are managed by the Calico Enterprise manager which is responsible for maintaining the consistency of service endpoints and their corresponding configurations.

For example, when an endpoint is created in Cluster A, Calico exchanges that resource’s pod/host identity with all connected remote clusters to be stored in their Typha resource cache. From here, any changes to this resource in the local cluster will be propagated to the remote clusters, allowing all participating meshed clusters to have the latest information about the resources.

Shared services

A federated service consolidates multiple services into a single endpoint. Calico Enterprise identifies services across a cluster mesh (local and remote) and creates a federated service on the local cluster that has endpoints from both local and remote services. The Tigera Federated Service Controller manages these federated services, ensuring their endpoints are up-to-date, but it doesn’t modify configurations on remote clusters.

The following example annotates a service to participate as a gateway for federated clusters, and will include endpoints from all services in the cluster mesh that match the label selector:

apiVersion: v1
kind: Service
metadata:
 name: my-app-federated
 namespace: default
 annotations:
   federation.tigera.io/serviceSelector: run == "my-app"
...

Isolation and security

Since federated clusters share knowledge of resources, we can use Kubernetes-based labels to refer to workloads from remote clusters and create policies that can easily target these remote workloads.

Similar to a single cluster policy, all you need to do is create a security policy resource with the resource labels you want to target. Calico Federation takes care of translating labels to the remote endpoints. On top of that, you could also use unique federated labels such as federation.tigera.io/remoteClusterName to refer to a single cluster resource.

The following sudo policy highlights a simple policy that allows outgoing traffic to flow out of clusterA and reaching clusterB if port is 80 and denies ICMP traffic:

apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
  name: block-icmp
spec:
  selector: app == 'clusterA'
  types:
  - Egress
  egress:
  - action: Allow
    protocol: TCP
    destination:
      selector: app == 'clusterB'
      ports:
      - 80
  - action: Deny
    protocol: ICMP
    destination:
      selector: app == 'clusterB'

Conclusion

Calico Enterprise cluster mesh provides extensive functionality in a multi-cluster environment. With its established connectivity, it offers high availability and better scalability across multiple regions, as all meshed clusters have information about the resources in your environment. Shared services among clusters add a layer of high availability for your workloads. In the event of a cluster malfunction, these shared services and endpoints can be used to shift traffic to other functioning clusters in different sites or regions.

Learn more about using Calico to create a Kubernetes cluster mesh for multi-cluster environments.