When setting up a Kubernetes environment with Amazon Elastic Kubernetes Service (EKS), it is crucial to understand your available networking options. EKS offers a range of networking choices that allow you to build a highly available and scalable cloud environment for your workloads.
In this blog post, we will explore the networking and policy enforcement options provided by AWS for Amazon EKS. By the end, you will have a clear understanding of the different networking options and network policy enforcement engines, and other features that can help you create a functional and secure platform for your Kubernetes workloads and services.
Amazon Elastic Kubernetes Service (EKS)
Amazon Elastic Kubernetes Service (EKS) is a managed Kubernetes service that simplifies routine operations, such as cluster deployment and maintenance, by automating tasks such as patching and updating operating systems and their underlying components. EKS enhances scalability through AWS Auto Scaling groups and other AWS service integrations and offers a highly available control plane to manage your cluster.
Amazon EKS in the cloud has two options:
- Managed
- Self-managed
Managed clusters rely on the AWS control plane node, which AWS hosts and controls separately from your cluster. This node operates in isolation and cannot be directly controlled or accessed via methods such as SSH. While a limited number of configurations in the control plane node can be adjusted through Kubernetes resources, users do not have direct control over the instance or its routing table.
Managed instances are tightly integrated with AWS services, enabling automated provisioning, updates, scaling, and other management tasks. AWS handles these aspects automatically, and their support team assists in troubleshooting any issues that may arise.
Self-managed instances offer users greater flexibility and control. With self-managed clusters, users are responsible for deployment, scaling, and updates of the instances. While AWS support extends to self-managed instances, it is limited to the aspects partially managed by AWS, with users retaining greater responsibility for the overall management of these instances.
Bring your own cloud (BYOC) option is mostly used by organizations that may require stringent control over networking, instance management, updates, and overall environment setup due to compliance or security requirements. The BYOC model may be ideal, as it permits full customization from the operating system to network routing, thereby enhancing data security and compliance. However, the main drawback of BYOC is the need for considerable IT expertise and resources to manage and maintain the cloud environment.
Pick one and stick with it. Your deployment flavor is chosen at the time of cluster setup and can not be changed on the go. If, for any reason, you want to change your deployment variation, you have to recreate your environment and applications accordingly.
Kubernetes networking
AWS-CNI as CNI
Networking in AWS involves essential components like VPC, ENIs, ACLs, and others. After establishing AWS resources for a Kubernetes setup, you must select a CNI, which serves as the critical connection facilitating internal network communications within your cluster and enabling external resources to interact with your internal services.
AWS offers an official network called AWS-CNI, for the connectivity within workloads running on Amazon EKS service. It assigns workload IP addresses from the available addresses associated with your instance’s ENI and within the primary range of your VPC. Assigning IP addresses from your VPC has its advantages (mainly performance), but it presents an issue that can limit your ability to scale your environment after a certain point. To solve this issue, you can implement custom networking for your VPCs.
Custom networking
Each AWS account is limited to 5 VPCs per region. When provisioning a VPC, you have to choose a private IP range, which is the range that resources assigned to that VPC share. Since AWS-CNI relies on the IP addresses from VPCs, deployments with numerous workloads can quickly cause VPC IP exhaustion.
In such a scenario, you could use a secondary IP range to provide IP addresses to your Kubernetes pod. At the same time, other AWS resources get their IP and connectivity from your primary VPC range.
Calico as CNI and Network Policy Engine
Using Calico as your CNI allows you to expand your workload scale without issues related to IP address exhaustion or ENI IP limits.
In addition to networking, Calico also has an IP address management (IPAM) component that allows the creation of private networks within your cluster through IP pools, and these IP pools can be assigned to workloads and tunnels. It’s important to note that this architecture requires an overlay network, with VxLAN being the recommended option. This is because when you choose any CNI to create a private network within your cluster, the underlying network fabric is unaware of the private IP CIDR. This setup can lead to traffic being dropped as it moves from one node to another. To address this, Calico encapsulates your traffic with an encapsulation technology (VxLAN/IPIP) to allow smooth transportation of your valuable data.
On top of that, Calico’s pluggable data plane allows you to choose between technologies such as IPtables, eBPF, and VPP to get the best performance for your environment.
Calico also offers robust traffic encryption via its WireGuard integration and a more refined policy engine that extends the basic Kubernetes policy model with features such as application layer policy, explicit actions, and others.
The following table compares Calico and AWS-CNI features to help users achieve their desired outcome.
| EKS Managed/Self-managed | ||
| CNI | AWS-CNI | Calico |
| Networking | IPtables | IPtables, eBPF, VPP, Windows HNS |
| Policy enforcement | eBPF-based (AWS Network policy engine) | IPtables, HNS, eBPF based |
| Application layer policy | Not supported | Envoy, Istio, Linkerd |
| Observability | eBPF-based (AWS Network policy engine) | IPtables logs |
| Cluster mesh | Not Supported | BGP, VxLAN* |
| Encryption | Not Supported (You may use VPC native encryption) | WireGuard |
| Overlay network | Yes, Secondary IP but limited to the number of IPs that he instance ENI supports. | Yes, VxLAN and BGP. Calico IPAM can generate as many private networks as required. |
| IPv6 | Yes | Yes |
* Requires Calico Enterprise
** Because the AWS control plane is isolated, users cannot influence the machine routes, which means private networks are not accessible by the control plane.
Calico as a network policy engine
Calico can also be used as a policy enforcement engine for your cluster. In such a scenario, networking is done by AWS-CNI, and Calico is used as a policy engine.
Calico offers a fully compliant policy engine that is not vendor-locked and deploys policies by utilizing technologies such as iptables, eBPF, Windows HNS, and others. Calico extends the original Kubernetes policy model and adds many features, such as global network policies, a fine-grained syntax, and tuneable policies supporting hostendpoint protection to defend against host-level attacks.
Calico network policy resources also add some crucial features to the basic Kubernetes policy model, such as policy order, explicit action, traffic logging, service account selectors, logical selectors, and others.
Before the release of Kubernetes 1.25 on the EKS platform, policy enforcement was officially encouraged via Calico CNI due to the lack of policy enforcement support within the AWS-CNI plugin.
AWS network policy agent
Recently, AWS released an open-source companion to their CNI called Amazon Network Policy Controller. This controller instructs the Amazon network policy agent to deploy eBPF programs on the node to enable policy enforcement in your cluster.
| Calico policy engine | Amazon network policy agent | ||
| Policy Technology | IPtables, nftables | eBPF based | eBPF based |
| Kubernetes policy compliant | Fully compliant | Fully compliant | Partially |
| Application layer policy | Supported via Istio, Envoy | Not supported | |
| Global network policies | Yes, via Calico global network policies | Not supported | |
| Host protection policies | Yes, via hostendpoint policies | Limited to IP CIDR | |
| Policy order | Yes, policies can be prioritized | Not supported | |
| Policy action | Yes, Allow, Deny, Log, Pass | Not supported | |
| Unique policy resources | Yes, Calico supports service accounts, extended match operators | Limited to Kubernetes basic model | |
Conclusion
The default networking engine for EKS on the AWS platform, AWS-CNI, delivers networking by allocating the primary or secondary IP addresses from a VPC to your workloads, leading to a layer of complexity for environments at scale. Calico ensures you can scale your workloads without concerns about IP address exhaustion or ENI IP limits.
For policy enforcement, AWS-CNI supports policy enforcement through the Amazon Network Policy Controller, which uses eBPF programs to enforce Kubernetes policies. Calico supports policy enforcement and can be installed as a standalone policy engine on top of AWS-CNI. It provides a fully compliant policy engine with capabilities like global network policies, host endpoint protection, and fine-grained syntax. Calico policies provide precedence, explicit action, and service accounts that allow you to fine-tune your cluster security.
To build a secure and efficient Kubernetes environment with EKS, it’s essential to understand and utilize these networking options, whether your focus is on performance, flexibility, or policy enforcement.
Ready to try Calico network policies? Sign up for a free trial of Calico Cloud.
