When deploying a Kubernetes cluster, a critical architectural decision is how pods on different nodes communicate. The choice of networking mode directly impacts performance, scalability, and operational overhead. Selecting the wrong mode for your environment can lead to persistent performance issues, troubleshooting complexity, and scalability bottlenecks.
The core problem is that pod IPs are virtual. The underlying physical or cloud network has no native awareness of how to route traffic to a pod’s IP address, like 10.244.1.5 It only knows how to route traffic between the nodes themselves. This gap is precisely what the Container Network Interface (CNI) must bridge.
The CNI employs two primary methods to solve this problem:
- Overlay Networking (Encapsulation): This method wraps a pod’s packet inside another packet that the underlying network understands. The outer packet is addressed between nodes, effectively creating a tunnel. VXLAN and IP-in-IP are common encapsulation protocols.
- Underlay Networking (Routing): This method teaches the network fabric itself how to route traffic directly to pods. It uses a routing protocol like BGP to advertise pod IP routes to the physical network, making pods routable without encapsulation.
This guide dives into the technical differences of these options to help you select the right mode for your environment.
The Cost of a Mismatched Network Mode
An architectural mismatch between your CNI’s networking mode and your underlying infrastructure introduces significant technical debt.
3 key factors that should drive your decision are:
- Your Environment: This dictates the fundamental feasibility of using an underlay network. Are you in a public cloud (AWS, Azure, GCP) where the network is a managed service, or an on-premises data center where you control the hardware?
- Network Control: This determines your capability to implement the chosen approach. Do you have the administrative access and expertise to configure your network routers and switches?
- Performance Needs: Once feasibility (Environment) and capability (Network Control) are satisfied, this helps you optimize the choice between the remaining viable options. Are you running latency-sensitive workloads or general-purpose applications where minor overhead is acceptable?
It’s important to avoid an architectural mismatch between your CNI’s networking mode and your underlying infrastructure. Failing to align your networking choice can result in possible friction points:
🔻 Performance Penalty: Encapsulation is not free. It consumes CPU for packet processing and increases packet size, which can lead to MTU issues and fragmentation, increasing latency.
🔻 Operational Complexity: Choosing BGP without control over the network fabric or the necessary in-house expertise creates operational dead ends. Conversely, using an overlay in a high-performance data center fails to leverage the network’s capabilities.
🔻 Scalability Limits: At scale, the CPU overhead of encapsulation becomes a significant resource drain, potentially limiting pod density and increasing costs.
🔻 Poor Visibility: Encapsulated traffic obscures the original source and destination IPs from traditional network monitoring tools. For a NetOps team, all pod-to-pod traffic appears as a single, opaque stream (e.g., UDP traffic for VXLAN) between nodes, making troubleshooting difficult.
The key to avoiding these pitfalls is selecting the right tool for the environment. The following section provides the necessary technical context, detailing exactly how each networking mode functions so you can make an informed, long-term decision.
A Technical Breakdown of Networking Modes
Understanding the trade-offs requires examining the underlying mechanisms of each mode; VXLAN, IP-in-IP, and BGP, which allows you to see exactly where the performance cost or benefit originates.
| Mode | Type | Encapsulation |
|---|---|---|
| VXLAN | Overlay | UDP-based tunneling with ~50 bytes overhead |
| IP-in-IP | Overlay | IP-in-IP encapsulation with ~20 bytes overhead |
| BGP | Underlay | No encapsulation – native routing via BGP |
Calico supports all three primary networking modes detailed below, allowing you to choose the optimal data plane for your specific needs.
VXLAN (Virtual Extensible LAN)
| Description | VXLAN is an overlay technology that creates a virtual Layer 2 network over a Layer 3 infrastructure. |
| How It Works | When a pod sends a packet to a pod on another node, the local CNI component (acting as a VXLAN Tunnel Endpoint or VTEP) on the source node encapsulates the original packet inside a new UDP packet.
This outer packet is addressed using the source and destination Node IPs. The VTEP on the destination node decapsulates it and forwards the original packet to the destination pod. The VXLAN header adds approximately 50 bytes of overhead. |
| When to Use | VXLAN is the standard choice for public cloud environments (AWS, Azure, GCP) or any network where you lack direct control.
Since cloud providers don’t allow peering with their network fabric, overlays like VXLAN are often the only option. |
| Pros |
|
| Cons |
|
IP-in-IP
| Description | IP-in-IP is a simpler Layer 3 overlay protocol that wraps an IP packet directly inside another IP packet. |
| How It Works | The process is similar to VXLAN, but the encapsulation is more direct. The original IP packet is wrapped in a new IP header with the source and destination node IPs.
The protocol field in the outer header is set to This method adds approximately 20 bytes of overhead. |
| When to Use | Use IP-in-IP when you require an overlay but want slightly less overhead than VXLAN.
It can also be useful in environments where network firewalls block UDP traffic but permit other IP protocols. |
| Pros |
|
| Cons |
|
BGP (Border Gateway Protocol)
| Description | BGP is a non-encapsulated, pure routing approach used to peer Kubernetes nodes directly with your physical network infrastructure. |
| How It Works | Each Kubernetes node runs a BGP client provided by the CNI. These clients are configured to peer with your network routers (typically Top-of-Rack switches).
When a pod is scheduled, the CNI advertises a route for that pod’s CIDR block to the peered router (e.g., “to reach The network fabric learns these routes and forwards packets to the correct nodes natively, with no encapsulation or overhead. |
| When to Use | BGP is ideal for on-premises data centers where you control the network fabric and performance is a top priority.
It is especially well-suited for high-throughput, latency-sensitive workloads. |
| Pros |
|
| Cons |
|
Decision Matrix
Use this chart to match your environment, performance, and control needs to the right networking technology:
| Criteria | VXLAN (Overlay) | IP-in-IP (Overlay) | BGP (No Overlay) |
|---|---|---|---|
| Best-Fit Environment | ☁️ Public Cloud (AWS, Azure, GCP) or any network you don’t control. | Hybrid Cloud or specific on-prem networks where UDP is restricted. | 🏢 On-Premises Data Center with a configurable L3 fabric. |
| Performance Overhead | Medium (~50 bytes per packet + CPU). | Low (~20 bytes per packet + CPU). | None. Native performance. |
| Requires Network Integration | No. | No. | Yes. Requires BGP peering. |
| Troubleshooting Visibility | 🔻 Low. Network tools see only node-to-node tunnels. | 🔻 Low. Network tools see only node-to-node tunnels. | ✅ High. Pod-to-pod traffic is fully visible. |
Simplifying Operations Across Network Modes
The choice of networking mode should be driven by the environment, not by the limitations of your CNI. The reality for many organizations is a diverse footprint, often involving both public cloud environments and on-premises data centers. Managing different CNI solutions to match the optimal networking mode for each location can introduce significant operational friction.
A flexible platform that supports all three primary data planes: overlay (VXLAN/IP-in-IP) and underlay (BGP), allows operators to standardize on a single CNI across all environments. You can run VXLAN in your AWS clusters while leveraging BGP for high-performance on-prem clusters, all managed through a consistent set of tools and APIs.
Calico’s support for all of these modes allows you to standardize on a single CNI across different environments. You can run VXLAN in your AWS clusters while using BGP for your high-performance on-prem cluster, all managed with a consistent set of tools and APIs.
This flexibility is powerful because feature parity is maintained. You get the same powerful network policy enforcement, observability, and security capabilities regardless of whether you use an overlay or a routed data plane. The primary benefit is architectural consistency: you can select the optimal data plane for each specific environment without changing your tooling for security and observability.
Final Verdict: Selecting the Right Networking Mode for Your Workload
The choice between BGP, VXLAN, and IP-in-IP is a direct trade-off between performance, control, and environmental constraints. By evaluating your infrastructure and workload requirements, you can select the networking mode that provides the right balance for your cluster:
- For public cloud or hands-off networks, VXLAN is the standard, reliable choice.
- For high-performance on-premises environments where you control the network, BGP delivers superior performance and visibility.
- IP-in-IP serves as a slightly more performant overlay alternative to VXLAN in specific scenarios.
To learn more about Calico’s networking capabilities, explore the networking guide or connect with our engineers and community members on the Calico Slack.
Ready to Simplify Kubernetes Networking?
Choosing the wrong networking mode can create performance bottlenecks, operational complexity, and hidden costs — but it doesn’t have to. Calico supports BGP, VXLAN, and IP-in-IP, so you can optimize for performance and control without compromising on visibility or security.
Whether you’re running in the cloud, on-prem, or both — we’ll show you how to architect it right the first time.
👉 Request a personalized demo to see how Calico can streamline your Kubernetes networking strategy.
