In the world of Kubernetes, optimizing cluster performance and reliability is paramount, especially when it comes to fundamental operations like DNS lookups. NodeLocal DNSCache is one such solution that helps reduce DNS latency by caching responses locally on each node. While this tool is effective in standard Kubernetes setups, complications arise when integrating it with advanced networking solutions such as eBPF-based dataplanes.
This blog post explores the intricacies of using NodeLocal DNSCache in Kubernetes clusters configured with eBPF dataplanes, particularly in Calico environments. We’ll examine the root cause of compatibility issues and provide actionable solutions to ensure seamless functionality. Whether you’re managing large-scale clusters or experimenting with eBPF for improved performance, understanding this interaction is critical.
Background: NodeLocal DNSCache and eBPF Dataplanes
NodeLocal DNSCache is designed to intercept DNS queries on a node and resolve them locally when possible, improving performance and reducing reliance on external DNS servers. By default, it works smoothly in clusters where kube-proxy handles service traffic. However, the rise of eBPF-based dataplanes, which aim to replace kube-proxy for faster and more efficient packet processing, introduces a fundamental conflict.
eBPF (Extended Berkeley Packet Filter) enables programmable networking at the kernel level, providing significant advantages in terms of speed and flexibility. However, this efficiency comes at a cost: it bypasses the iptables rules that NodeLocal DNSCache relies on to hijack and redirect DNS traffic. The result? DNS queries fail to be cached locally, and the expected performance gains are lost.
The problem is that eBPF dataplanes resolve the service much earlier than the injected iptables rules can intercept it. The service is resolved either at the time when an application issues a connect() call (yes, UDP sockets also have connect() !) or at the tc hook at a device. See the eBPF architecture overview.
Long story short, to make NodeLocal DNSCache work with Calico’s eBPF data plane, we need to tell the kube-proxy replacement that it should not resolve traffic going to a certain service IP. There are two ways to achieve that. First, you can use the BPFExcludeCIDRsFromNAT option and set it to the DNS service IP. You can use this option to exclude multiple IPs from kube-proxy’s address translation. Secondly, and perhaps more conveniently, you can annotate the DNS service with projectcalico.org/natExcludeService=true, which achieves the same.
Beyond Just the NodeLocal DNSCache
With the process described above, traffic is not only excluded from the service resolution, but it is also excluded from bypassing the host network stack. This ensures that the iptables rules injected by the NodeLocal DNSCache are executed and the traffic is steered towards the cache. eBPF dataplanes reduce latency and improve throughput by bypassing the host stack by forwarding from one interface straight to another. Not doing so does not impact performance of NodeLocal DNSCache since the data plane would not forward this traffic anyway as the idea of a node-local cache is that the traffic is destined for a local process.
You can see that to address the NodeLocal DNSCache issue, we opted for a more generic solution, so you can use the same mechanism if you desire to use your own iptables/nftables rules to achieve a specific task and do not want the eBPF data plane to interfere with it, similar to setting chainInsertMode=Append in the Calico’s iptables/nftable dataplanes.
Conclusion
While NodeLocal DNSCache and eBPF dataplanes serve the common goal of optimizing Kubernetes performance, their integration requires careful configuration to avoid conflicts. By understanding the underlying mechanisms—such as how eBPF resolves service IPs earlier than iptables rules—and applying targeted solutions like BPFExcludeCIDRsFromNAT or the natExcludeService annotation, you can achieve seamless operation of DNS caching in an eBPF environment.
For practitioners leveraging the power of eBPF with tools like Calico, these adjustments are essential to maintaining reliable DNS performance while benefiting from the enhanced network capabilities of eBPF. As Kubernetes and eBPF continue to evolve, staying informed about such compatibility nuances will empower you to build resilient and high-performing clusters.
About Calico eBPF and How to Use It
Calico’s eBPF data plane is a powerful alternative to traditional kube-proxy networking. It leverages eBPF to provide a high-performance, low-latency networking layer with features like native service handling, network policies, and traffic optimization. By replacing kube-proxy, Calico eBPF enhances throughput, reduces CPU usage, and improves pod-to-pod communication speed.
To get started with Calico’s eBPF data plane:
- Enable eBPF Mode: Ensure your nodes support eBPF (Linux kernel version 5.3 or later is recommended). Update your Calico configuration by enabling
bpfEnabledin yourcalico-configConfigMap. - Monitor and Debug: Calico provides built-in tools for monitoring eBPF metrics and diagnosing issues. Use
calicoctlto inspect your setup and resolve configuration errors. You can also use Calico Cloud for easy observability.
Optional configuration to achieve seamless operation of DNS caching:
- Configure Service Handling: Use options like
BPFExcludeCIDRsFromNATto fine-tune how traffic is handled, especially for specific services like NodeLocal DNSCache. - Annotate Services for Special Handling: For DNS or other critical services, use the
projectcalico.org/natExcludeService=trueannotation to bypass NAT and ensure seamless integration.
By integrating Calico’s eBPF data plane effectively, you can maximize Kubernetes networking performance while ensuring compatibility with tools like NodeLocal DNSCache. For detailed information, visit Calico eBPF documentation.
For more information and to get started, visit the Calico eBPF documentation.
