TL/DR
- A new Grafana dashboard that helps you monitor Calico Typha’s performance and troubleshoot issues.
- Calico eBPF data plane IPv6 is now GA. It supports true IPv6-only clusters as well as dual-stack clusters. 🐝
- Optional Pod startup delay to ensure networking is up in high-churn scenarios.
- Tigera operator now supports multiple IP pools, IP pool modification, affinity for operator pods, priorityclassname, and more!
- Improved policy performance in both eBPF and iptables.
- Calico now ships with a pprof server. Activate the performance server for real-time views of Typha and Felix components and real-time debugging.
🚨 Important changes 🚨
Calico 3.28 now has enabled VXLAN checksum offload by default for environments with the kernel version of 5.8 or above. In the past, offloading was disabled due to kernel bugs.
Please keep in mind, if you are upgrading to 3.28 this change will take effect after node restarts.
If you encounter unexpected performance issues, you can use the following command to revert to the previous method by using the following command:
kubectl patch felixconfiguration default --type="merge" -p='{"spec":{"FeatureDetectOverride":"ChecksumOffloadBroken=true"}}'Please keep in mind that you can report any issues via GitHub tickets or Slack and include a detailed description of the environment (NIC hardware, kernel, distro, workload).
eBPF 🐝
BPFKubeProxyEndpointSlicesEnabled config option is deprecated, has no effect, and will be removed in future releases.
☢️ Breaking changes ☢️
Kubernetes uses a system-generated string called unique identify objects (UIDs). These UIDs are assigned to resources for the lifetime of your cluster and are intended to distinguish between historical occurrences of similar entities.
By upgrading to Calico 3.28 or above the UID of projectcalico.org/v3 resources will change, which should be followed by a restart of any software that depends on these resources. (example: kube-controller-manager)
This change is a result of a bug fix that resolved an issue where resources with OwnerReferences were not properly cleaned up by the Kubernetes garbage collector.
Community shout-outs 🎉
Similar to our previous releases, this one is also packed with many of your contributions! Here are a few new community members who merged their first contribution to the project:
- Thanks to AMacedoP for offering an improvement for a single load balancer IP in a BGP scenario to be announced via RouteReflectors.
- Thanks to amrut-asm for improving the way Calico interacts with cgroupv2 fs for distros such as Talos Linux.
- Thanks to alrs for improving the Calico IPAM binary to correctly output errors if they occur.
- Thanks to chess-knight for catching a problem with YAML files; we can all agree that dealing with YAML is quite fun!!
- Thanks to fm9282 for improving the spoofing check via source IP and annotation for a scenario that uses the etcd datastore.
- Thanks to davhdavh, gyuho, and elsnepal for their contributions to Tigera-operator. The Operator helm chart now supports Kubernetes-services-endpoint creation that can easily automate an eBPF cluster creation via CI/CD pipelines, and Helm charts also support setting affinity for the Tigera-operator pod itself, and adjusting prorityClassName via chart values.
- Thanks to KonstantinVishnivetskii for improving the reinstallation process while using the tigera-operator.
These are just a few people who contributed to this release. As always, the full list of changes and people who participated in it can be found in the release notes.
Calico Observability updates
Typha Grafana dashboard
Typha is a component that sits behind Calico and your Kubernetes API server and plays a caching role in your environment. In 3.28, we are introducing a new Typha observability dashboard to help you identify your datastore performance and pinpoint any issues that might be affecting your environment.
The following image illustrates the new Typha Dashboard:
Calico Felix and Typha now offer real-time debugging
Felix has been packed with the ability to create a CPU and memory snapshot for a while now, an ability that is invaluable in troubleshooting performance issues.
Starting from version 3.28, Calico will include a pprof server that can be activated through configuration. This feature provides a real-time view of the inner workings of its components, such as Typha and Felix. You can easily gather CPU, memory, and other profiles by enabling this server. This information is invaluable for troubleshooting existing issues and gaining a deeper understanding of your environment before scaling your solution.
The following image illustrates the CPU usage of Calico Felix in detail:
Calico eBPF data plane IPv6 is now GA
In our previous releases, we shipped the Calico eBPF data plane with the ability to start IPv6-only clusters. With this release, we now support Dual-Stack setups, allowing you to create clusters that can utilize both IPv6 and IPv4 capabilities using eBPF. In addition, our IPv6 code is now GA.
Click here if you would like to learn more about eBPF IPv6 support.
Pod startup delay
Consistent communication and low latency are essential in a networking environment. However, in a high-churn environment, applications may start before networking is fully established, leading to unstable behavior.
With this release, Calico mitigates this issue by introducing an optional feature to delay pod initialization by setting the linuxPolicySetupTimeoutSeconds config in your “Installation” resource’s CalicoNetwork. This ensures that the host-local data plane and policies are deployed before the workload begins its work, promoting reliable connectivity.
The following is an example that adds a 10-second delay to the pod startup routine. This delay allows the data plane programming to have up to 10 seconds to implement the networking bed before connecting the workload into the system:
kubectl patch installation default --type="merge" -p='{"spec":{"caliconetwork": { "linuxPolicySetupTimeoutSeconds": 10 }}}'Note: data plane programming is robust, but in some corner cases, such as when you have a massive network with tens of millions of routes and policies, it might take a second for your data plane manager to program all the networking routines. The startup delay is a fail-safe mechanism to ensure networking, routes, and policies are in place before hotplugging a workload into the network. The startup delay gives Felix an additional period to get all the cats in a row!
Calico policy engine is even faster now
Introducing policy jump rules!
Policy jumps are not new. In fact, these rules have been around since the early days of iptables, but we haven’t encountered scenarios that require such policies. With this release, Felix can break up “policy jump rules” into new iptables “policy group chains” by the selector. For example, if two endpoints share a common sequence of policies, they will share the same group chain, which reduces the number of rules that need to be programmed.
eBPF policy improvements
Calico eBPF data plane can already program roughly 2k rules per endpoint direction. With 3.28, this number is increased to approximately 15k rules per endpoint direction. This improvement is due to the fact that the eBPF policy compiler now supports splitting policy programs into smaller chunks if they get larger than the Kernel eBPF limit.
These are just a few changes; check out the full release notes on our website.
Join our community
If you’d like to work on the next anticipated Calico Open Source feature, join our contributor’s Slack channel. Feel free to tell us about your vision and the community will help you achieve it.
You’re also welcome to take part in our next virtual community meeting, where we will discuss the future of Calico Open Source and spotlight community members who might have made some of the favorite features you are currently using.
Did you know we have an ambassador program? Join Calico Big Cats today and help us grow our Calico Open Source community.
