The National Research Platform (NRP) operates a globally distributed, high-performance computing and networking environment, with an average of 15,000 pods across 450 nodes supporting more than 3,000 scientific project namespaces. With its head node in San Diego, NRP connects research institutions and data centers worldwide via links ranging from 10 to 400 Gbps, serving more than 5,000 users in 70+ locations.
- Non-profit company
- Uses Calico Open Source
NRP is a partnership of more than 50 institutions, led by researchers at UC San Diego, University of Nebraska-Lincoln, and Massachusetts Green High Performance Computing Center and includes contributions by the National Science Foundation, the Department of Energy, the Department of Defense, and many research universities and R&E networking organizations in the US and around the world.
Challenges
Complex Network Visibility and Debugging
NRP needed a way to diagnose connectivity problems across globally distributed storage nodes. Frequent changes to edge network configurations, ACLs, firewalls, and static routes caused blocked ports, forcing manual troubleshooting with tools such as nmap and iperf. This process slowed down root-cause analysis and problem resolution.
Balancing Performance with Granular Security at Scale
Scientific workflows demanded maximum throughput over 100/400 Gbps links and jumbo frames. Traditional host firewalls introduced unacceptable performance penalties, preventing researchers from mounting data and forcing the team to disable them. NRP required a centralized, high-performance network security solution that could enforce fine-grained policies at scale without degrading throughput.
Managing Advanced Heterogeneous Networking
NRP’s workflows relied on advanced capabilities such as Layer 2 paths over WAN using ESnet Sense and AutoGOLE, dual-stack IPv4/IPv6 support, and experiments with multipath BGP. They also needed integration with specialized hardware such as FPGAs and smartNICs for P4 packet processing. Managing this mix of protocols, services, and hardware required a flexible CNI that could support complex, multi-layer orchestration.
Solutions
Enhanced Observability with Calico Telemetry
With the release of Calico Open Source 3.30, the National Research Platform (NRP) intends to adopt new telemetry and observability features to replace a patchwork of existing tools. One key open source observability tool is Calico’s Goldmane API, which enables gRPC-based ingestion of flow logs. The other is Calico Whisker UI, an open source tool for visualizing network flow data, these features provide deep visibility into Kubernetes network traffic.
For NRP, this means they can easily see which network policies were being applied to each flow and identify issues without relying on manual tools like nmap or iperf. Instead of spending hours debugging, engineers can passively detect problems and quickly resolve them, reducing operational overhead while improving network reliability.
Performant Network Policy Enforcement
Calico’s network policies and host endpoints enabled NRP to centralize security across pods and nodes. Using Global Network Policy and a central gateway, NRP could enforce policies cluster-wide without relying on host firewalls. This preserved high throughput while delivering the required security granularity.
Flexible Kubernetes Networking
Calico’s flexibility as a CNI (Container Network Interface) allowed NRP to handle jumbo frames across mixed networks (IPv4-only, dual-stack, and low-MTU nodes) using three Calico daemonsets. Integration with Multus delegated Layer 2 networking while Calico managed Layer 3 routing. This made it possible to establish high-performance “hyperchannels” across ESnet, conduct BGP experiments, and support P4 packet analysis with FPGAs and smartNICs.
Results
- Optimized performance and security: NRP eliminated host firewalls, maintaining throughput while enforcing network policies with Calico.
- Enhanced research capabilities: Flexible orchestration enabled integration of WAN services, IPv4/IPv6 dual-stack, and advanced hardware acceleration.
- Scalability and reliability: Calico delivered a scalable foundation that supports one of the largest distributed Kubernetes clusters for scientific research.
Customer Perspective
“Calico is absolutely central to our operations; it allows us to control all network security and manage host-level firewalls across our entire cluster. This is critical because traditional firewalls severely degrade performance, which we simply cannot afford with our high-speed data flows.”
— Dima Mishin, Full Stack DevOps, Architect and Magician, San Diego Supercomputer Center at NRP
What’s Next
NRP plans to extend its use of Calico telemetry to replicate advanced services such as ESnet High Touch Service, enabling deep flow analysis and troubleshooting without relying on specialized hardware. With Calico as its Kubernetes networking and security backbone, NRP is positioned to support the next generation of large-scale scientific computing.
See how NRP tackles complex network challenges and scales its global scientific research with Calico. Click here to watch our user story.
