Senior Software Engineer - Cluster Networking

About the role

NVIDIA is a pioneer in accelerated computing, known for inventing the GPU and driving breakthroughs in gaming, computer graphics, high‑performance computing, and artificial intelligence. Our technology powers everything from generative AI to autonomous systems, and we continue to shape the future of computing through innovation and collaboration. Within this mission, the Managed AI Research Superclusters (MARS) team builds and scales the infrastructure, platforms, and tools that enable researchers and engineers to develop the next generation of AI/ML systems. As a senior networking engineer you will lead the network architecture of our GPU superclusters, which run frontier model training across tens of thousands of GPUs on multiple clouds and are scaling toward clusters of ten thousand nodes and beyond.

Responsibilities

• Technical owner of internal and external cluster communication, including the CNI data plane, overlay mesh, and inter‑region/provider node connections.
• Own and evolve the Kubernetes networking architecture for GPU clusters at multi‑thousand‑node scale.
• Design, operate, and scale the overlay network – CNI, mesh, and VPN topologies (Tailscale, WireGuard) and the gateways that connect control and data planes.
• Design, operate, and scale L7 gateways/load balancers/tunnels (Envoy, Cloudflare).
• Identify and eliminate scale ceilings such as packet loss under load, control‑plane saturation, IP address management exhaustion, and failure modes that appear only above a few thousand nodes.
• Build scale‑test environments and validation suites to catch networking regressions before they reach production.
• Diagnose hard, ambiguous problems across the stack—tracing symptoms in Slurm or training jobs back to mark collisions, stale routes, or saturated tunnels.
• Partner with cloud and neocloud providers on network topology, requirements, and capabilities when bringing up new clusters.
• Provide senior technical judgment to a distributed team and depth in the cluster networking domain.

Requirements

• BS/MS in Computer Science, Electrical Engineering, or a related field, or equivalent experience.
• 6+ years of professional experience in systems, network, or infrastructure software engineering.
• Deep command of Kubernetes networking architecture and CNI standards; production experience with Calico is strongly preferred.
• Proficiency designing and maintaining modern mesh and VPN networking topologies (Tailscale, WireGuard, or equivalent).
• Strong Linux networking fundamentals: routing, netfilter/iptables/nftables, packet marking, network namespaces, and interaction with container runtimes.
• Demonstrated ability to debug distributed network problems at scale using packet capture, tracing, and cross‑host correlation.
• Proficiency in Go, Python, C, or a comparable systems language.
• Clear written and verbal communication skills; ability to work effectively with engineers across multiple time zones.

Ways to stand out

• Direct experience architecting and operating massive‑scale Kubernetes topologies across thousands of concurrent nodes.
• Experience with high‑performance fabrics in AI or HPC environments (InfiniBand, RoCE, or RDMA over converged networks).
• Upstream contributions to Calico, Cilium, Tailscale, or Kubernetes networking SIGs.
• Experience operating networking across multiple public clouds and on‑premises environments simultaneously.
• Familiarity with Slurm or other HPC schedulers running on Kubernetes.

Pay

Base salary range: $184,000 – $287,500 USD for Level 4; $224,000 – $356,500 USD for Level 5. Eligibility for equity and benefits.