Senior Back-End Network Engineer - AI Infrastructure Operations
About the role
Within Nscale, the Network Operations team is responsible for the performance and reliability of the high-speed interconnect fabrics that underpin our AI and HPC platforms. These networks are critical to distributed training and inference workloads and demand a deep operational focus.
Responsibilities
- Owning the operational health, configuration consistency, and performance tuning of large-scale Infiniband and RoCE fabrics supporting AI and HPC workloads
- Leading the diagnosis and resolution of complex network incidents (P0/P1), spanning firmware, kernel drivers, switch hardware, and application or middleware layers
- Driving blameless postmortems and implementing preventative fixes to improve long-term fabric stability and availability
- Partnering with SREs to define requirements for automation and tooling, and contributing where appropriate to network provisioning, validation, and monitoring systems
- Collaborating with Network Architecture and Engineering teams to validate fabric designs and enforce standards for routing, congestion control, and firmware baselines
- Monitoring fabric utilisation and performance, identifying bottlenecks, and tuning for congestion, microbursts, and predictable latency
- Acting as a subject matter expert for cross-functional teams on high-speed networking, RDMA behaviour, and fabric-level performance characteristics
- Participating in an on-call rotation supporting mission-critical, customer-facing infrastructure
Requirements
5+ years of experience in network engineering, with at least 3 years operating HPC or large-scale AI interconnect networks
Deep, hands-on operational experience with Infiniband and/or modern RoCE deployments
Expert understanding of RDMA concepts, protocols, and troubleshooting techniques
Strong fundamentals in data centre networking, including TCP/IP, BGP, OSPF, and leaf-spine architectures
Proven ability to troubleshoot complex network issues using Linux-based tooling and fabric diagnostics
Proficiency in Python, Go, or shell scripting for automation, data analysis, or configuration management
Experience working in a 24/7 operational environment with a strong focus on reliability and toil reduction
Nice to have
- Experience operating NVIDIA/Mellanox Spectrum switches and ConnectX NICs at scale
- Familiarity with AI or ML training workflows and the impact of network performance on distributed frameworks
- Experience with network observability and telemetry systems such as streaming telemetry, sFlow, Prometheus, or Grafana
- Knowledge of GPU communication libraries such as NVIDIA NCCL
Qualifications
Master's degree in Computer Science, Electrical Engineering, or related field
CISSP certification preferred
Skills
- Linux-based tooling and fabric diagnostics
- RDMA concepts, protocols, and troubleshooting techniques
- Data centre networking fundamentals
- Python, Go, or shell scripting for automation, data analysis, or configuration management
- Network observability and telemetry systems
- GPU communication libraries
Benefits
Highest competitive package (base + equity) with reviews every 12 months
Join the fastest-growing tech startup, your chance to push boundaries, collaborate with brilliant minds, and make your mark on cutting-edge AI
Dynamic progression plan tailored to your ambitions
Competitive benefits package including medical, dental, vision, flexible paid time off, parental leave, and retirement plan participation
Salary Range: $100,000 - $165,000 USD