Senior Back-End Network Engineer - AI Infrastructure Operations

About the role

Within Nscale, the Network Operations team is responsible for the performance and reliability of the high-speed interconnect fabrics that underpin our AI and HPC platforms. These networks are critical to distributed training and inference workloads and demand a deep operational focus.

Responsibilities

• Owning the operational health, configuration consistency, and performance tuning of large-scale Infiniband and RoCE fabrics supporting AI and HPC workloads
• Leading the diagnosis and resolution of complex network incidents (P0/P1), spanning firmware, kernel drivers, switch hardware, and application or middleware layers
• Driving blameless postmortems and implementing preventative fixes to improve long-term fabric stability and availability
• Partnering with SREs to define requirements for automation and tooling, and contributing where appropriate to network provisioning, validation, and monitoring systems
• Collaborating with Network Architecture and Engineering teams to validate fabric designs and enforce standards for routing, congestion control, and firmware baselines
• Monitoring fabric utilisation and performance, identifying bottlenecks, and tuning for congestion, microbursts, and predictable latency
• Acting as a subject matter expert for cross-functional teams on high-speed networking, RDMA behaviour, and fabric-level performance characteristics
• Participating in an on-call rotation supporting mission-critical, customer-facing infrastructure

Requirements

• 5+ years of experience in network engineering, with at least 3 years operating HPC or large-scale AI interconnect networks
• Deep, hands-on operational experience with Infiniband and/or modern RoCE deployments
• Expert understanding of RDMA concepts, protocols, and troubleshooting techniques
• Strong fundamentals in data centre networking, including TCP/IP, BGP, OSPF, and leaf-spine architectures
• Proven ability to troubleshoot complex network issues using Linux-based tooling and fabric diagnostics
• Proficiency in Python, Go, or shell scripting for automation, data analysis, or configuration management
• Experience working in a 24/7 operational environment with a strong focus on reliability and toil reduction

Nice to have

• Experience operating NVIDIA/Mellanox Spectrum switches and ConnectX NICs at scale
• Familiarity with AI or ML training workflows and the impact of network performance on distributed frameworks
• Experience with network observability and telemetry systems such as streaming telemetry, sFlow, Prometheus, or Grafana
• Knowledge of GPU communication libraries such as NVIDIA NCCL

Qualifications

• BS/MS in Computer Science, Electrical Engineering, or related field
• CISSP or equivalent security certification preferred

Skills

• Linux-based tooling and fabric diagnostics
• Python, Go, or shell scripting for automation, data analysis, or configuration management
• Network observability and telemetry systems such as streaming telemetry, sFlow, Prometheus, or Grafana
• GPU communication libraries such as NVIDIA NCCL

Benefits

• Highly competitive package (base + equity) with reviews every 12 months
• Join the fastest-growing tech startup, your chance to push boundaries, collaborate with brilliant minds, and make your mark on cutting-edge AI
• Dynamic progression plan tailored to your ambitions

Pay

$100,000 - $165,000 USD

Schedule

Full-time