Senior Back-End Network Engineer - AI Infrastructure Operations
About the role
Nscale is the GPU cloud engineered for AI. We provide cost-effective, high-performance infrastructure for AI start-ups and large enterprise customers. Nscale enables AI-focused companies to achieve superior results by reducing the complexity of AI development. Our GPU cloud bolsters technical capabilities and directly supports strategic business outcomes, including cost management, rapid innovation, and environmental responsibility. We thrive on a culture of relentless innovation, ownership, and accountability, where every team member takes pride in their work and drives it with excellence and urgency. As an Nscaler, you’ll build trust through openness and transparency, where everyone is inspired to do their best work. If you join our team, you’ll be contributing to building the technology that powers the future.
Responsibilities
- Owning the operational health, configuration consistency, and performance tuning of large-scale Infiniband and RoCE fabrics supporting AI and HPC workloads
- Leading the diagnosis and resolution of complex network incidents (P0/P1), spanning firmware, kernel drivers, switch hardware, and application or middleware layers
- Driving blameless postmortems and implementing preventative fixes to improve long-term fabric stability and availability
- Partnering with SREs to define requirements for automation and tooling, and contributing where appropriate to network provisioning, validation, and monitoring systems
- Collaborating with Network Architecture and Engineering teams to validate fabric designs and enforce standards for routing, congestion control, and firmware baselines
- Monitoring fabric utilisation and performance, identifying bottlenecks, and tuning for congestion, microbursts, and predictable latency
- Acting as a subject matter expert for cross-functional teams on high-speed networking, RDMA behaviour, and fabric-level performance characteristics
- Participating in an on-call rotation supporting mission-critical, customer-facing infrastructure
Requirements
- 5+ years of experience in network engineering, with at least 3 years operating HPC or large-scale AI interconnect networks
- Deep, hands-on operational experience with Infiniband and/or modern RoCE deployments
- Expert understanding of RDMA concepts, protocols, and troubleshooting techniques
- Strong fundamentals in data centre networking, including TCP/IP, BGP, OSPF, and leaf-spine architectures
- Proven ability to troubleshoot complex network issues using Linux-based tooling and fabric diagnostics
- Proficiency in Python, Go, or shell scripting for automation, data analysis, or configuration management
- Experience working in a 24/7 operational environment with a strong focus on reliability and toil reduction
Qualifications
- Nice to have: Experience operating NVIDIA/Mellanox Spectrum switches and ConnectX NICs at scale
- Familiarity with AI or ML training workflows and the impact of network performance on distributed frameworks
- Experience with network observability and telemetry systems such as streaming telemetry, sFlow, Prometheus, or Grafana
- Knowledge of GPU communication libraries such as NVIDIA NCCL
Skills
- Hands-on operational experience with Infiniband and/or modern RoCE deployments
- Expert understanding of RDMA concepts, protocols, and troubleshooting techniques
- Strong fundamentals in data center networking, including TCP/IP, BGP, OSPF, and leaf-spine architectures
- Proven ability to troubleshoot complex network issues using Linux-based tooling and fabric diagnostics
- Proficiency in Python, Go, or shell scripting for automation, data analysis, or configuration management
Benefits
- Highly competitive package (base + equity) with reviews every 12 months
- Dynamic progression plan tailored to your ambitions
- Opportunity to push boundaries, collaborate with brilliant minds, and make your mark on cutting-edge AI
Pay
Salary Range: $100,000 USD - $165,000 USD
Schedule
Flexible schedule to accommodate 24/7 operational needs
Company Information
Nscale is the GPU cloud engineered for AI. We provide cost-effective, high-performance infrastructure for AI start-ups and large enterprise customers. Nscale enables AI-focused companies to achieve superior results by reducing the complexity of AI development. Our GPU cloud bolsters technical capabilities and directly supports strategic business outcomes, including cost management, rapid innovation, and environmental responsibility. We thrive on a culture of relentless innovation, ownership, and accountability, where every team member takes pride in their work and drives it with excellence and urgency. As an Nscaler, you’ll build trust through openness and transparency, where everyone is inspired to do their best work. If you join our team, you’ll be contributing to building the technology that powers the future.
Equal Opportunities Statement
We strongly encourage applications from people of colour, the LGBTQ+ community, people with disabilities, neurodivergent people, parents, carers, and people from lower socio-economic backgrounds. If there’s anything we can do to accommodate your specific situation, please let us know.