Software Engineer, DGX Cloud AI Infrastructure

What You’ll Be Doing

• Bring up, validate, and debug large-scale AI clusters, infrastructure, and end-to-end workloads.
• Bring up, tune, and benchmark AI pre-training, post-training, and inference workloads using PyTorch, NeMo / Megatron, TensorRT-LLM, and adjacent NVIDIA AI software stacks.
• Perform root-cause analysis of failures in large distributed environments.
• Contribute to the resilience and failure-attribution tooling that detects, triages, and attributes node, fabric, and workload failures across the cluster.
• Build and maintain repeatable benchmark suites, automation, acceptance criteria, and qualification workflows on new platforms.
• Tune runtime settings, communication parameters, and deployment configurations in close partnership with framework, systems, and platform teams.
• Deliver actionable, data-driven recommendations based on profiling, benchmark results, and cluster characterization.

What We Need To See

• Bachelor’s or Master’s in Computer Science or a related technical field (or equivalent experience).
• 3+ years of experience developing software for AI, HPC, or systems-level applications.
• Hands-on experience with multi-GPU or multi-node workloads and CUDA-aware distributed execution.
• Background with debugging and scaling distributed systems.
• Experience debugging and triaging AI applications across the full stack, from the application level toward the hardware.
• Experience operating workloads in scheduled, containerized cluster environments.
• Excellent analytical, debugging, and communication skills, and a collaborative approach across teams.
• Strong Python and C/C++ programming skills.

Ways To Stand Out From The Crowd

• Hands-on experience with NCCL and CUDA-aware distributed execution.
• Deep familiarity with the RDMA software stack (NCCL, IB verbs, UCX, libfabric) and with InfiniBand / RoCE congestion debugging.
• Experience building acceptance tests, benchmark harnesses, regression gates, or cluster qualification tooling for AI platforms, including MLPerf.
• Experience diagnosing performance jitter.
• Experience building resilience, fault-detection, or failure-attribution systems for datacenter-scale infrastructure.

Pay

Your base salary will be determined based on your location, experience, and the pay of employees in similar positions. The base salary range is $116,000 - $189,750 for Level 2, and $140,000 - $224,250 for Level 3. You will also be eligible for equity and benefits.