Senior HPC Cluster Engineer - AI, ML

About the role

NVIDIA is a pioneer in accelerated computing, known for inventing the GPU and driving breakthroughs in gaming, computer graphics, high-performance computing, and artificial intelligence. Our Managed AI Superclusters (MARS) team builds and scales the infrastructure, platforms, and tools that enable researchers and engineers to develop the next generation of AI/ML systems. As a Senior AI/ML HPC Cluster Engineer, you will provide leadership and strategic guidance on the management of large-scale HPC systems, including the deployment of compute, networking, and storage, while working with a team of skilled engineers.

Responsibilities

• Provide leadership in systems administration and service delivery on our AI/HPC fleet by coordinating system upgrades, responding to incidents, and delivering reliability improvements.
• Collaborate closely with global teams to deliver a world-class user experience in AI and HPC research.
• Own day-to-day operations of production AI/HPC clusters, ensuring system health, user satisfaction, and efficient resource utilization.
• Develop and improve our ecosystem around GPU-accelerated computing, including developing scalable automation solutions.
• Build and maintain heterogeneous AI/ML clusters on-premises and in the cloud.
• Create and cultivate customer and cross-team relationships to meet evolving user needs.
• Support researchers to run their workloads, including performance analysis and optimizations.
• Analyze and optimize cluster efficiency, job fragmentation, and GPU waste to meet internal SLA targets.
• Conduct root cause analysis and suggest corrective actions.
• Proactively find and fix issues before they occur.
• Lead SEV triage and postmortems for reliability incidents affecting users or infrastructure.
• Participate in on-call rotation and incident response for critical production GPU clusters.

Requirements

• Bachelor’s degree in Computer Science, Electrical Engineering, or related field, or equivalent experience.
• Minimum 5 years of experience designing and operating large-scale compute infrastructure.
• Experience with AI/HPC advanced job schedulers, such as Slurm, K8s, PBS, RTDA, BCM, or LSF.
• Proficient in administering CentOS/RHEL and/or Ubuntu Linux distributions.
• Solid understanding of cluster configuration management tools (BCM, Terraform, Ansible, Puppet, Salt, etc.).
• Experience with container technologies (Docker, Singularity, Podman, Shifter, Charliecloud).
• Proficiency in Python programming and bash scripting.
• Applied experience with AI/HPC workflows that use MPI.
• Experience analyzing and tuning performance for a variety of AI/HPC workloads.
• Passion for continual learning and staying ahead of emerging technologies in the HPC and AI/ML infrastructure fields.

Skills

• Background with NVIDIA GPUs, CUDA Programming, NCCL, and MLPerf benchmarking.
• Experience with AI/ML concepts, algorithms, models, and frameworks (PyTorch, TensorFlow).
• Experience with InfiniBand with IPoIB and RDMA.
• Understanding of fast, distributed storage systems such as Lustre and GPFS for AI/HPC workloads.