Senior Deep Learning Engineer – Autonomous Vehicles

About the role

The Senior Deep Learning Systems Engineer will play a pivotal role in advancing NVIDIA's Autonomous Vehicles project. The position involves building and scaling training libraries and infrastructure that support the development of end-to-end autonomous driving models.

Responsibilities

• Crafting, scaling, and hardening deep learning infrastructure libraries and frameworks for training on multi-thousand GPU clusters.
• Improving efficiency throughout the training stack: data loaders, distributed training, scheduling, and performance monitoring.
• Building robust training pipelines and libraries to handle massive video datasets and enable rapid experimentation.
• Collaborating with researchers, model engineers, and internal platform teams to enhance efficiency, minimize stalls, and improve training availability.
• Owning core infrastructure components such as orchestration libraries, distributed training frameworks, and fault-resilient training systems.
• Partnering with leadership to ensure infrastructure scales with growing GPU capacity and dataset size while maintaining developer efficiency and stability.

Requirements

• BS, MS, or PhD in Computer Science, Electrical/Computer Engineering, or a related field, or equivalent experience.
• 12+ years of professional experience building and scaling high-performance distributed systems, ideally in ML, HPC, or large-scale data infrastructure.
• Extensive knowledge in deep learning frameworks (PyTorch is preferred), large scale training (DDP/FSDP, NCCL, tensor/pipeline parallelism), and performance profiling.
• Strong systems background: datacenter networking (RoCE, IB), parallel filesystems (Lustre), storage systems, schedulers (Slurm, Kubernetes, etc.).
• Proficiency in Python and C++, with experience writing production-grade libraries, orchestration layers, and automation tools.
• Ability to work closely with multi-functional teams (ML researchers, infra engineers, product leads) and translate requirements into robust systems.

Qualifications

• Shown experience scaling large GPU training clusters with >1,000 GPUs.
• Contributions to open-source ML systems libraries (e.g., PyTorch, NCCL, FSDP, schedulers, storage clients).
• Expertise in fault resilience and high availability, including elastic training and large-scale observability.
• Led leadership skills as a hands-on technical authority, encouraging others and establishing guidelines for ML systems engineering.
• Familiarity with reinforcement learning (RL) at scale, particularly in the context of simulation-heavy workloads.

Skills

• Experience with deep learning frameworks (PyTorch preferred).
• Knowledge of distributed training techniques (DDP/FSDP, NCCL).
• Understanding of performance profiling and optimization.
• Experience with datacenter networking technologies (RoCE, IB).
• Knowledge of parallel filesystems (Lustre).
• Experience with storage systems and schedulers (Slurm, Kubernetes).
• Proficiency in Python and C++.
• Experience in building and managing large-scale training pipelines and libraries.
• Experience with fault resilience and high availability.
• Experience with reinforcement learning (RL) at scale.

Benefits

• Competitive base salary ranging from $224,000 to $356,500 based on location, experience, and the pay of employees in similar positions.
• Eligibility for equity and benefits.

Pay

Your base salary will be determined based on your location, experience, and the pay of employees in similar positions. The base salary range is $224,000 - $356,500.

Schedule

Not specified.

Benefits

• Not specified.

Cookie/Navigation/Equal Opportunity/Scam Warning Boilerplate

Removed.