Senior AI Infrastructure Engineer - Model Training

About the role

Kodiak Robotics, Inc. is seeking a Senior AI Infrastructure Engineer to join their team. The ideal candidate will have hands-on experience with distributed training frameworks and techniques, as well as a deep understanding of GPU performance.

Responsibilities

• Design high-throughput data loading and streaming systems for multimodal sensor data (camera, LiDAR, radar)
• Build and optimize distributed training infrastructure across multi-node GPU clusters
• Maximize utilization of modern accelerators such as NVIDIA B200s through mixed-precision training (BF16/FP8), fused kernels, memory optimization, and communication/computation overlap
• Profile end-to-end training pipelines to find and eliminate bottlenecks across storage, network, CPU preprocessing, and GPU compute
• Develop scalable dataset construction pipelines that convert petabytes of raw driving logs into training-ready, streamable formats
• Partner with ML teams to scale new architectures from prototype to full-cluster training runs efficiently and reliably

Requirements

• BS, MS, or PhD in Computer Science or a related field, and at least 2-3 years of industry experience in ML systems or infrastructure
• Hands-on experience with distributed training frameworks and techniques (PyTorch DDP/FSDP, DeepSpeed, Megatron, NCCL) and a strong grasp of parallelism trade-offs
• Experience building high-performance data pipelines for large-scale training, including streaming dataset formats (WebDataset, MosaicML Streaming/MDS, or similar), sharding, and storage/network-aware loading
• Deep understanding of GPU performance: mixed precision, memory hierarchy, kernel fusion, profiling tools (Nsight, PyTorch Profiler), and interconnects (NVLink, InfiniBand)
• Strong Python skills and proficiency in PyTorch internals; systems-level experience (C++/CUDA/Triton) a plus

Qualifications

• Passion for building the infrastructure that lets AI for the physical world train faster, scale further, and improve continuously

Skills

• Experience with distributed training frameworks and techniques (PyTorch DDP/FSDP, DeepSpeed, Megatron, NCCL)
• Hands-on experience with GPU performance optimization
• Proficiency in Python and PyTorch internals
• Experience with high-performance data pipelines for large-scale training

Benefits

• Competitive compensation package including equity and annual bonuses
• Excellent Medical, Dental, and Vision plans through Kaiser Permanente, Cigna, and MetLife (including a medical plan with infertility benefits)
• MetLife Legal Services, Identity & Fraud Protection, Hospital Indemnity Insurance, Accident Insurance, & Critical Illness Insurance
• Flexible PTO, 10 paid holidays, and generous parental leave policies
• Office perks: dog-friendly, free catered lunch, a fully stocked kitchen, and free EV charging
• Long Term Disability, Short Term Disability, Life Insurance
• Wellbeing Benefits - Headspace through Cigna, Calm through Kaiser, One Medical, Gympass, Spring Health through Cigna, Rula (mental health navigation)
• Fidelity 401(k)
• Commuter, FSA, Dependent Care FSA, HSA
• Variety of incentive programs (referral bonuses, patent bonuses, etc.)

Pay

California Pay Range: $190,000 - $260,000 USD