Helix AI Engineer, Training Infrastructure

Responsibilities

• Design, deploy, and maintain Figure's training clusters
• Architect, optimize, and maintain scalable deep learning frameworks for training on massive robot datasets
• Work together with AI researchers to implement training of new model architectures at a large scale
• Implement distributed training, advanced parallelization strategies, and high-performance data loaders to reduce model development cycles
• Profile, identify, and eliminate training bottlenecks at the hardware and software levels to maximize Model FLOPs Utilization (MFU)
• Implement tooling for data processing, model experimentation, and continuous integration

Requirements

• Strong software engineering fundamentals
• Bachelor's or Master's degree in Computer Science, Robotics, Engineering, or a related field
• Extensive professional experience with Python and PyTorch
• Proven track record of scaling and running large-scale training experiments personally on 800+ GPUs
• Experience managing HPC clusters for deep neural network training
• Minimum of 4 years of professional, full-time experience building reliable backend systems and infrastructure

Bonus Qualifications

• Experience contributing to or maintaining open-source distributed training frameworks (Megatron-LM, DeepSpeed, TorchTitan)
• Experience managing cloud infrastructure (AWS, Azure, GCP)
• Experience with job scheduling / orchestration tools (SLURM, Kubernetes, LSF, etc.)
• Experience with configuration management tools (Ansible, Terraform, Puppet, Chef, etc.)
• Deep understanding of CUDA and hands-on experience writing custom GPU kernels to optimize training