Member of Technical Staff — Pretraining Infra (Experienced)

About the role

This role sits at the intersection of research, systems, and GPU-scale execution — building the training stack from 0→1 and scaling it: distributed execution, parallelism, GPU communication, data loading, checkpointing, observability, and debugging.

Direct impact on what models we can train, how fast research can iterate, and how reliably we scale.

Responsibilities

• Own the distributed training stack for omni model pretraining, from 0→1 system design to 1→10 scaling across large GPU clusters.
• Build and operate the core training runtime: job orchestration, distributed execution, checkpointing, recovery, monitoring, and debugging for long-running training jobs.
• Optimize large-scale training performance across parallelism strategy, GPU communication, memory usage, data throughput, MFU, step time, and end-to-end training efficiency.
• Build infrastructure for omni training workloads: high-throughput audio/video/text data loading, temporal alignment, variable sequence handling, multimodal synchronization, and memory-efficient training.
• Evolve the platform as model architectures, training recipes, data mixtures, sequence lengths, hardware constraints, and research directions change.

Requirements

• Hands-on experience running large-scale distributed training jobs across large GPU clusters; experience at hundreds of GPUs minimum, 1,000+ GPUs a strong plus.
• Deep understanding of distributed training mechanics: data/tensor/pipeline/sequence parallelism, gradient communication, collectives, mixed precision, activation checkpointing, optimizer state, memory pressure, and framework-level tradeoffs.
• Strong understanding of GPU communication and performance debugging: NCCL, all-reduce/all-gather/reduce-scatter, communication-computation overlap, topology, synchronization, stragglers, low MFU, OOMs, checkpoint bottlenecks, and data starvation.
• Practical experience with at least one major large-scale training stack such as Megatron, PyTorch FSDP, DeepSpeed, or equivalent internal infrastructure.
• Understanding of omni or multimodal training challenges, especially audio/video/language data, long temporal context, variable sequence lengths, modality-specific bottlenecks, and high-throughput dataloading.
• Strong software engineering fundamentals, curiosity, and adaptability to new model architectures, training frameworks, hardware constraints, and research ideas.

Bonus points

• Prior 0→1 experience building large-scale training infrastructure or deeply modifying core training frameworks, runtimes, checkpointing, or debugging systems.
• Experience training large omni or multimodal models involving audio, video, text, or long-context temporal data.
• Experience with adjacent infrastructure areas such as RL/post-training, data infrastructure, synthetic data generation, evaluation, or serving.
• Publications or substantial open-source contributions in ML systems, distributed systems, HPC, GPU performance, or training infrastructure.