Site Reliability Engineer, Model Post Training

About the role

We're looking for a Site Reliability Engineer (SRE) to drive reliability for Model Post Training end-to-end. Post-training spans several distinct workloads — supervised fine-tuning, reinforcement learning, preference optimization, and distillation — each with its own resource profile, failure signature, and tolerance for interruption or delay. You'll set the reliability strategy across all of them, not just react to what breaks.

Responsibilities

• Define and own end-to-end reliability for Model Post Training, from CI/CD flows to production observability and incident response, across SFT, RL, DPO, and distillation workloads.
• Develop Service Level Objectives for distributed training systems, balancing job completion reliability and scheduling latency against development velocity, and tailoring them to each training method's tolerance for delay, restart, and data loss.
• Design and implement monitoring and observability across the full training path — data loading, forward/backward passes, optimizer steps, checkpointing, and sampling — with enough signal to distinguish an infrastructure failure from a training or modeling issue.
• Drive incident response for Model Post Training platform issues, ensuring rapid recovery, thorough incident reviews, and systematic improvements that prevent recurrence.
• Harden multi-tenant isolation and resource scheduling so that LoRA-based workload co-scheduling maximizes utilization without compromising reliability or data separation.
• Build and harden checkpointing and recovery for long-running training jobs, so trainer, optimizer, or sampler failures cost minutes of progress, not hours.
• Mentor other engineers on production ML reliability practices, and collaborate with security teams to address production vulnerabilities.

Requirements

• 7+ years of experience in distributed systems, cloud infrastructure, or site reliability engineering, including direct on-call ownership of a live production service.
• Hands-on experience operating production machine learning systems — training, fine-tuning, or inference infrastructure — and reasoning about how infrastructure failures manifest as training or model-quality issues.
• Proficiency writing software to solve reliability problems, including building tooling and automation, typically in Python and/or a systems language such as Go or Rust.
• Track record of driving production incident response, postmortems, and systematic reliability improvement, including for complex, cross-team, or ambiguous issues.
• Strong communication skills and a track record of setting technical direction and coordinating across engineering and research teams.

Qualifications

• Deep experience operating production cloud services at scale (e.g., public cloud platforms, internal cloud services).
• Working knowledge of post-training methods — SFT, RL (e.g., PPO, GRPO), DPO, or knowledge distillation — and the distinct infrastructure demands each one places on a shared platform.
• Background in distributed training frameworks (e.g., PyTorch FSDP/DDP, Megatron, DeepSpeed) and experience distinguishing infrastructure failures from optimization or data issues in training runs.
• Track record building checkpoint and recovery systems for long-running distributed jobs.
• Expertise in Kubernetes at scale: deploying, operating, debugging, and tuning clusters handling heterogeneous GPU workloads.
• Experience setting reliability standards or SRE practices for an ML platform from the ground up, rather than inheriting an established one.

Pay

Depending on background, skills and experience, the expected annual salary range for this position is $350,000 – $475,000 USD.

Schedule

This role is based in San Francisco, California.