Infrastructure Engineer
Autolab · San Francisco, CA · 2 mo ago
Information TechnologyFull-time
About the role
Build the distributed systems that run thousands of concurrent ML experiments: job orchestration, GPU scheduling, artifact storage, observability.
Responsibilities
- Design and operate the orchestration layer that schedules thousands of concurrent experiments across heterogeneous GPU fleets.
- Build GPU scheduling that keeps clusters near full utilization, from a single lab 3090 to multi-node H100 clusters.
- Own artifact storage for checkpoints, datasets, and logs: versioned, deduplicated, and fast to fetch.
- Build the observability pipeline that ingests training telemetry at high volume without slowing runs down.
- Make experiment launches reproducible: environments, dependencies, and data pinned by default.
- Set the engineering practices, CI, deployment, on-call, that the team grows into.
Requirements
- You have built and operated distributed systems in production: schedulers, queues, or storage.
- Strong Python plus a systems language (Go, Rust, or C++).
- Hands-on experience with cluster tooling such as Kubernetes, Slurm, or Ray.
- You have run GPU or HPC workloads and know where they break.
- Pragmatic by default: ship the simple version, measure, then harden.
- Comfortable owning large areas with little process in an early-stage team.