Software Engineer Supercomputing

About the role

Operate and automate large GPU clusters including provisioning, imaging, and capacity planning.
Write software that abstracts cluster management and presents a unified interface for training and inference.
Extend scheduling/orchestration (Kubernetes, Slurm, or similar) for topology-aware placement, preemption, quotas, and fair-share multi-tenancy.
Monitor and improve operational metrics of speed, reliability, and error recovery.
Build reliable storage and artifact paths for datasets, checkpoints, and logs with clear retention and lineage.
Partner with researchers to unblock scale runs and advise on parallelism and performance trade-offs.

Responsibilities

• Operate and automate large GPU clusters including provisioning, imaging, and capacity planning.
• Write software that abstracts cluster management and presents a unified interface for training and inference.
• Extend scheduling/orchestration (Kubernetes, Slurm, or similar) for topology-aware placement, preemption, quotas, and fair-share multi-tenancy.
• Maintain and monitor operational metrics of speed, reliability, and error recovery.
• Build reliable storage and artifact paths for datasets, checkpoints, and logs with clear retention and lineage.
• Partner with researchers to unblock scale runs and advise on parallelism and performance trade-offs.

Requirements

• Minimum qualifications: Bachelor’s degree or equivalent experience in computer science, engineering, or similar.
  Proficiency in at least one backend language (we use Python or Rust).
  Experience operating large-scale clusters and container orchestration systems (e.g. Kubernetes or Slurm).
  Comfort operating across the stack and owning projects end-to-end.
• Bias for action with a mindset to take initiative to work across different stacks and different teams where you spot the opportunity to make sure something ships.

Qualifications

• Strong systems background: Linux, networking, and infrastructure-as-code.
• Familiarity with CUDA/NCCL and performance profiling for distributed training/inference.
• Prior work supporting large-scale model training or inference environments.
• Understanding of deep learning frameworks (e.g., PyTorch, TensorFlow, JAX) and their underlying system architectures.
• Track record of working in fast-paced environments balancing care with urgency.

Skills

• Minimum qualifications: Bachelor’s degree or equivalent experience in computer science, engineering, or similar.
• Proficiency in at least one backend language (we use Python or Rust).
• Experience operating large-scale clusters and container orchestration systems (e.g. Kubernetes or Slurm).
• Comfort operating across the stack and owning projects end-to-end.
• Bias for action with a mindset to take initiative to work across different stacks and different teams where you spot the opportunity to make sure something ships.

Benefits

• Generous health, dental, and vision benefits.
• Unlimited PTO.
• Paid parental leave.
• Relocation support as needed.

Pay

Depending on background, skills and experience, the expected annual salary range for this position is $350,000 - $475,000 USD.

Schedule

This role is full-time.

Logistics

• Location: San Francisco, California.
• Compensation: Depending on background, skills and experience, the expected annual salary range for this position is $350,000 - $475,000 USD.
• Visa sponsorship: We sponsor visas. While we can't guarantee success for every candidate or role, if you're the right fit, we're committed to working through the visa process together.
• Benefits: Thinking Machines offers generous health, dental, and vision benefits, unlimited PTO, paid parental leave, and relocation support as needed.

EEO

We do not discriminate on the basis of any protected group status under any applicable law.

Equal Employment Opportunity

Thinking Machines Lab will consider for employment qualified applicants with criminal histories in a manner consistent with the requirements of the California Fair Chance Act, the San Francisco Fair Chance Ordinance, and any other applicable state or local fair chance ordinance or law.