Staff ML Engineer – AWS Trainium & SageMaker

Robots & Pencils is an AWS Partner building production AI systems for enterprise clients who need real engineering, not proofs of concept that never ship. We work forward‑deployed, embedded directly with client teams, solving problems that are too new or too specialized for a typical vendor relationship to handle.

The Role

We are looking for an engineer who can operate and train models on Amazon SageMaker running on AWS Trainium, AWS's custom silicon built specifically for large‑scale model training. This role goes beyond calling an API; you will work up the stack from the hardware and compiler level through PyTorch training code to a production SageMaker pipeline.

What You’ll Do

• Train and operate models on Amazon SageMaker with AWS Trainium as the underlying compute.
• Write and optimize PyTorch training code with a real understanding of how it compiles and executes on Trainium (NeuronCore architecture, compiler behavior, memory and throughput trade‑offs).
• Diagnose training run issues that arise specifically because of the hardware, distinguishing data or code problems from compiler or device‑level issues.
• Translate a request for “a Trainium job” into an actual working, cost‑aware training pipeline, end‑to‑end.
• Tune distributed training runs for throughput and cost on SageMaker’s training infrastructure.
• Work directly with client and internal engineering teams to scope and deliver real production training workloads, not experiments that stay in a notebook.

What You’ll Bring

• Strong, hands‑on PyTorch experience, ideally including distributed or multi‑device training.
• Production experience with Amazon SageMaker for training and/or inference.
• Comfort working close to the hardware layer: understanding device‑specific compilation and debugging accelerator‑related issues.
• Experience with AWS Trainium or Inferentia (Neuron SDK) is a strong plus; if you don’t have it yet but have deep PyTorch expertise and a track record of picking up new hardware targets quickly, we want to talk to you.
• Solid Python fundamentals and comfort operating in a client‑facing, production engineering environment.