Senior Forward Deployed Engineer (Model Customization and Optimization)

About the role

As a Senior Forward Deployed Engineer (Senior FDE) on the Custom Model and Optimization team, you are the technical lead embedded inside a strategic customer, accountable for building, optimizing, and running the custom generative AI models at the heart of their business. You set the technical direction for model customization and optimization engagements, deciding what to build, how to measure it, and how it reaches production. You combine the technical depth of an AWS senior engineer with the urgency of a startup builder, using your judgment and ownership to deliver measurable business outcomes in quality, latency, and cost. Senior FDEs work on the hardest problems where models meet systems: building domain foundation models, post-training frontier-scale open-weight models with reinforcement learning, and migrating large training and inference workloads onto AWS Trainium. They take on problems where the business problem itself is ambiguous, define the approach, and break it into workstreams others can own. They make long-term architectural trade-offs, balance customer urgency against sustainable design, and are trusted by customer engineering leadership as a technical peer. They raise the bar for the team through design reviews, code reviews, mentorship, and hiring, and they shape how AWS builds. Requires up to 25% travel.

Responsibilities

• Lead the technical engagement: Embed with the customer as the technical lead for their model customization and optimization program, spanning multiple workstreams. Partner with customer engineering and product leadership to frame the business problem, define success metrics and the evaluation bar, and set technical roadmap that moves from proof of value to production. Communicate trade-offs, risks, and results clearly to executives and engineers alike.
• Define the customization strategy: Decide the right path to the customer's quality, latency, and cost targets — prompting, distillation, supervised fine-tuning, preference optimization, reinforcement learning (including RL with verifiable rewards and multi-turn agentic RL), continued pre-training, task-specific adapters, or building a domain foundation model. Own the data strategy, synthetic data generation, and evaluation methodology that make those decisions defensible, and drive them to production for patterns such as coding agents, vertical foundation models, and multi-adapter serving.
• Architect training and inference systems at scale: Design distributed training and RL post-training systems for large dense and mixture-of-experts models using frameworks such as PyTorch FSDP2, Megatron-LM, Miles, and verl, and production inference architectures using engines such as SGLang and vLLM. Architect RL interaction harnesses using frameworks such as NeMo Gym and OpenEnv, defining how agent workflows, tools, environments, and reward functions connect to training and evaluation. Set parallelism, precision, and serving strategies that balance throughput, latency, memory, reliability, and cost per token, and ensure the designs hold up under production load.
• Lead optimization on AWS AI chips: Lead migrations of training and inference workloads to AWS Trainium using the AWS Neuron SDK. Diagnose performance and accuracy-parity gaps against GPU baselines, direct kernel-level optimization where it matters, and work with AWS Neuron and service teams to close capability gaps. Turn benchmark evidence into recommendations that shape customer infrastructure and capacity decisions.
• Lead, mentor, and raise the bar: Provide technical leadership to a team of FDEs and scientists on the engagement, breaking ambiguous problems into well-scoped workstreams and unblocking others. Mentor FDE IIs on model systems design and production ownership. Lead design reviews, provide high-judgment code review, and actively participate in hiring and interviewing.
• Multiply impact across AWS: Identify patterns across engagements and turn them into reusable fine-tuning recipes, evaluation frameworks, benchmarking tools, and reference architectures adopted by other teams. Influence the roadmaps of AWS services, including Amazon Bedrock, Amazon SageMaker AI, and AWS Neuron, with field evidence. Identify and shape opportunities to expand model customization and optimization within your customer and across similar customers.
• Oversee engagement execution: Own the delivery plan, set milestones, track progress against deadlines, and ensure the quality and completeness of all deliverables. Hold yourself and the team accountable for commitments made to the customer, escalate risks early, and adjust scope or resources when timelines are at risk.

Team

The Custom Models & Optimization team, part of the AWS Forward Deployed Engineering team, partners with enterprises, AI-native startups, and foundation model providers to build, customize, and optimize generative AI models at scale. Our work spans domain foundation models, post-training of open-weight models, and moving training and inference onto AWS Trainium. As embedded engineers within the AWS Forward Deployed Engineering motion, we bring deep model and optimization expertise to customers' most strategic AI workloads, and we turn what we learn into tooling and patterns that scale across AWS.

Requirements

• 5+ years of non-internship professional software development experience
• 5+ years of programming with at least one software programming language
• 5+ years of leading design or architecture (design patterns, reliability and scaling) of new and existing systems
• Experience as a mentor, tech lead, or leading an engineering team
• Hands-on experience training, fine-tuning, or optimizing deep learning models
• Experience communicating technical concepts to both technical and non-technical audiences, including senior leadership

Preferred qualifications

• Bachelor's degree in computer science or equivalent
• 2+ years of experience deploying and operating ML models in customer production environments, working with teams that build and scale end-to-end production systems
• Experience defining evaluation strategy and building evaluation frameworks or benchmarks for generative models
• Experience leading optimization of model training or inference for latency, throughput, or cost, using serving engines such as SGLang, vLLM, or TensorRT-LLM, on AWS Trainium (Neuron SDK), on AWS services such as SageMaker AI, SageMaker HyperPod, or EKS, or by writing custom kernels (CUDA, Triton, NKI)
• Experience leading LLM post-training efforts, including SFT, DPO, RLHF, and RL with verifiable rewards or multi-turn agentic RL, using frameworks such as Miles, slime, verl, NeMo-RL, OpenRLHF, SkyRL, or TRL
• Experience leading large-scale distributed training of LLMs, including mixture-of-experts models, on multi-node accelerator clusters, using frameworks such as PyTorch FSDP2, TorchTitan, Megatron-LM/Megatron Core, NeMo, DeepSpeed, or NeuronX Distributed

Pay

• USA, MA, Boston: $168,100 – $227,400 annually
• USA, NY, New York: $184,900 – $250,200 annually
• USA, VA, Arlington: $168,100 – $227,400 annually
• USA, WA, Seattle: $168,100 – $227,400 annually

Compensation includes sign-on payments and restricted stock units (RSUs). Final compensation determined by experience, qualifications, and location. Comprehensive benefits include health insurance (medical, dental, vision, prescription, Basic Life & AD&D, optional Supplemental life, EAP, Mental Health Support, Medical Advice Line, Flexible Spending Accounts, Adoption and Surrogacy Reimbursement), 401(k) matching, paid time off, and parental leave.

Job ID: A10585968 | Company: Amazon Web Services, Inc.