AI Language Engineer, Alexa for Shopping

About the role

The Conversational Shopping team is looking for a Language Engineer to drive efficiencies and innovation in delivering a seamless, fluent, and engaging experience for AI-assisted shopping. This role is part of the high-performing team behind Amazon’s Generative AI shopping initiatives, such as Rufus AI, Amazon’s Conversational Shopping assistant. The objective is to make it easy for customers worldwide to find and discover the best products, meet personalized needs through product research, comparisons, recommendations, and answering specific product questions.

This role is inherently high-visibility and highly cross-functional, requiring collaboration and influence across global product, design, science, and engineering teams. You will work at the intersection of language and technology, developing automated, scalable solutions for challenges in the Large Language Model (LLM) space. Combining expertise in LLMs, coding, and linguistics (semantics, syntax, pragmatics), you will tackle complex problems in natural language processing (NLP), language understanding, and automated AI evaluations.

Within the Editorial team, you will act as a driving force behind our evaluation-driven product development strategy. You will design processes to produce high-quality editorial data to evaluate and improve the Shopping AI experience across different languages. This includes creating and developing LLM-assisted editorial tools, automated verification scripts, and automated annotations (e.g., LLM-as-a-judge) to support humans-in-the-loop (HITL) work. You will lead data annotation tasks, write intuitive annotation guidelines, and guide the creation of tools tailored to these workflows. Additionally, you will analyze data to track team productivity and measure output quality.

You will collaborate closely with Language Engineers, AI Editors, Product Managers, Applied Scientists, and Software Engineers on initiatives that drive editorial quality, speed, and consistency. By creating and synthesizing quality metrics, you will guide Conversational Shopping teams in meeting internal stakeholder requirements and achieving desired customer outcomes.

Responsibilities

• Produce, process, and manipulate different types of language data, analyze, and provide efficient solutions.
• Automate operations and perform data analysis using coding/scripting languages (e.g., Python).
• Develop LLM-assisted workflows and annotation solutions (e.g., LLM-as-a-judge) to support human-in-the-loop evaluations.
• Design and lead editorial data production/collection by defining scope with internal customer teams.
• Define clear editorial workflows (SOPs) to meet or exceed the quality bar.
• Adopt and design control mechanisms, metrics, and methodologies for editorial and annotation quality.
• Maximize productivity, process efficiency, and quality through streamlined workflows, process standardization, documentation, audits, and investigations.
• Collaborate with editors, applied scientists, engineers, and product managers to deliver the optimal customer experience and define metrics, guidelines, and workflows.
• Establish processes and mechanisms to onboard and train editors on an ongoing basis.
• Handle work prioritization and deliver based on business priorities.
• Adapt workflows in response to customer requests and changing conventions.

A day in the life

• Build an LLM-powered judge that automatically scores thousands of AI responses.
• Write a clear evaluation guideline, then pair with a PM and Scientist to validate it captures what "good" looks like.
• Debug a Python pipeline that processes multilingual annotation data, spot a pattern in the errors, and ship a fix.
• Join a cross-functional sync to align on quality metrics for an upcoming feature launch.
• Analyze evaluation results to surface insights that shape what the product team prioritizes next.

You'll split your time between hands-on technical work (code, prompts, data) and collaborative problem-solving with editors, engineers, and PMs.

About the team

We're the language technology team within Amazon's conversational shopping organization. We build and operate LLM-as-a-Judge systems that automatically measure response quality across every customer experience, develop agentic evaluation architectures that resolve cases single-hop judges can't, and create the tooling and automation that let a small team evaluate millions of AI responses at scale. You'll work alongside language engineers, AI editors, data scientists, and product managers, collaborating cross-functionally with applied scientists and software engineers to ship judges, define quality standards, and turn evaluation data into product decisions for customers around the world.

Requirements

• Master's degree or above in Computational Linguistics.
• Experience owning and executing language data collection projects, including guidelines, labelset, and annotation workflow development.

Preferred Qualifications

• 2+ years of experience in computational linguistics, language data processing, semantics, or philosophy of language.
• Experience in development in the last 3 years, or experience with Machine Learning and Large Language Model fundamentals, including architecture, training/inference lifecycles, and optimization of model execution.

Pay

USA, WA, Seattle - $82,700.00 - $131,600.00 USD annually.

Benefits

• Sign-on payments and restricted stock units (RSUs).
• Comprehensive health insurance (medical, dental, vision, prescription, Basic Life & AD&D insurance, and optional supplemental life plans).
• Employee Assistance Program (EAP) and Mental Health Support.
• Medical Advice Line and Flexible Spending Accounts.
• Adoption and Surrogacy Reimbursement coverage.
• 401(k) matching.
• Paid time off and parental leave.

Learn more about our benefits at amazon.jobs/en/benefits.