Applied Scientist, Artificial General Intelligence, Foundational AI

About the role

The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems. This position is based in Bellevue, WA, USA.

Responsibilities

• Collaborate closely with the core scientist team developing Amazon Nova models.
• Lead the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows.
• Design auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that help Nova improve performance on benchmarks.
• Perform expert-level manual audits.
• Conduct meta-audits to evaluate auditor performance.
• Provide targeted coaching to uplift overall quality capabilities.
• Develop and maintain LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment.
• Set up the configuration of data collection workflows and communicate quality feedback to stakeholders.
• Directly impact customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services.
• Support quality solution design.
• Conduct root cause analysis on data quality issues.
• Research new auditing methodologies.
• Find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards.
• Work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.

Requirements

• Master's degree in computer science, mathematics, statistics, machine learning or equivalent quantitative field.
• Experience programming in Java, C++, Python or related language.
• Experience with SQL and an RDBMS (e.g., Oracle) or Data Warehouse.

Qualifications

• Experience implementing algorithms using both toolkits and self-developed code.
• Have publications at top-tier peer-reviewed conferences or journals.

Benefits

• Health insurance (medical, dental, vision, prescription, Basic Life & AD&D insurance and option for Supplemental life plans, EAP, Mental Health Support, Medical Advice Line, Flexible Spending Accounts, Adoption and Surrogacy Reimbursement coverage).
• 401(k) matching.
• Paid time off.
• Parental leave.

Pay

The base salary range for this position is $136,000.00 – $184,000.00 USD annually (Bellevue, WA). The Amazon package includes sign-on payments and restricted stock units (RSUs). Final compensation will be determined based on factors including experience, qualifications, and location.