Applied Scientist, Foundational AI
Description
The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems.
Key job responsibilities
Collaborate closely with core scientist team developing Amazon Nova models.
Lead the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows.
Design auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that help Nova improve performances on benchmarks.
Perform expert-level manual audits, conduct meta-audits to evaluate auditor performance, and provide targeted coaching to uplift overall quality capabilities.
Develop and maintain LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment.
Set up the configuration of data collection workflows and communicate quality feedback to stakeholders.
Enhance customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services.
A day in the life
Support quality solution design.
Conduct root cause analysis on data quality issues.
Research new auditing methodologies.
Optimize data quality while setting examples for the team on quality assurance best practices and standards.
Work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.
Basic Qualifications
Master's degree in computer science, mathematics, statistics, machine learning or equivalent quantitative field.
Experience programming in Java, C++, Python or related language.
Experience with SQL and an RDBMS (e.g., Oracle) or Data Warehouse.
Preferred Qualifications
Experience implementing algorithms using both toolkits and self-developed code.
Publications at top-tier peer-reviewed conferences or journals.
Company
Amazon is an equal opportunity employer and does not discriminate on the basis of protected veteran status, disability, or other legally protected status. Our inclusive culture empowers Amazonians to deliver the best results for our customers. If you have a disability and need a workplace accommodation or adjustment during the application and hiring process, including support for the interview or onboarding process, please visit here for more information.
Application and Hiring Process
If the country/region you’re applying in isn’t listed, please contact your Recruiting Partner.
Salary Range
The base salary range for this position is listed below. Your Amazon package will include sign-on payments and restricted stock units (RSUs). Final compensation will be determined based on factors including experience, qualifications, and location.
USA, WA, BELLEVUE - 136,000.00 - 184,000.00 USD annually
Benefits
Learn more about our benefits at here.