Member of Technical Staff - Data Quality Engineer (Post-training)

About the role

Your mission is to ensure that the data used to train and evaluate our models meets a high bar for quality, reliability, and downstream impact. You will directly shape how our models perform on critical capabilities — agentic tool use, long-horizon reasoning and robust safety alignment. Working with world-class researchers on our post-training teams, you’ll help turn fuzzy notions of “good data” into concrete, measurable standards that scale across large data campaigns.

Responsibilities

• Own upstream data quality for LLM post-training and evaluation by analyzing expert-developed datasets and operationalizing quality standards for reasoning, alignment, and agentic use cases
• Partner closely with research and post-training teams to translate requirements into measurable quality signals, and provide actionable feedback to external data vendors
• Design, validate, and scale automated QA methods, including LLM-as-a-Judge frameworks, to reliably measure data quality across large campaigns
• Build reusable QA pipelines that reliably deliver high-quality data to post-training teams for model training and evaluation
• Monitor and report on data quality over time, driving continuous iteration on quality standards, processes, and acceptance criteria

Qualifications

• Strong engineering fundamentals with experience building data pipelines, QA systems, or evaluation workflows for post-training data and agentic environments
• Detail-oriented with an analytical mindset, able to identify failure modes, inconsistencies, and subtle issues that affect data quality
• Solid understanding of how data quality impacts training (SFT and RL) and evaluation, with the ability to translate quality concerns into concrete signals, decisions, and feedback
• Experience designing and validating automated quality checks, including rule-based systems, statistical methods, or model-assisted approaches such as LLM-as-a-Judge
• Comfortable working autonomously, owning problems end-to-end, and collaborating effectively with researchers, engineers, and operations partners

Skills

• Proficiency in Python and building ML / LLM workflows
• Must be comfortable debugging and writing scalable code
• Experience working with large datasets and automated evaluation or quality-checking systems
• Familiarity with how LLMs work and can describe how models are trained and evaluated
• Excellent communication skills with the ability to clearly articulate complex technical concepts across teams

What we offer

• Top-tier compensation: Salary and equity structured to recognize and retain our talent globally
• Stock options: Everyone who joins and contributes to Reflection's success gets to share in the upside through stock options
• Health & wellness: Comprehensive medical, dental, vision, and life, with an annual wellness allowance
• Meals: Lunch and dinner are provided in the office daily
• Life & family: 22 weeks paid parental leave for all new birthing and non-birthing parents, including adoptive and surrogate journeys
• Vacation days: Unlimited paid time off in the U.S. and 30 days in the U.K.
• Sponsorship support: We sponsor visas to help exceptional talent join our team and support long-term immigration pathways where applicable
• Team building: We have regular off-sites, happy hours, and team celebrations