Research Engineer

About the role

We are building infrastructure to create RL training data and evals for frontier AI agents, as well as a marketplace to sell this data to frontier labs. Our platform is used by companies like Fortune 500 and startups.

Responsibilities

• Define and enforce quality standards for training data
• Build tooling and workflows to audit supplier-generated datasets, including sampling strategies, validation pipelines (rule-based and model-assisted), and feedback loops
• Determine if and how human-in-the-loop review workflows can be used to optimize QA
• Partner with data vendors to debug quality issues, provide actionable feedback, and improve their data generation processes
• Continuously integrate QA learnings into infrastructure tools and data vendor portal to reduce anomalies, inconsistencies, and edge cases

Requirements

• Proficiency in Python, Docker, and Linux environments
• Worked with large-scale datasets
• Evidence of rapid learning and adaptability in technical environments (e.g., programming competitions)
• Startup experience in early-stage technology companies with ability to work independently in fast-paced environments
• Familiarity with current AI tools and LLM capabilities
• Strong communication skills for remote collaboration across time zones

Qualifications

• Understanding of common failure modes in training data
• Experience building data validation pipelines and/or human-in-the-loop review systems
• Detail-oriented and able to spot subtle inconsistencies or edge cases in data
• Comfortable designing metrics, experiments, and QA processes, not just executing them

Skills

• Python
• Docker
• Linux
• Large-scale datasets
• AI tools and LLM capabilities
• Rapid learning and adaptability
• Data validation pipelines
• Human-in-the-loop review systems
• Communication skills

Benefits

• Competitive compensation based on experience and location
• 100% covered top-of-the-line medical, dental, and vision from Blue Shield of CA
• Lunch and dinner when you’re in the office
• Company-wide holiday break (Christmas Eve to New Year’s Day) on top of PTO and paid holidays
• Equinox membership, 401k, and commuter benefits
• Unlimited access to tokens for ChatGPT, Claude Code, Cursor, etc.

Pay

$140K - $250K

Schedule

Full-time

Location

On-site in the San Francisco Bay Area