QA/Test Engineer

About the role

We are seeking experienced QA and Test Engineers to join a pioneering AI initiative focused on building the next generation of evaluation benchmarks for frontier AI models.

Responsibilities

• Design comprehensive test cases that validate evaluation tasks, including complex edge cases and unexpected scenarios
• Review benchmark tasks and reference solutions to identify ambiguity, inconsistencies, missing requirements, and grading gaps
• Debug task environments and Python-based evaluation scripts to ensure reliable execution and accurate results
• Develop repeatable quality assurance processes, validation checklists, and testing frameworks for benchmark creation
• Identify potential shortcuts, exploits, or weaknesses that could compromise evaluation accuracy or benchmark integrity
• Collaborate with AI researchers, engineers, and task authors to improve task quality, reproducibility, and technical rigor

Requirements

• Master's degree, PhD, or equivalent practical experience in a STEM discipline involving software engineering, research, or advanced technical problem solving
• Minimum 1 year of professional experience in Quality Assurance, Test Engineering, Software Engineering, Research Engineering, or a related technical field with strong quality ownership
• Proven experience designing test cases, validating complex software systems, and debugging end-to-end workflows
• Strong proficiency in Python and Git, with the ability to troubleshoot unfamiliar codebases and technical environments
• Excellent analytical thinking, problem-solving skills, and exceptional attention to detail
• Experience documenting bugs, test strategies, and technical findings with clear written communication
• Experience evaluating AI systems, machine learning models, or AI-generated outputs is preferred
• Able to work independently while managing multiple complex tasks with minimal supervision
• Able to commit approximately 35 hours per week on a consistent basis

Preferred Qualifications

• Experience with AI evaluation, benchmark development, or quality assurance for machine learning systems
• Familiarity with large language models, AI agent workflows, or evaluation pipelines
• Experience creating repeatable QA processes for research or engineering projects