Senior Evaluation Specialist, AI Operations

About the role

At Fetch, we're building AI and automation systems that make our work smarter, faster, and more scalable. The AI Operations team ensures our models and workflows perform with quality, reliability, and measurable impact. As a Senior Automation Specialist, you’ll own evaluation and dataset workstreams that improve AI system performance. You’ll define what “good” looks like, design how we measure it, and build the datasets, evaluations, and automations that turn insights into action.

This role is ideal for someone who is self-directed, biases toward action and rapid iteration, and takes initiative to turn ambiguous problems into structured, scalable solutions. This is a full-time role that can be held from one of our US offices or remotely in the United States.

Role Responsibilities

• Own evaluation & datasets: Define evaluation approaches, design gold datasets (GDS), and ensure coverage of real-world scenarios and edge cases

• Build evaluation systems: Develop manual and automated evals, including LLM-as-judge patterns, to measure model quality and performance

• Translate ambiguity into structure: Turn open-ended questions into clear evaluation frameworks and execution plans

• Build automations: Create automations that improve workflows, including dataset creation, evaluation pipelines, and lightweight operational processes

• Measure and iterate: Define and track performance metrics; refine datasets, evaluations, and workflows based on results

• Drive execution forward: Operate with urgency and ownership; identify next steps, unblock progress, and move work forward with minimal oversight

• Collaborate cross-functionally: Partner with other Automation Specialists, engineering, cross-functional stakeholders, and project leads to ensure high-quality, timely project deliverables

• Improve systems: Identify gaps and implement scalable improvements to evaluation and data workflows

Minimum Requirements

• 3+ years of experience designing or working with evaluation frameworks, datasets, or quality measurement systems

• Experience building or managing datasets (labeling, QA, iteration)

• Ability to independently drive tasks from problem definition to execution

• Hands-on experience with AI tools, LLM workflows, or automation platforms

• Basic scripting or data skills (SQL, Python, etc.)

Preferred Requirements

• Experience with LLM-as-judge or model evaluation techniques

• Familiarity with prompt evaluation or benchmarking approaches

• Experience productionizing evaluation workflows with engineering teams