Senior AI Agent Quality Engineer
About the role
We are seeking a Senior AI Operations Engineer to serve as the quality gatekeeper for the agentic AI systems powering Claritev's next generation of healthcare products. This is a hands-on Agentic QA role for an engineer who is passionate about answering a deceptively hard question: how do you prove an AI agent works? You will design and run the test suites, benchmarks, and evaluation pipelines that validate agent behavior before and after release — catching regressions, hallucinations, broken tool calls, and unsafe actions before they reach production healthcare workflows. You will work closely with our Principal Agentic AI Operations Engineer, AI engineering teams, and Product stakeholders to build golden datasets, automate agent testing in CI/CD, triage and reproduce agent failures, and turn evaluation results into actionable quality insights. This role is an excellent path for a strong QA/test automation, MLOps, or backend engineer looking to specialize in the fast-growing discipline of AI agent quality and evaluation.
Responsibilities
- Design, build, and maintain automated test suites and evaluation pipelines for agentic AI systems, covering single-turn, multi-turn, and end-to-end workflow scenarios.
- Develop and curate golden datasets, test scenarios, and simulation environments that reflect real healthcare workflows and edge cases.
- Execute and automate agent benchmarking, including regression testing across model, prompt, and tool changes, with clear pass/fail quality gates in CI/CD.
- Measure and report agent quality metrics: task completion, tool-call accuracy, grounding/hallucination rates, response quality, latency, cost per task, and safety compliance.
- Implement LLM-as-judge and rubric-based scoring approaches, and validate automated scores against human review.
- Triage, reproduce, and