Senior Member of Technical Staff, AI Quality

What You'll Do

• Build capability + regression eval suites for your assigned agents — intake, submissions, placements, renewals, CRM, or voice.
• Curate golden datasets from real failure modes: real transcripts, real underwriter back-and-forth, real call recordings. 20–50 sharp cases per agent, not thousands of synthetic ones.
• Design graders. Deterministic first (string match, state check, tool-call assertions); LLM-as-judge where deterministic fails; human calibration on samples.
• Ship pre-merge eval gates. Every PR touching an agent, prompt, or tool runs the relevant suite in CI. Below threshold, it's blocked.
• Wire production trajectory monitoring. Online evaluators score live trajectories; drift gets caught within hours.
• Turn ops findings into permanent tests. Every flagged failure becomes a regression case; every repeat issue becomes a test that catches it forever.

What We're Looking For

• 3–6 years building software, with hands-on production LLM/agent eval experience — capability + regression suite design, LLM-as-judge graders, golden datasets.
• You can describe a specific regression an eval suite you built caught — and exactly how it would have leaked otherwise.
• You've designed an LLM-as-judge rubric that survived human calibration, and you debug a hallucination by reading transcripts, not aggregate dashboards.
• Familiar with at least one major eval framework; strong written communication (rubric docs, failure-mode taxonomies).
• You write code with AI daily and have real opinions on which agent behaviors actually matter.
• Bonus: open-source eval-framework contributions; red-team/adversarial testing; voice eval (latency, interruption, transcription accuracy); ML eval/observability background.