Senior AI Agent & Evaluations Engineer

Responsibilities

• Design and improve AI agents that directly impact customer experiences and operational efficiency
• Build and maintain evaluation frameworks, golden datasets, grading systems, and regression testing pipelines for agent quality and reliability
• Develop guardrails and safe-failure mechanisms for customer-facing and financially sensitive workflows
• Monitor production performance, investigate failures, identify edge cases, and continuously improve agent outcomes through data-driven iteration
• Collaborate with business stakeholders to translate policies and operational requirements into measurable agent behavior
• Create scalable frameworks and reusable patterns for deploying AI agents across new workflows and use cases
• Establish best practices for prompt engineering, evaluation methodologies, observability, and agent operations

Qualifications

• Proven experience shipping and owning production AI agents or LLM-powered systems beyond proof-of-concept
• Deep expertise in prompt engineering (system prompts, tool usage, context management, output constraints, agent behavior design)
• Hands-on experience building evaluation frameworks with golden datasets, scoring rubrics, LLM-as-judge, regression testing
• Strong familiarity with AI development tools like Claude Code, Codex, or similar coding agents
• Experience with agent observability/evaluation platforms (LangSmith, Langfuse, Arize, Galileo, or comparable)
• Ability to distinguish prompt issues from data, tooling, model, or evaluation failures
• Strong written and verbal communication skills across engineering and business teams
• Demonstrated ownership mindset with passion for building reliable, measurable, continuously improving AI systems