Senior AI Agent & Evaluations Engineer

Key Responsibilities

• Design, refine, and optimize prompts, tool definitions, routing logic, and decision-making behavior across Vacatia's AI agent ecosystem
• Build and maintain evaluation frameworks, golden datasets, grading systems, and regression testing pipelines
• Develop guardrails and safe-failure mechanisms for customer-facing and financially sensitive workflows
• Monitor production performance, investigate failures, identify edge cases, and improve agent outcomes through data-driven iteration
• Partner with business stakeholders to translate policies and domain expertise into measurable agent behavior
• Collaborate with engineering teams on context requirements, tool contracts, and integration specifications
• Create scalable frameworks and reusable patterns for deploying AI agents across new workflows
• Establish best practices for prompt engineering, evaluation methodologies, observability, and agent operations

Requirements

• Proven experience shipping and owning production AI agents or LLM-powered systems beyond proof-of-concept
• Deep expertise in prompt engineering (system prompts, tool usage, context management, output constraints, agent behavior design)
• Hands-on experience building evaluation frameworks using golden datasets, scoring rubrics, LLM-as-judge methodologies, and regression testing
• Strong familiarity with modern AI development tools (Claude Code, Codex, or similar coding agents)
• Experience with agent observability and evaluation platforms (LangSmith, Langfuse, Arize, Galileo, or comparable)
• Able to distinguish prompt issues from data, tooling, model, or evaluation failures
• Strong written and verbal communication skills
• Demonstrated ownership mindset with passion for reliable, measurable AI systems

Strongly Preferred

• Experience building agents for communication-based workflows (emails, support tickets, chat interactions, transcripts)
• Experience with multiple agent frameworks and understanding of their tradeoffs
• Familiarity with the evolving LLM landscape and model selection strategies
• Experience designing end-to-end evaluation pipelines and agent operations workflows
• Production experience with online evaluation systems and automated scoring of live traffic

Nice to Have

• Experience integrating AI systems with Salesforce, AWS Connect, or customer engagement platforms
• Background in customer-facing industries requiring accuracy, compliance, and communication quality
• Contributions to open-source projects, technical writing, or public thought leadership in AI/prompt engineering/agent development