Senior AI Engineer, Quality

About the role

Fieldguide is building AI agents for the most complex audit and advisory workflows. We're a San Francisco-based Vertical AI company building in a $100B+ market undergoing rapid transformation. Over 50 of the top 100 accounting and consulting firms trust us to power their most mission-critical work.

Responsibilities

• Design and build a unified evaluation platform that serves as the single source of truth for all of our agentic systems and audit workflows
• Build observability systems that surface agent behavior, trace execution, and failure modes in production, and feedback loops that turn production failures into first-class evaluation cases
• Own the evaluation infrastructure stack including integration with LangSmith and LangGraph
• Translate customer problems into concrete agent behaviors and workflows
• Integrate and orchestrate LLMs, tools, retrieval systems, and logic into cohesive, reliable agent experiences
• Rapidly build and deploy LLM-powered features serving production traffic
• Build evaluation frameworks for model outputs and agent behaviors
• Design guardrails and monitoring systems that catch quality regressions before they reach customers
• Define and document evaluation standards, best practices, and processes for the engineering organization
• Partner with product and ML engineers to integrate evaluation requirements into agent development from day one
• Take full ownership of large product areas rather than executing on narrow tasks

Qualifications

• Multiple years of experience shipping production software in complex, real-world systems
• Experience with TypeScript, React, Python, and Postgres
• Built and deployed LLM-powered features serving production traffic
• Implemented evaluation frameworks for model outputs and agent behaviors
• Designed observability or tracing infrastructure for AI/ML systems
• Experience with evaluation platforms (LangSmith, Langfuse, or similar)
• Comfort operating in ambiguity and taking responsibility for outcomes
• Deep empathy for professional-grade, mission-critical software