AI Diagnostics & Observability Engineer

The Role

Every day, our AI agents handle thousands of real patient conversations. Those conversations contain the information needed to improve the system. Today much of that learning process is still manual. Humans review calls, identify issues, investigate failures, and work with engineers to improve agent behavior. Your mission is to build the systems that close that loop.

What You'll Do

• Build AI Quality and Feedback Systems
• Design systems that continuously analyze production conversations and identify quality issues
• Develop automated workflows that transform human feedback into actionable improvements
• Create evaluation pipelines that measure agent performance across key dimensions
• Build systems that detect recurring failure patterns and cluster similar issues
• Create mechanisms for routing issues to the appropriate AI, engineering, or operational workflows
• Improve Agent Reliability
• Investigate production failures and identify root causes across transcription, reasoning, retrieval, and orchestration systems
• Build tooling that helps engineers quickly understand why an agent behaved a certain way
• Establish quality metrics and reliability standards for production agents
• Partner with AI engineers to measure the impact of improvements and validate fixes
• Automate Learning Loops
• Build systems that reduce manual effort required to improve agents
• Create workflows that propose fixes, recommendations, or improvements based on production behavior
• Explore approaches for automatically incorporating feedback into agent development workflows
• Help define the long-term architecture for self-improving AI systems
• Partner Across Teams
• Work closely with AI engineers developing production agents
• Collaborate with operations teams who review calls and provide quality feedback
• Help define processes for identifying, prioritizing, and resolving agent issues
• Mentor junior engineers working in the AI quality and evaluation space

We're Looking For

We're looking for an engineer who has built and operated production AI systems and is excited about improving their quality over time. You likely have experience in one or more of the following:

• Building AI evaluation platforms
• Developing feedback systems for ML and LLM applications
• Operating production AI agents or conversational AI systems
• Creating observability, reliability, or quality tooling for AI products
• Designing automated workflows that improve model or agent performance
• Strong backend engineering skills and systems thinking
• Experience working with ambiguous problems and defining solutions from first principles
• Excellent communication and collaboration skills

Bonus Points For

• Experience at AI-native companies building production AI applications
• Experience with evaluation frameworks and model quality measurement
• Experience with voice AI systems
• Experience designing human-in-the-loop workflows
• Experience building internal platforms used by engineering teams

Why This Role Matters

AI agents are only as good as their ability to learn from real-world usage. As our systems scale, manually identifying and fixing issues does not scale with them. The systems you build will directly influence how quickly our agents improve, how reliably they serve patients, and how effectively healthcare organizations can trust AI in critical workflows. This is an opportunity to work on one of the most important challenges in production AI: turning real-world interactions into continuous learning and improvement.