AI Evaluation Engineer

Position Summary

As an AI Evaluation Engineer at Judi Health, you will build the testing frameworks, metrics, and tooling used to assess the safety, reliability, and accuracy of AI models and autonomous agents in production. This role bridges the gap between model development and real-world usage by translating ambiguous product goals into measurable quality targets. We're looking for someone to lead evaluation end-to-end — from unit and integration testing to offline, online, and statistical evaluations of probabilistic systems.

What we need

We're looking for someone who can design and operate robust evaluation frameworks, partner with scientists and engineers, and ensure we can confidently answer questions like: "Did this change improve or degrade quality, safety, or user outcomes?"

Position Responsibilities

• Data Engineering
• Build and maintain ETL pipelines for heterogeneous data sources (traces, logs, transcripts, user feedback)
• Implement complex data stitching and session reconstruction logic
• Manage dataset versioning, provenance, and lifecycle
• Platform & Observability
• Develop dashboards and monitoring tools for AI quality metrics
• Integrate evaluations into CI/CD pipelines for scheduled and gated runs
• Implement alerting on quality and safety signals, not just infrastructure health
• AI / ML Evaluation Tooling
• Apply and extend LLM-as-judge evaluation patterns
• Design metrics and scoring approaches suitable for stochastic, non-deterministic systems
• Use tools like LangSmith to track runs, traces, experiments, and evaluation results
• Collaboration
• Partner closely with data science, engineering, and product teams
• Translate between research goals, product intent, and engineering constraints
• Advocate for strong developer experience and usability in the tools you build
• Responsible for adherence to the Capital Rx Code of Conduct including the reporting of non-compliance

Required Qualifications

• 4+ years of experience in data engineering, ML engineering, or software engineering
• Bachelor’s or Master’s degree in Computer Science, Machine Learning, or a related quantitative field
• Strong proficiency in Python
• Experience building and maintaining production data pipelines
• Strong SQL skills
• Experience working with at least one cloud platform (AWS preferred)

Nice-to-Haves

• Prior work on LLM or agent evaluation infrastructure
• Familiarity with designing metrics for safety, reliability, or quality in AI systems
• Experience with voice or call-center data (audio, transcripts, sentiment)
• Experience with browser automation tools (e.g., Playwright) for end-to-end evals
• Deep SQL expertise