Member of Technical Staff - AI Evaluations, Health

About the role

The Evals Engineer role at MAI Health focuses on ensuring the safety and effectiveness of AI agents deployed in partnership with Mayo Clinic. This position involves evaluating patient-facing AI experiences and working closely with clinicians to develop high-quality evaluation tools.

Responsibilities

• Own the end-to-end evaluation strategy for MAI Health's Mayo Clinic project, focusing on clinical accuracy and patient safety.
• Build layered evaluations that span the full interaction spectrum, including static benchmarks and adaptive simulations.
• Work embedded with Mayo clinicians to design high-quality tooling and run evaluations for LLM-based tools.
• Design and write LLM-as-judge prompts for medical content and build automated evaluations that scale clinical review.
• Evaluate patient-facing AI experiences for factuality, groundedness, safety, and health literacy.
• Serve as the technical partner ensuring high-quality evaluations are created at Mayo.
• Partner with data teams to develop scalable pipelines and dashboards tracking evaluation performance.
• Translate evaluation insights into actionable product improvements and inform launch decisions for clinical features.
• Monitor trends in model performance and continuously refine evaluation approaches.
• Coordinate cross-team collaboration to align stakeholders on goals and deliverables.

Requirements

• Bachelor's Degree in Computer Science or related technical field and 6+ years of technical engineering experience with coding in languages such as C, C++, C#, Java, JavaScript, or Python.
• 3+ years of experience leading multi-disciplinary projects, defining requirements, developing project plans, and executing with cross-functional teams.
• Experience in healthcare, health tech, or other regulated, safety-critical domains.
• Familiarity with HIPAA, PHI handling, and lifecycle regulatory approaches to clinical AI.
• Technical depth in software development, data science, and machine learning.

Benefits

Employees at Microsoft are often offered comprehensive, “world-class” benefits—including health and mental-wellness programs, competitive pay with bonuses and stock awards, and retirement/savings options. Time-off and flexibility are common, with generous vacation and holidays, parental and caregiver leave, and flexible work schedules, alongside learning support, employee resource groups, product discounts, and matching-gifts/volunteering programs. Specific benefits can vary by region.