Member of Technical Staff - AI Evaluations, Health

Jobs/Member of Technical Staff - AI Evaluations, Health Homesvg]:w-3.5 [&>svg]:h-3.5"> Jobssvg]:w-3.5 [&>svg]:h-3.5"> Companiessvg]:w-3.5 [&>svg]:h-3.5"> Microsoftsvg]:w-3.5 [&>svg]:h-3.5"> Member of Technical Staff - AI Evaluations, Health Member of Technical Staff - AI Evaluations, Health Microsoft Member of Technical Staff - AI Evaluations, Health Microsoft Location New York, NY Type Full-time Posted 8/26/2026 Compensation $119,800 - $304,200 per year Undergraduate with 5+ Years of Experience H-1B FY202696.0% approval503 approvals3,062 LCA5.7% entry-level-64% YoY 👑 Elite sponsor Job Description The Evals Engineer role at MAI Health focuses on ensuring the safety and effectiveness of AI agents deployed in partnership with Mayo Clinic. This position involves evaluating patient-facing AI experiences and working closely with clinicians to develop high-quality evaluation tools. The role requires a blend of technical expertise and strong interpersonal skills to navigate the complexities of healthcare technology. The ideal candidate will thrive in a hybrid work environment, balancing remote work with regular travel to Mayo Clinic and collaboration with various stakeholders. Requirements Bachelor's Degree in Computer Science or related technical field and 6+ years of technical engineering experience with coding in languages such as C, C++, C#, Java, JavaScript, or Python.3+ years of experience leading multi-disciplinary projects, defining requirements, developing project plans, and executing with cross-functional teams.Experience in healthcare, health tech, or other regulated, safety-critical domains.Familiarity with HIPAA, PHI handling, and lifecycle regulatory approaches to clinical AI.Technical depth in software development, data science, and machine learning. Responsibilities Own the end-to-end evaluation strategy for MAI Health's Mayo Clinic project, focusing on clinical accuracy and patient safety.Build layered evaluations that span the full interaction spectrum, including static benchmarks and adaptive simulations.Work embedded with Mayo clinicians to design high-quality tooling and run evaluations for LLM-based tools.Design and write LLM-as-judge prompts for medical content and build automated evaluations that scale clinical review.Evaluate patient-facing AI experiences for factuality, groundedness, safety, and health literacy.Serve as the technical partner ensuring high-quality evaluations are created at Mayo.Partner with data teams to develop scalable pipelines and dashboards tracking evaluation performance.Translate evaluation insights into actionable product improvements and inform launch decisions for clinical features.Monitor trends in model performance and continuously refine evaluation approaches.Coordinate cross-team collaboration to align stakeholders on goals and deliverables. Benefits Employees at Microsoft are often offered comprehensive, “world-class” benefits—including health and mental-wellness programs, competitive pay with bonuses and stock awards, and retirement/savings options. Time-off and flexibility are common, with generous vacation and holidays, parental and caregiver leave, and flexible work schedules, alongside learning support, employee resource groups, product discounts, and matching-gifts/volunteering programs. Specific benefits can vary by region. H-1B filing history Public USCIS petition and DOL LCA counts latest USCIS FY2026, LCA FY2026 Filing entity: Microsoft Corporation As of Jul 17, 2026 Initial approvals 503 FY2026 Approval rate 96.0% FY2026 LCA certified 3,062 FY2026 Entry-level share 5.7% FY2026 Initial approvals YoY 64% Trend LCA certified YoY 83% Trend Initial approvals by fiscal year FY2022 FY2024 FY2026 0 350 700 1050 1400 Approval rate by fiscal year FY2022 FY2023 FY2024 FY2026 0% 25% 50% 75% 100% Continuing vs initial approvals FY2022 FY2024 FY2026 0 2000 4000 6000 8000 Initial Continuing LCA certified positions by quarter 2022-Q1 2023-Q2 2024-Q3 2026-Q2 0 2500 5000 7500 10000 LCA certified positions by fiscal year FY2022 FY2023 FY2024 FY2025 FY2026 0 5000 10000 15000 20000 Based on public USCIS and DOL filings; not a sponsorship guarantee. Is this posting expired or inaccurate? Member of Technical Staff - AI Evaluations, Health Microsoft Something wrong with this listing? At a glance New York, NY Full-time 8/26/2026 $119,800 - $304,200 per year