Clinical AI Safety Evaluator - Physician / Clinician

Help shape the future of healthcare AI. Highspring is seeking practicing clinicians to evaluate the accuracy, safety, and quality of generative AI healthcare applications for a Fortune 100 client.

Responsibilities

• Evaluate AI-generated healthcare responses for clinical accuracy, safety, and quality.
• Conduct red-team testing by creating realistic clinical scenarios and challenging AI models.
• Identify hallucinations, unsafe recommendations, bias, and other potential clinical risks.
• Provide clear, evidence-based rationale for evaluation scores and identified issues.
• Apply structured evaluation rubrics consistently across a variety of healthcare scenarios.
• Test AI responses from different patient perspectives, including complex or high-risk situations.

Requirements

• MD, MBBS, or equivalent medical degree from an accredited institution.
• Board Certification in Psychiatry, Clinical Psychology, Family Medicine, Primary Care, or Internal Medicine.
• Active, unrestricted U.S. clinical license in good standing.
• Current, hands-on patient care experience.
• Experience using generative AI tools such as ChatGPT, Gemini, Claude, or similar platforms.
• Strong analytical, clinical reasoning, and written communication skills.
• Understanding of LLM limitations, including hallucinations, bias, sycophancy, and unsafe clinical recommendations.
• Ability to create structured prompts and evaluate AI responses objectively.

Preferred Qualifications

• Experience with healthcare AI, AI safety, red teaming, LLM evaluation, Trust & Safety, RLHF, or medical informatics.
• Advanced degree in a related field such as MPH, Medical Informatics, Computer Science, or Behavioral Science.
• Multilingual skills and understanding of diverse cultural approaches to healthcare.

Schedule

• Remote
• Approximately 5 hours per week
• Monday-Friday, flexible hours
• Daily check-in required
• Remote until further notice

This is a unique opportunity to combine your clinical expertise with emerging AI technology and help improve the safety, accuracy, and reliability of healthcare AI.