Medical Expert AI Evaluation Specialist

Key Responsibilities

• Develop Clinical Scenarios
• Create realistic, specialty-specific clinical cases that reflect real-world medical practice
• Design prompts covering areas such as: Diagnostic reasoning, Treatment planning, Risk stratification, Differential diagnosis, Guideline-based clinical decision-making
• Create Reference-Quality Solutions
• Produce high-quality clinical responses that reflect attending physician-level reasoning
• Ensure recommendations align with current evidence-based practices and accepted clinical guidelines
• Evaluate AI Performance
• Assess AI-generated medical responses using structured evaluation frameworks and scoring rubrics
• Identify strengths, weaknesses, omissions, and reasoning errors
• Provide detailed written feedback to improve model performance and clinical accuracy
• Collaborate with Clinical Experts
• Participate in onboarding sessions and specialty calibration meetings
• Work with multidisciplinary research teams to maintain consistency and quality across medical evaluations

Required Qualifications

• Attending Physicians: Board-certified physician with a current, active, and unrestricted medical license
• Resident Physicians: Currently in the final year of an accredited residency program, recent graduates must be board-eligible
• Fellows: Board-certified or board-eligible in their primary specialty, current, active, and unrestricted medical license
• Physicians from a wide range of specialties are encouraged to apply, including but not limited to: Family Medicine, Pediatrics, Anesthesiology, General Surgery, ENT (Otolaryngology), Physical Medicine & Rehabilitation (PM&R), Occupational Medicine, Other medical and surgical specialties

Preferred Qualifications

• Strong understanding of evidence-based medicine and clinical practice guidelines
• Excellent written communication skills
• Experience with medical education, clinical research, peer review, or healthcare quality initiatives
• Interest in artificial intelligence and healthcare innovation