Applied Data Scientist, Health AI Evaluation & Datasets

About the role

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for an Applied Data Scientist, Health AI Evaluation & Datasets based in the United States. This role sits at the intersection of clinical expertise, data science, and cutting-edge AI evaluation, focusing on building trustworthy datasets for healthcare generative AI systems.

Responsibilities

• Design, define, and operationalize high-quality healthcare datasets used for training, fine-tuning, and evaluating generative and multimodal AI systems across clinical and life sciences domains.
• Translate complex healthcare objectives such as diagnosis support, medical summarization, RAG-based retrieval, and patient communication into structured dataset specifications, labeling schemas, and evaluation rubrics.
• Develop clinically grounded evaluation frameworks that measure model performance across safety, accuracy, faithfulness, guideline adherence, and workflow relevance.
• Build multimodal datasets spanning clinical notes, imaging, structured EHR data, claims, literature, and patient-provider communications while ensuring clinical validity and statistical rigor.
• Define sampling strategies, annotation guidelines, SME review workflows, inter-annotator agreement standards, and quality assurance processes for healthcare datasets.
• Ensure data governance, compliance, and auditability across PHI-sensitive workflows, including de-identification, provenance tracking, and version control.
• Evaluate model behavior beyond accuracy, including calibration, hallucination risk, safety-critical failure modes, and fairness across patient populations.
• Collaborate with engineers and research scientists to integrate datasets into evaluation pipelines, including LLM-as-judge systems, benchmarking frameworks, and model comparison workflows.
• Contribute to reusable internal assets such as taxonomies, rubrics, evaluation templates, and gold-standard datasets.

Requirements

• 5+ years of experience in data science, including at least 2+ years working directly with healthcare, biomedical, clinical, payer, pharma, or life sciences data.
• Strong understanding of healthcare data systems and standards, including EHR structures and clinical coding systems such as ICD-10, CPT, SNOMED CT, LOINC, and RxNorm.
• Proven experience designing and building ML datasets, including annotation guidelines, sampling strategies, QA processes, and dataset validation frameworks.
• Hands-on experience with LLM-based AI workflows, including evaluation design, prompt engineering, retrieval-augmented systems, and rubric-based assessment methods.
• Strong programming skills in Python and SQL, with familiarity in tools such as pandas, scikit-learn, statsmodels, and modern ML/LLM ecosystems.
• Solid statistical background covering sampling methods, bias and fairness analysis, inter-annotator agreement, hypothesis testing, and uncertainty estimation.
• Deep understanding of healthcare privacy and compliance frameworks such as HIPAA, de-identification methodologies, and secure handling of sensitive data.
• Able to collaborate effectively with clinicians, engineers, researchers, and business stakeholders in complex, cross-functional environments.
• Strong communication skills with the ability to translate clinical and technical complexity into actionable insights and structured evaluation frameworks.
• Advanced degree in a relevant field (biostatistics, epidemiology, health informatics, data science, computer science, or related discipline) or equivalent experience; clinical backgrounds (MD, RN, PharmD, MPH, PhD) are a strong advantage.

Benefits

• Competitive annual salary ranging from $150,000 to $175,000 USD, based on experience and qualifications.
• Fully remote work opportunity across the United States.
• Opportunity to work on high-impact healthcare AI systems used across clinical, payer, and life sciences domains.
• Collaborative environment working alongside leading experts in AI, clinical science, and data engineering.
• Exposure to cutting-edge generative AI, evaluation frameworks, and multimodal dataset development.
• Support for continuous learning in advanced ML, healthcare data systems, and AI evaluation methodologies.