Post Doctoral Researcher, Evaluation & Standards in Clinical Operations

Job Description

About the Role

• We are deploying generative AI agents across clinical operations and global development workflows: clinical data review, anomaly detection in trial data, protocol Q&A for site teams, Informed Consent Form generation and translation, and protocol element document assembly.
• Each system needs rigorous evaluation. In specialized clinical contexts, "good" is not a single number. It depends on the user, the regulatory context, the therapeutic area, and the cost profile of errors.

Key Responsibilities

• Develop evaluation frameworks and rubrics for AI agents operating in clinical operations and global development contexts.
• Define quality dimensions for specialized outputs, e.g. clinical data queries, anomaly detection, protocol Q&A responses, ICF content, regulatory documents, that reflect how clinical-ops users actually judge quality.
• Design and run human evaluation protocols with clinical, regulatory, and site-facing subject matter experts.
• Build benchmark datasets and evaluation pipelines that the broader GenAI organization uses across AI applications.
• Partner with our AI engineering teams to translate evaluation findings into measurable improvement.
• Contribute to the methodology of how a regulated pharma organization evaluates AI in clinical work.

What We're Looking For

• Domain expertise: Strong understanding of clinical operations and global development workflows, familiarity with how clinical-ops specialists, monitors, regulatory authors, and site teams actually do their work and assess quality.
• PhD or equivalent research experience: in a clinical, biomedical, regulatory, biostatistics, or closely related field.
• Enthusiasm for AI agents: Genuine interest in working hands-on with generative AI and agentic systems, curiosity about how LLMs behave in specialized domains and where they fail, comfort prototyping with modern AI tooling (LLM APIs, vector databases, evaluation frameworks).
• Aptitude for evaluation systems: Demonstrated ability to develop frameworks, rubrics, and structured assessment protocols, experience designing evaluations with multiple quality dimensions, expert raters, or non-trivial inter-rater reliability considerations, strong analytical and writing skills; comfort with ambiguity and unsolved problems.

Practical Skills

• Python and the modern ML/data stack: Comfort working across disciplines, e.g. clinical operations, regulatory affairs, AI engineering, data science.

Practical Details

• Duraton: 2-year postdoctoral appointment
• Reports to: [TBD]
• Works closely with: Generative AI Teams, Scientific Fellow, clinical operations and regulatory subject matter experts across J&J global development

Required Skills

• Domain expertise: Strong understanding of clinical operations and global development workflows, familiarity with how clinical-ops specialists, monitors, regulatory authors, and site teams actually do their work and assess quality.
• PhD or equivalent research experience: in a clinical, biomedical, regulatory, biostatistics, or closely related field.
• Enthusiasm for AI agents: Genuine interest in working hands-on with generative AI and agentic systems, curiosity about how LLMs behave in specialized domains and where they fail, comfort prototyping with modern AI tooling (LLM APIs, vector databases, evaluation frameworks).
• Aptitude for evaluation systems: Demonstrated ability to develop frameworks, rubrics, and structured assessment protocols, experience designing evaluations with multiple quality dimensions, expert raters, or non-trivial inter-rater reliability considerations, strong analytical and writing skills; comfort with ambiguity and unsolved problems.

Preferred Skills

• Python and the modern ML/data stack: Comfort working across disciplines, e.g. clinical operations, regulatory affairs, AI engineering, data science.

Benefits

• Annual bonus with set target (% of pay) depending on pay grade / location, where the actual amount is based on the employees’ and companies’ performance of the previous calendar year, or sales commissions.
• Vacation days, parental leave for a minimum of 12 weeks, bereavement leave, caregiver leave, volunteer leave, well-being reimbursement, programs for financial, physical and mental health.
• Service anniversary and recognition awards, and subject to the terms of their respective plans, employees - and in some location’s eligible dependents - can participate in several insurance plans.