Principal Quantitative User Experience Researcher, AI

About the role

The Principal Quantitative UX Researcher will lead the research strategy for AI-powered product areas, ensuring that we measure the quality, trust, and effectiveness of intelligent systems at scale.

Responsibilities

• Define the quantitative research strategy for AI-powered product areas, establishing how we measure the quality, trust, and effectiveness of intelligent systems at scale.
• Design and execute large-scale surveys, behavioral studies, and log-data analyses—linking attitudinal data from surveys to behavioral signals from product logs to generate integrated insights.
• Build and validate measurement frameworks—including psychometric instruments, experience metrics, and AI evaluation rubrics—that apply statistical rigor to challenges like LLM output quality, human-in-the-loop assessment, and benchmark validation.
• Write and maintain complex SQL queries and Python or R scripts to access, clean, analyze, and build scalable datasets that track AI quality and experience outcomes over time.
• Partner with other Researchers, AI/ML scientists, Data Science, Product, and Design to embed human-centered measurement into AI development workflows and evaluation pipelines.
• Mentor researchers across the team and contribute to thought leadership in AI evaluation, raising the bar for quantitative rigor both internally and in the broader community.

Requirements

• Master's degree or PhD in Human-Computer Interaction (HCI), Computer Science, Statistics, Psychology, or a related field — or equivalent professional experience.
• 8+ years with no advanced degree, or 5–10 years with a suitable advanced degree.
• Expert in quantitative research methods: survey design and psychometrics, experimentation, key driver analysis, and hypothesis testing.
• Expert-level SQL skills and expert-level proficiency in Python or R (or both) for statistical analysis, modeling, and data visualization.
• Deep experience connecting survey-based attitudinal data to behavioral log data to generate integrated insights.
• Demonstrated understanding of AI/ML systems— including how large language models, recommendation systems, or agentic workflows function—and the ability to design research that evaluates them from a human perspective.
• Strong ability to define and operationalize metrics that are scientifically valid and meaningful to product and engineering partners, with the communication skills to make them land.

Preferred Qualifications

• Experience designing or contributing to AI evaluation frameworks, including human evaluation protocols, LLM-as-judge validation, or automated benchmark quality assessment.
• Familiarity with psychometric validity frameworks (construct validity, criterion validity, reliability) and their application to AI output quality measurement.
• Experience working on generative AI, conversational AI, or agentic product experiences.
• Background working within or alongside AI/ML teams, not just researching their outputs.
• Experience with survey platforms (e.g., Qualtrics) and site intercept methodology.
• Contributions to the research or AI community via publications, conference presentations, or open-source work.

Benefits

We provide a full benefits package, including exciting travel perks, generous time-off, parental leave, a flexible work model (with some pretty cool offices), and career development resources, all to fuel our employees' passion for travel and ensure a rewarding career journey.