User Researcher, AI Evaluations

About the role

We’re seeking an experienced UX Researcher to define and scale how we evaluate Notion’s AI-powered experiences. This role sits at the intersection of research craft and evaluation operations.

Responsibilities

• Define what “good” looks like (frameworks & rubrics): Establish clear, reusable evaluation criteria that reflect real user expectations—helpfulness, trust, tone, control, and transparency. Translate qualitative insight into scoring guidance that can be applied consistently across teams and over time.
• Run recurring evals (longitudinal & feature-specific): Run recurring longitudinal and feature-specific surveys and studies to measure experience quality over time against defined rubrics. Lead qualitative studies, side-by-side comparisons, and human-in-the-loop evaluation efforts to deepen understanding of where experiences break down and how they can improve.
• Anchor evaluation in real workflows (context > isolated feedback): Ensure evals reflect jobs-to-be-done, user intent, and the full interaction journey (goal setting, delegation, review, iteration), not just decontextualized thumbs up/down. Help teams understand who is evaluating, what they’re trying to do, and why outputs succeed or fail.
• Identify failure modes & recovery behavior (guardrails): Uncover breakdowns, regressions, and edge cases across the system—from model behavior to UI and integrations—and study how people notice issues, correct them, and continue their work. Turn these insights into actionable guidance for guardrails, fixes, and prioritization.
• Operationalize evaluation with partners (process & tooling): Collaborate closely with Product, Design, Engineering, and Data Science to align on target use cases and build scalable evaluation loops (human-in-the-loop review, longitudinal studies, and calibration of automated/LLM-judge approaches against human judgment).

Requirements

• Ability to operationalize insight into measurement: Comfortable turning “soft” user expectations (trust, tone, usefulness, clarity) into concrete rubrics, scoring guidelines, and observable metrics.
• AI fluency and systems thinking: Curious and hands-on with AI products, reasoning about how model behavior, uncertainty, and system constraints shape user experience. Experience evaluating AI-enabled products (LLMs, agents, generative UI/workflow automation) and working with Data Science/ML partners on measurement strategy and evaluation tooling.
• Clear communication and impact orientation: Align diverse partners around shared definitions of quality and create artifacts that enable teams to act consistently. Tailor storytelling to different audiences, connect research to business outcomes, and drive follow-through so insights translate into product change.
• Strong UX research craft (quant + qual): Choose the right methods for the question—interviews, benchmarking, surveys, experiments—and synthesize into actionable guidance. Prioritize ruthlessly, work through ambiguity, and balance scrappy iteration with deep dives when needed.
• Pragmatism in fast-moving environments: Prioritize ruthlessly, work through ambiguity, and balance scrappy iteration with deep dives when needed.
• Experience: 5+ years doing UX research in industry.

Qualifications

• Nice To Haves: Familiarity with LLM-as-judge methods, prompt design for evaluators, or “golden dataset” creation. Experience using AI research tooling for rapid synthesis and communication (e.g., Dovetail, Listen Labs, Maze, Outset, etc.), as well as AI observability tooling like Braintrust. Experience using data querying languages (e.g., SQL), scripting languages (e.g., Python), or statistical/mathematical software (e.g., R, SAS, Matlab, etc.). Master’s or PhD in HCI, Psychology, Behavioral Science, Anthropology, Sociology, or a related field. Familiar with the work of computing heroes like Douglas Engelbart, Alan Kay, Bret Victor, etc. — and understand why we're big fans.