Model Behavior Engineer

About the role

You'll own the quality bar for Notion AI products. You’ll work with product and engineering teams to build systems to define what “good” looks like, measure our progress, and drive changes to deliver reliable and high-quality AI experiences. Your work directly shapes how Notion's AI products behave for millions of users.

Responsibilities

• Context engineering — Design, test, and iterate on system prompts, tool prompts, and context strategies that shape how Notion's AI products behave.
• Understand the nuances of how models respond to different context structures and use that knowledge to drive quality improvements directly.
• Understand & debug — Live in production data: transcripts, logs, user feedback. Reproduce issues, identify root causes, and translate symptoms into actionable problem statements.
• Build evals & Measurement — Design eval strategies, build datasets, run evaluations. Track quality over time. Identify issues before users do. Own the loop: define quality goals, create evals, test and improve.
• Evaluate and launch new models with leading research labs — Evaluate and launch models from OpenAI, Anthropic, Google, and others. Benchmark across dimensions: quality, latency, cost, edge cases.
• Drive quality priorities — Work embedded with eng and product teams to surface the most important issues. Own the quality narrative: severity, frequency, what to fix and why. Be the voice of quality in the room.
• Build tooling & systems — Help manage AI observability and eval platforms (e.g., Braintrust). Build the playbooks and tools that enable all teams at Notion to build AI products.

Requirements

Driver mentality — You treat problems as yours. If something's broken, it's your job to fix it, even if you didn't cause it. You have a bias to action.