Product Content Engineer

Responsibilities

• Define content quality standards and use them to systematically evaluate how AI models are performing across our products and content experiences
• Design golden sets, taxonomies, and guidelines that enable consistent, repeatable content quality assessments
• Build repeatable workflows for collecting, annotating, and analyzing AI outputs so evaluations can run efficiently as models evolve
• Evaluate successive model releases through structured comparison, documenting what improved, what regressed, and what to prioritize next
• Design evaluation frameworks that integrate qualitative and quantitative signals to measure dimensions like user trust, content depth, and topical relevance
• Develop processes to track content quality and model performance over time and flag regressions
• Synthesize evaluation results into structured error patterns and concrete recommendations that engineering and product teams can act on
• Work cross-functionally with engineers, data scientists, product managers, and content strategists to align AI behaviors with real-world user expectations

Requirements

• 5+ years of experience working collaboratively with product, engineering, design, and user research teams
• 1+ years working with generative AI products, AI evaluation, prompt engineering, annotation, and/or content labeling and analysis
• Experience designing and implementing evaluation frameworks, annotation guidelines, or quality rubrics for AI/ML systems
• Demonstrated data analysis skills, with experience exploring data, identifying patterns, and producing actionable insights
• Experience building new products or platform/ecosystem products
• Critical thinking, experience leading data-driven analyses to inform product or content decisions, and experience communicating to executive leadership
• Proven track record of cross-functional collaboration and delivering results in environments with evolving requirements and competing priorities

Qualifications

• Background in content strategy, information quality, or trust and safety
• Experience adhering to and implementing responsible, ethical AI practices (e.g., risk assessment, bias mitigation, quality and accuracy reviews)
• Familiarity with AI evaluation methods such as human eval, model-as-judge, A/B testing, or red-teaming
• Experience building dashboards, scripts, or workflows that codify evaluation metrics
• Demonstrated ongoing AI skill development (e.g., prompt/context engineering, agent orchestration) and staying current with emerging AI technologies
• Experience with Python, SQL, or other tools for data analysis and evaluation automation
• Demonstrated ability to integrate AI tools to optimize/redesign workflows and drive measurable impact (e.g., efficiency gains, quality improvements)

Benefits

$162,000/year to $227,000/year + bonus + equity + benefits