Principal Product Manager, Agentic Evals

About the role

The Principal Product Manager, Agentic Evals will lead the development of Expedia Group’s AI evaluation framework, ensuring that every AI-powered experience meets high standards of clarity, accuracy, and trust. This role requires a strong background in product management, particularly with AI, ML, data platforms, or evaluation systems.

Responsibilities

• Define Expedia Group’s AI evaluation framework, principles, and adoption strategy
• Build tools and workflows that support reliable, scalable measurement across brands
• Partner with engineering and data science on metric design, instrumentation, and model evaluation
• Create clear, durable standards for accuracy, safety, transparency, and performance
• Publish evaluation insights, scorecards, and recommendations that shape roadmaps
• Influence strategic decisions on AI investments and risk management
• Drive alignment across product, engineering, data science, and platform teams on evaluation strategy, standards, and prioritization
• Create governance models that ensure consistency and responsible deployment at scale
• Shape cross-team operating models and evaluation practices that guide how AI products are built and deployed across the company

Requirements

• Bachelor's degree in a technical, quantitative, or related field, or equivalent practical experience
• 10+ years of product management experience, including significant work with AI, ML, data platforms, or evaluation systems
• Strong technical depth and experience partnering with DS/ML teams
• Proven ability to build frameworks, metrics, or tools that scale across large organizations
• Experience with model evaluation, instrumentation, experimentation, or system reliability
• Strong written and verbal communication that brings clarity to complex technical spaces
• Track record of influencing cross-functional leaders and driving alignment

Qualifications

• Experience building evaluation platforms, ML tooling, or governance frameworks
• Familiarity with LLM behavior, model evaluation methods, or prompt testing
• Understanding of trustworthy AI principles, safety evaluation, and failure-mode analysis

Benefits and Perks

Expedia Group offers a comprehensive benefits package including medical, dental, and vision coverage, paid time off, an Employee Assistance Program, wellness and travel reimbursement, travel discounts, and International Airlines Travel Agent Network (IATAN) membership. Learn more about life at Expedia Group at https://careers.expediagroup.com/life.

Equal Opportunity

Expedia Group is committed to creating an inclusive work environment with a diverse workforce. All qualified applicants will receive consideration for employment without regard to race, color, religion, gender, gender identity or expression, sexual orientation, national origin, genetics, disability, age, veteran status, or any other characteristic protected by law. This employer participates in E-Verify. The employer will provide the Social Security Administration (SSA) and, if necessary, the Department of Homeland Security (DHS) with information from each new employee's I-9 to confirm work authorization.