Technical Product Manager – AI Product Operations

Responsibilities

• Act as a "player-coach" managing a team of specialists, while treating the AI monitoring and evaluation infrastructure as an internal product with a defined strategic roadmap
• Treat the AI monitoring and evaluation infrastructure as an internal product; define its strategic roadmap, gather stakeholder requirements, and drive its continuous improvement
• Lead the creation of real-time dashboards to interpret evaluation results, building complex monitors to capture agent traces, human-in-the-loop decisions, and user feedback
• Drive root-cause analysis and cross-functional remediation for AI system issues (including documenting post-mortems), while creating scalable runbooks (SOPs) to streamline deployments
• Partner with product managers and engineering leaders to design and implement evaluators directly into the product lifecycle, leading training efforts on how to accurately interpret and use the results
• Collaborate closely with audit methodology specialists, risk, and compliance teams to ensure all AI evaluation flows, metrics, and guardrails actively support firm audit quality standards

Qualifications

• Minimum eight years of recent overall professional experience in Technical Product Management, Product Operations, or Data Science, including a proven track record of building and managing high-performing teams
• Master's degree from an accredited college or university is preferred; minimum of a Bachelor's degree from an accredited or university is required
• Strong technical background with a distinct product management mindset; you don't need to write production code, but you must deeply understand ML/AI concepts to design sophisticated evaluation products
• Extensive knowledge of Generative AI/LLMs, prompt engineering, and GenAI evaluation techniques (for example, automated scoring, human-in-the-loop review, hallucination detection)
• Demonstrated ability to conceptualize scalable measurement frameworks, define critical KPIs, and design structured workflows out of complex, unstructured AI outputs
• Exceptional written and verbal communication skills, with a proven ability to translate highly technical AI performance metrics into clear business narratives for executive stakeholders