AI Enablement & Governance- AI Quality & Evaluation Lead

Responsibilities

• Partner with AI Engineers and Data Scientists during design to set quality acceptance criteria
• Embed quality standards into model architecture from the beginning, particularly for complex AI systems
• Define golden truth and dataset standards, ensuring they reflect real-world complexities
• Establish quality expectations for third-party AI models to maintain governance standards
• Develop a structured evaluation framework to assess AI systems against quality metrics like Goodness-of-Fit
• Create automated performance metrics for Generative AI, including detection of hallucinations and completeness
• Set up ongoing monitoring for model performance including triggers for necessary interventions

Qualifications

• 5-8 years of experience in Data Science, ML Engineering, or AI Quality with a focus on evaluation and statistical validation
• Practical experience in designing RAG, LLM-based Agents, or traditional ML pipelines alongside engineers
• Expert-level Python skills, particularly with tools like Pandas and Scikit-learn, and familiarity with evaluation frameworks like RAGAS or MLflow
• Experience in influencing stakeholders to adopt and implement quality practices across various teams
• Strong communication skills that connect governance policies with technical implementation nuances
• Bachelor's degree in a technical field or equivalent experience

Benefits

• Opportunity to impact the AI lifecycle through quality and evaluation leadership
• Collaborative environment working alongside AI Engineering and Data Science teams
• Exposure to advanced AI technologies, including RAG systems and Generative AI
• Gaining experience in governance frameworks for AI that are increasingly important in the industry