Model Validation Senior Manager

Responsibilities

• Lead end-to-end validation of AI, GenAI, and Agentic AI solutions, from initial review of objectives and data through production readiness and ongoing monitoring.
• Design and execute fit-for-purpose validation plans (testing strategy, acceptance criteria, documentation requirements, and traceability) aligned to broader risk management processes for AI.
• Perform effective independent challenge of model design choices, data suitability, assumptions, limitations, and intended use.
• Validate model performance and stability using appropriate methods (for example: benchmarking, back-testing where applicable, sensitivity and stress testing, error analysis, and scenario-based evaluation).
• Develop models (example: credit risk models) and conceptualize modernization of associated processes.
• Validate GenAI and Agentic AI behaviors and controls (for example: evaluation of response quality, hallucination and grounding checks for RAG, prompt and tool-use testing, guardrails, escalation paths, and audit logging).
• Assess trustworthiness topics and recommend mitigations (for example: bias and fairness considerations, explainability, robustness, privacy and security considerations, and operational controls).
• Develop and embed automated processes for model validation, monitoring, and reporting (for example: standardized test harnesses, evaluation pipelines, and model documentation templates).
• Serve as a key contributor to project planning and direction-prioritizing client goals, managing technical risks, and ensuring teams execute to plan.
• Translate technical findings into clear, decision-ready messages for client stakeholders, and influence decisions with evidence-based recommendations.
• Supervise, mentor, and develop team members through coaching, technical review, and hands-on support.

Qualifications

• Undergraduate degree in Computer Science, Data Science, Artificial Intelligence, Applied Mathematics, or a related field.
• Advanced Python skills; ability to guide production-quality code (readable, well-tested, with well-designed APIs).
• Experience with GenAI frameworks and components such as Hugging Face, OpenAI APIs, Llama models, Gemini, Claude, Granite, retrieval-augmented generation (RAG), and Stable Diffusion-particularly as relevant to evaluation, controls, and validation.
• Experience with Agentic AI frameworks such as LangChain, LangGraph, AutoGen, Semantic Kernel, and CrewAI-particularly as relevant to tool-use testing, safety controls, and operational risk.
• Experience with at least 1 deep learning framework such as PyTorch or TensorFlow/Keras.
• Experience building and deploying AI solutions on AWS, Azure, or GCP; familiarity with containerization (Docker, Kubernetes) for scalable deployments.
• Strong understanding of ML/DL methods and architectures, performance assessment, and model validation.
• Expert understanding of AI best practices and sound engineering judgment for complex issues.
• Excellent written and verbal communication skills.
• Ability to lead teams effectively through coaching, technical direction, and quality assurance.