Manager, AI Quality & Reliability Engineering

Responsibilities

• Lead day-to-day AI Quality Engineering activities supporting AI-powered applications, LLM-enabled workflows, intelligent automation solutions, agentic systems, and enterprise AI platforms.
• Implement and mature AI Quality Engineering practices, including AI-native testing approaches, validation processes, runtime quality controls, reusable testing accelerators, and scalable testing frameworks.
• Capture and coordinate AI validation efforts, including functional testing, prompt testing, workflow validation, regression testing, defect management, release readiness, and production quality assurance.
• Partner with AI Engineering, AIOps, LLMOps, Security, Governance, Clinical, Data, and Product teams to ensure scalable and effective quality engineering processes across AI initiatives.
• Support runtime quality and reliability through observability, monitoring, telemetry analysis, incident support, drift detection, distributed tracing, and continuous improvement initiatives.
• Drive adoption of AI-assisted testing, intelligent automation, reusable test assets, and modern quality engineering practices that improve efficiency and quality outcomes.
• Lead delivery coordination activities including sprint execution, testing planning, issue tracking, risk identification, dependency management, and release support.
• Collaborate with clinical, operational, and engineering stakeholders to validate healthcare workflows, payer operations, and AI-enabled business processes while supporting responsible AI deployment.
• Mentor quality engineers, analysts, contractors, and delivery teams while fostering a culture of continuous learning, engineering excellence, and operational accountability.

Qualifications

• Bachelor's degree in Computer Science, Information Systems, Engineering, or related field preferred.
• 7 or more years of experience in Quality Engineering, Software Testing, Quality Assurance, or related technology disciplines required.
• Experience supporting AI-powered applications, machine learning solutions, LLM-enabled workflows, intelligent automation, or emerging AI technologies required.
• Experience implementing test strategies, validation frameworks, automation solutions, and quality controls across complex enterprise technology environments required.
• Knowledge of AI Quality Engineering concepts including AI validation, prompt testing, model evaluation, runtime assurance, observability, monitoring, and production reliability required.
• Experience partnering with cross-functional teams including Engineering, Product, Data, Security, Governance, Clinical, and Operations stakeholders required.
• Demonstrated ability to lead testing efforts, manage quality risks, coordinate releases, and support delivery across multiple initiatives required.
• Strong analytical, problem-solving, and communication skills with the ability to translate technical quality concerns into actionable business recommendations required.
• Experience within healthcare, payer, provider, or highly regulated industries preferred.
• Experience with AI-assisted testing tools, test automation frameworks, observability platforms, and modern software delivery practices preferred.