AI Quality Engineer

About the role

One of our Financial clients is seeking an experienced AI Quality Engineer to establish and execute the quality, validation, certification, and production readiness processes for AI-powered SDLC and enterprise automation solutions.

This is not a traditional manual testing role. It is a specialized engineering position focused on validating AI systems, Retrieval-Augmented Generation (RAG) architectures, agent workflows, orchestration frameworks, observability platforms, security controls, and production readiness requirements.

Responsibilities

• Define and implement AI certification, validation, and production readiness standards for enterprise AI agents and intelligent automation solutions.
• Develop and execute evaluation frameworks that measure:
  • Accuracy
  • Relevance
  • Groundedness
  • Completeness
  • Hallucination rates
  • Retrieval quality
  • Recommendation quality
  • User satisfaction
• Design and maintain AI validation datasets, benchmark scenarios, regression test suites, and golden test datasets using platforms such as LangSmith and Azure AI Foundry.
• Validate Retrieval-Augmented Generation (RAG) solutions built using Azure AI Search, Azure AI Foundry, LangChain, LangGraph, and enterprise knowledge repositories.
• Perform architecture reviews and quality assessments for AI solutions deployed on Azure Container Apps, Azure Functions, Azure Databricks, Azure SQL, Cosmos DB, and related cloud-native services.
• Evaluate AI orchestration workflows including prompt execution, agent routing, tool calling, guardrails, retrieval pipelines, human-in-the-loop processes, and MCP integrations.
• Validate security, governance, auditability, and compliance controls including Entra ID integration, RBAC, managed identities, Key Vault integrations, data protection policies, and sensitive data handling requirements.
• Create production readiness and certification checklists covering observability, monitoring, logging, resiliency, supportability, recoverability, and operational readiness.
• Analyze LangSmith traces, agent execution logs, AI evaluation metrics, and telemetry to identify quality concerns and improvement opportunities.
• Partner with development teams to identify and remediate quality, accuracy, security, and performance issues prior to production deployment.
• Create and maintain AI certification reports, scorecards, dashboards, and executive summaries for leadership and governance review boards.
• Drive continuous improvement of AI validation methodologies, testing strategies, and model evaluation frameworks across the organization.
• Establish independent quality gates that AI solutions must satisfy before production approval.
• Lead certification and production readiness reviews for Internal Developer Portal (IDP) capabilities, self-service workflows, engineering automation, and developer experience improvements.

Requirements

• Bachelor’s Degree in Computer Science, Software Engineering, Data Science, Artificial Intelligence, Information Systems, or a related technical field.
• 5+ years of software quality engineering, test architecture, platform engineering, software development, machine learning engineering, or related experience.
• 2+ years of experience working directly with Generative AI, Large Language Models (LLMs), RAG architectures, AI agents, or AI-powered business applications.
• Strong understanding of AI system evaluation methodologies including accuracy testing, hallucination detection, groundedness validation, and AI quality measurement.
• Experience with one or more of the following:
  • Azure AI Foundry
  • Azure OpenAI
  • LangChain
  • LangGraph
  • LangSmith
  • AI agent frameworks
  • RAG architectures
• Experience with cloud-native application architectures and Azure services including Container Apps, Functions, Cosmos DB, Azure AI Search, Azure SQL, Key Vault, and Databricks.
• Experience evaluating or supporting developer platforms, internal developer portals, DevOps platforms, CI/CD pipelines, platform engineering initiatives, or self-service engineering capabilities.
• Experience designing automated test strategies, quality frameworks, regression testing, and certification processes.
• Strong understanding of DevSecOps, CI/CD, observability, monitoring, and enterprise software development practices.
• Experience evaluating APIs, microservices, integrations, and distributed systems.
• Strong analytical and troubleshooting skills with the ability to investigate complex technical issues.
• Excellent documentation, communication, and stakeholder engagement skills.
• Ability to operate independently and provide objective, unbiased quality assessments.

Qualifications

The ideal candidate is a highly technical engineer who understands both modern software engineering and AI systems. This individual can independently assess whether an AI solution is accurate, secure, explainable, compliant, observable, and ready for production.

Skills

• Experience building or validating enterprise AI agents and multi-agent systems.
• Experience within Platform Engineering, Developer Experience (DevEx), Site Reliability Engineering (SRE), or DevOps organizations.
• Experience with AI observability, tracing, and evaluation platforms such as LangSmith.
• Experience validating Azure AI Search implementations, vector databases, semantic search, embeddings, and knowledge retrieval solutions.
• Experience implementing AI governance, Responsible AI, model risk management, or AI compliance frameworks.
• Experience in financial services, banking, insurance, healthcare, or other regulated industries.
• Familiarity with Azure DevOps, GitHub, GitHub MCP, Azure DevOps MCP, enterprise developer platforms, and SDLC tooling.
• Experience with performance engineering, chaos testing, resilience testing, and production readiness reviews.
• Knowledge of identity and access management concepts including Entra ID, managed identities, Key Vault integration, and RBAC controls.
• Experience creating quality scorecards, executive dashboards, KPI frameworks, and reporting metrics.
• Experience in Quality Engineering, Test Architecture, Security Engineering, Platform Engineering, or AI Engineering roles.