AI Quality Engineer - 17397

About the role

We are seeking an experienced AI Quality Engineer to lead the validation, certification, and production readiness of enterprise Generative AI and AI-powered automation solutions. This is a highly technical engineering role focused on ensuring AI systems are accurate, reliable, secure, explainable, compliant, and ready for enterprise production deployment.

Responsibilities

• Develop and implement AI validation, certification, and production readiness standards for enterprise AI solutions.
• Design evaluation frameworks to measure AI accuracy, response relevance, groundedness, completeness, hallucination detection, retrieval effectiveness, recommendation quality, and user satisfaction.
• Build and maintain AI validation datasets, benchmark scenarios, regression suites, and golden datasets using tools such as LangSmith and Azure AI Foundry.
• Validate RAG (Retrieval-Augmented Generation) solutions utilizing Azure AI Search, LangChain, LangGraph, Azure AI Foundry, and enterprise knowledge repositories.
• Review AI solution architectures deployed across Azure cloud services including Azure Container Apps, Azure Functions, Azure Databricks, and Azure SQL Cosmos DB.
• Evaluate AI agent workflows, orchestration pipelines, prompt execution, tool integrations, guardrails, human-in-the-loop processes, and MCP integrations.
• Assess AI security, governance, auditability, identity management, and compliance controls including Entra ID, RBAC, Managed Identities, Key Vault, and data protection requirements.
• Develop production readiness checklists covering observability, monitoring, resiliency, logging, supportability, recoverability, and operational excellence.
• Analyze AI telemetry, LangSmith traces, execution logs, and evaluation metrics to identify quality issues and optimization opportunities.
• Partner with engineering teams to resolve AI quality, security, and performance concerns before production deployment.
• Produce AI certification reports, quality scorecards, dashboards, and executive summaries for governance reviews.
• Establish independent quality gates and certification criteria for enterprise AI deployments.
• Lead validation and production readiness reviews for Internal Developer Portal (IDP), Developer Experience (DevEx), self-service engineering workflows, and platform automation initiatives.
• Drive continuous improvement of AI testing strategies, evaluation methodologies, and quality engineering practices.

Requirements

• Bachelor's degree in Computer Science, Software Engineering, Artificial Intelligence, Data Science, Information Systems, or a related technical discipline.
• 5+ years of experience in Software Quality Engineering, Test Architecture, Software Development, Platform Engineering, AI Engineering, Machine Learning Engineering, or related technical roles.
• 2+ years of hands-on experience with Generative AI, Large Language Models (LLMs), AI Agents, or Retrieval-Augmented Generation (RAG) solutions.
• Strong understanding of AI evaluation techniques including hallucination detection, groundedness validation, accuracy testing, and AI quality metrics.
• Experience with one or more of the following technologies: Azure AI Foundry, Azure OpenAI, LangChain, LangGraph, LangSmith, AI Agent frameworks, RAG architectures.
• Experience with Azure cloud technologies including Azure AI Search, Azure Container Apps, Azure Functions, Cosmos DB, Azure SQL, Azure Databricks, Azure Key Vault.
• Strong knowledge of automated testing frameworks, regression testing, AI validation methodologies, and quality certification processes.
• Experience with APIs, microservices, distributed systems, and cloud-native architectures.
• Familiarity with DevSecOps, CI/CD pipelines, observability, monitoring, and enterprise SDLC practices.
• Excellent analytical, troubleshooting, documentation, and stakeholder communication skills.
• Ability to work independently while providing objective, data-driven quality assessments.

Preferred Qualifications

• Experience validating enterprise AI agents or multi-agent systems.
• Background in Platform Engineering, Developer Experience (DevEx), Site Reliability Engineering (SRE), or DevOps.
• Experience with AI observability and evaluation platforms such as LangSmith.
• Knowledge of Azure AI Search, vector databases, semantic search, embeddings, and enterprise knowledge retrieval.
• Experience implementing Responsible AI, AI Governance, AI Risk Management, or AI Compliance frameworks.
• Experience in highly regulated industries such as financial services, banking, healthcare, or insurance.
• Familiarity with Azure DevOps, GitHub, GitHub MCP, Azure DevOps MCP, and enterprise SDLC tooling.
• Experience with performance engineering, resiliency testing, chaos engineering, and production readiness reviews.
• Knowledge of Entra ID, RBAC, Managed Identities, Azure Key Vault, and identity governance.
• Experience creating executive dashboards, KPIs, quality scorecards, and AI performance reporting.

Pay

The pay rate for this position is $53 - $60 per hour.

Schedule

This is a contract position, and the location is Vienna, VA / Remote.