AI Quality Engineer

About the role

An experienced AI Quality Engineer is sought to establish and execute the quality, validation, certification, and production readiness processes for AI-powered SDLC and enterprise automation solutions. This is a specialized engineering position focused on validating AI systems, Retrieval-Augmented Generation (RAG) architectures, agent workflows, and orchestration frameworks.

The successful candidate will serve as an independent reviewer and certification authority responsible for evaluating the quality and effectiveness of AI-generated outputs and agent behavior. The role is located in Merrifield, Virginia (Hybrid) and is also open to Remote U.S.

Responsibilities

• Define and implement AI certification, validation, and production readiness standards for enterprise AI agents and intelligent automation solutions.
• Develop and execute evaluation frameworks that measure accuracy, relevance, groundedness, completeness, hallucination rates, and user satisfaction.
• Design and maintain AI validation datasets, benchmark scenarios, and regression test suites using platforms such as LangSmith and Azure AI Foundry.
• Validate Retrieval-Augmented Generation (RAG) solutions built using Azure AI Search, Azure AI Foundry, LangChain, and LangGraph.
• Perform architecture reviews and quality assessments for AI solutions deployed on Azure Container Apps, Azure Functions, Azure Databricks, and related cloud-native services.
• Evaluate AI orchestration workflows including prompt execution, agent routing, tool calling, and retrieval pipelines.
• Validate security, governance, auditability, and compliance controls.
• Create production readiness and certification checklists covering observability, monitoring, logging, and operational readiness.
• Analyze LangSmith traces, agent execution logs, and AI evaluation metrics to identify quality concerns.
• Partner with development teams to remediate quality, accuracy, security, and performance issues prior to production deployment.
• Lead certification and production readiness reviews for Internal Developer Portal (IDP) capabilities and developer experience improvements.

Qualifications

A Bachelor’s Degree in Computer Science, Software Engineering, Data Science, Artificial Intelligence, Information Systems, or a related technical field is required.

This role requires 5+ years of experience in software quality engineering, test architecture, platform engineering, software development, or machine learning engineering. A minimum of 2+ years of experience working directly with Generative AI, Large Language Models (LLMs), RAG architectures, or AI agents is necessary.

Skills

• Experience with one or more of the following: Azure AI Foundry, Azure OpenAI, LangChain, LangGraph, LangSmith, AI agent frameworks, or RAG architectures.
• Experience with cloud-native application architectures and Azure services.
• A strong understanding of DevSecOps, CI/CD, and enterprise software development practices.

Benefits

Everforth Apex offers a range of supplemental benefits, including medical, dental, vision, life, disability, and other insurance plans that offer an optional layer of financial protection.

We offer an ESPP (employee stock purchase program) and a 401K program which allows you to contribute typically within 30 days of starting, with a company match after 12 months of tenure.

Everforth Apex also offers a HSA (Health Savings Account on the HDHP plan), a SupportLinc Employee Assistance Program (EAP) with up to 8 free counseling sessions, a corporate discount savings program and other discounts.