Lead AI Engineer - Agentic Test Automation

About the role

Fully onsite McLean, VA location: Tysons, Virginia

Responsibilities

• Agentic test automation foundation (reusable patterns + reference implementations)
• Minimum coverage expectations by service/component
• Test plan templates
• GenAI-assisted reporting and quality insights across microservices:
  • Test execution results (Karate/Playwright + CI runs)
  • Release readiness narratives
• "Quality gates" via agents:
  • Required fields present (acceptance criteria, testable outcomes, data needs, dependencies)
• Design and implement agentic testing patterns that can be adopted by multiple Underwriting teams (and later other domains)
• Create reference implementations (sample repos / templates) demonstrating:
  • Test generation assistance (from requirements, APIs, contracts, schemas)
  • Test maintenance assistance (auto-updating selectors/contracts, flaky test triage)
  • Failure analysis assistance (root cause suggestions, log correlation, defect drafting)
• Establish a standard architecture for test code organization, tagging, data management, and execution across UI + API + service layers
• Coverage standards, templates, and governance:
  • Define and publish coverage standards (what "good" looks like) including: test type mix (unit vs API vs UI vs contract vs integration) and risk-based prioritization and traceability to requirements
  • Provide templates usable across teams: test case/spec templates (Gherkin-style or equivalent) and Definition of Ready / Definition of Done quality checklists
  • Create a scalable tagging/metadata strategy (e.g., feature, service, risk, priority, data sensitivity) to support reporting and quality gates
• Build automated reporting that aggregates test + service data across multiple microservices, such as:
  • Service health signals (logs/metrics/traces if available)
  • Defect signals (issue tracker metadata if available)
• Generate GenAI-driven summaries:
  • Failure clustering and trend analysis
  • "What changed?" insights (commit/PR correlation)
• Produce outputs consumable by engineering leadership and teams (dashboards, markdown summaries in PRs, artifacts in CI)
• Build automated review agents that evaluate user stories/requirements for minimum required clarity and data before development/testing starts:
  • Ambiguity detection and missing edge cases
  • Data/privacy considerations and environment needs
• Integrate gates into workflow (PR checks, issue templates, GitHub Actions) to reduce churn and rework

Qualifications

Required Technical Skills (must-have)

• GenAI / LLM + agentic development:
  • Hands-on experience building LLM-powered agents (tool-using, multi-step reasoning, guardrails)
  • Experience with prompting patterns, structured outputs (JSON schemas), evaluation, and reducing hallucinations
• Ability to design agent workflows for:
  • Test generation/augmentation
  • Requirements review and completeness validation
  • Report generation and summarization
• GitHub platform + GHCP (Copilot) for engineering workflows:
  • Strong proficiency with GitHub Copilot in day-to-day development
• Deep experience with GitHub platform capabilities:
  • GitHub Actions (CI/CD pipelines, reusable workflows, composite actions)
  • PR checks, branch protections, CODEOWNERS, templates
  • Automation via GitHub APIs/webhooks (as needed)
• Test automation engineering (framework expertise):
  • Advanced experience designing and implementing automation with Karate (API testing, contract-like checks, data-driven testing, mocks)
  • Playwright (UI automation, selectors strategy, parallelization, trace/video artifacts)
• Strong understanding of test design and coverage:
  • Happy path scenarios
  • Negative/validation scenarios
  • Edge/boundary scenarios
  • Data setup/teardown strategies and test isolation
• Cross-service reporting and data aggregation:
  • Proven ability to aggregate and normalize results from multiple microservices and multiple pipelines
  • Experience producing actionable automated reports (trend analysis, failure clustering, service correlation)
• Automated requirements review agents:
  • Experience implementing automated checks that validate acceptance criteria completeness
  • Required test data and environment dependencies
  • Non-functional requirements (performance, security, observability) when applicable

Pay & Schedule

• Job type: Contract
• Salary: $65 - 90 per hour
• Work hours: 9am to 5pm
• Education: Bachelors