Lead AI Engineer - Agentic Test Automation

Fully onsite in McLean, VA (Tysons, Virginia). Contract role with work hours from 9am to 5pm.

Responsibilities

• Design and implement agentic testing patterns for adoption by multiple Underwriting teams and other domains, including:
  • Test generation assistance (from requirements, APIs, contracts, schemas)
  • Test maintenance assistance (auto-updating selectors/contracts, flaky test triage)
  • Failure analysis assistance (root cause suggestions, log correlation, defect drafting)
• Establish a standard architecture for test code organization, tagging, data management, and execution across UI + API + service layers.
• Define and publish coverage standards, including:
  • Test type mix (unit vs API vs UI vs contract vs integration)
  • Risk-based prioritization and traceability to requirements
• Provide reusable templates:
  • Test case/spec templates (Gherkin-style or equivalent)
  • Definition of Ready / Definition of Done quality checklists
• Create a scalable tagging/metadata strategy (e.g., feature, service, risk, priority, data sensitivity) to support reporting and quality gates.
• Build automated reporting aggregating test + service data across microservices:
  • Service health signals (logs/metrics/traces if available)
  • Defect signals (issue tracker metadata if available)
  • GenAI-driven summaries (failure clustering, trend analysis, "What changed?" insights via commit/PR correlation)
• Produce outputs consumable by engineering leadership and teams (dashboards, markdown summaries in PRs, artifacts in CI).
• Build automated review agents evaluating user stories/requirements for minimum clarity and data before development/testing:
  • Ambiguity detection and missing edge cases
  • Data/privacy considerations and environment needs
• Integrate gates into workflow (PR checks, issue templates, GitHub Actions) to reduce churn and rework.

Requirements

• GenAI / LLM + agentic development:
  • Hands-on experience building LLM-powered agents (tool-using, multi-step reasoning, guardrails)
  • Experience with prompting patterns, structured outputs (JSON schemas), evaluation, and reducing hallucinations
  • Ability to design agent workflows for:
    • Test generation/augmentation
    • Requirements review and completeness validation
    • Report generation and summarization
• GitHub platform + GHCP (Copilot) for engineering workflows:
  • Strong proficiency with GitHub Copilot in day-to-day development
  • Deep experience with GitHub platform capabilities:
    • GitHub Actions (CI/CD pipelines, reusable workflows, composite actions)
    • PR checks, branch protections, CODEOWNERS, templates
    • Automation via GitHub APIs/webhooks (as needed)
• Test automation engineering (framework expertise):
  • Advanced experience designing and implementing automation with:
    • Karate (API testing, contract-like checks, data-driven testing, mocks)
    • Playwright (UI automation, selectors strategy, parallelization, trace/video artifacts)
  • Strong understanding of test design and coverage:
    • Happy path scenarios
    • Negative/validation scenarios
    • Edge/boundary scenarios
    • Data setup/teardown strategies and test isolation
  • Cross-service reporting and data aggregation:
    • Proven ability to aggregate and normalize results from multiple microservices and pipelines
    • Experience producing actionable automated reports (trend analysis, failure clustering, service correlation)
• Automated requirements review agents:
  • Experience implementing automated checks validating:
    • Acceptance criteria completeness
    • Required test data and environment dependencies
    • Non-functional requirements (performance, security, observability) when applicable

Qualifications

Bachelor's degree.

Pay

$100 - $105 per hour.

Benefits

• Medical, prescription, dental, and vision insurance
• AD&D and life insurance offerings
• Short-term disability
• 401K plan