Lead Software Test Automation Engineer | Test Architect - Agentic AI

Key Responsibilities

• Define and own the test automation architecture and multi-layer strategy (UI/API/contract/integration/e2e), aligned to product risk and release cadence.
• Establish automation standards, patterns, and reference implementations (framework structure, page-object/screenplay usage, test data patterns, reporting).
• Drive shift-left testing practices (contract testing, component testing, pre-merge validation) and reduce reliance on fragile end-to-end tests.
• Design and implement testing approaches for non-deterministic AI outputs, including: confidence/threshold-based assertions, semantic similarity checks and rubric-based validation, goldens/benchmark datasets and evaluation harnesses, hallucination and groundedness testing strategies, and regression detection across model/prompt/version changes.
• Partner with ML/AI, Product, and Engineering to establish acceptance criteria for AI capabilities and measurable quality gates.
• Build and scale high-throughput automation projects that are reliable, fast, and easy to extend.
• Create actionable, engineering-friendly quality dashboards (pass/fail trends, flake rate, coverage, defect escape metrics).
• Integrate automation into CI/CD with quality gates (PR checks, nightly suites, release readiness criteria).
• Standardize observability for testing: logs, tracing hooks, screenshots/video, artifacts, and test run diagnostics.
• Define practices for test environments, data management, service virtualization, and dependency control.

Technical Leadership & Mentorship

• Serve as the QE technical authority: coach engineers on automation design, code quality, and best practices.
• Lead design reviews for test architecture and automation PRs; enforce coding standards and maintainability.
• Influence roadmap and delivery by translating risk into clear testing priorities.

Required Qualifications

• 8–12+ years in Quality Engineering / Test Automation with architect-level ownership of frameworks and strategy.
• Proven experience designing scalable automation across: API testing (contract + integration), UI automation with stable locator strategies (test IDs, accessibility roles), test data strategy and environment reliability.
• Strong programming skills in at least one modern language (e.g., Java, Python, TypeScript/JavaScript, C#).
• Expertise with automation tools/frameworks (examples): UI: Playwright / Cypress / Selenium, API: REST-assured / SuperTest / Postman/Newman / pytest, CI: GitHub Actions / Jenkins / Azure DevOps.
• Demonstrated ability to reduce flakiness and improve execution speed through engineering solutions.

Preferred Qualifications (Strongly Desired)

• Experience testing LLM/Agentic AI features or other probabilistic systems, including: evaluation harnesses, benchmark datasets, semantic scoring, and drift detection, prompt/version regression strategies and structured evaluation reports.
• Familiarity with modern quality approaches: test pyramid, contract testing, consumer-driven contracts, risk-based testing, shift-left.
• Knowledge of security/performance testing integration (baseline understanding; hands-on is a plus).
• Experience building reusable automation libraries for multiple teams/products.