Senior / Staff Software AI Test Engineer, AI Engineering

Key Responsibilities

• Framework and harness engineering
• Design and build scalable, reusable test automation frameworks for AI agents, LLM-powered applications, and underlying APIs.
• Write clean, maintainable Python for test harnesses, eval pipelines, synthetic data generation utilities, and internal tooling.
• Treat test code as production code: code review, type hints, documentation, library design.
• Evaluation infrastructure
• Build evaluation infrastructure for benchmarking agent performance against SOTA LLMs, competitors, and internal baselines.
• Own regression suites, golden datasets, rubric-based evals, and metric dashboards.
• Build tooling for synthetic test data generation, edge-case discovery, and adversarial testing.
• Resilience and load
• Design and run release, system, performance, and load tests against streaming, stateful, and async systems.
• Build chaos and fault injection tooling for token expiry, connection pool exhaustion, provider failover, and cache pressure scenarios.
• Drive contract testing across LLM providers (Bedrock, Anthropic, OpenAI) to catch parity drift.
• CI/CD and observability
• Integrate automated tests into CI/CD so every model, prompt, and code change is validated before it ships.
• Build trace-based assertions on LangGraph state, tool calls, and agent decisions — debugging an agent failure means replaying graph state, not re-running a prompt.
• Make observability a first-class testing surface (LangSmith, audit logs).
• Human-in-the-loop and partnership
• Implement HIL review workflows where automation alone cannot validate quality, then push the automation boundary outward.
• Partner with AI engineers and data scientists on model evaluation, training and eval data prep, and root-cause debugging of complex end-to-end failures.
• Champion quality engineering practices across the team: code review, coverage standards, observability, reproducibility.
• Ensure user-centric validation so AI outputs are accurate, reliable, and meet real-world application needs.

Requirements

• 3-7 years of software engineering experience, with a meaningful portion focused on test automation, SDET, or software engineering in test roles.
• Expert-level Python. You write Python every day, design libraries other engineers use, and apply OOP and clean-code practices.
• Hands-on Java experience, enough to read, write, and test Java services, not just touch them.
• Working understanding of the LangGraph or Vercel frameworks: graph state, nodes, edges, tool calls, and how to write evals against agentic flows.
• Demonstrated experience building eval sets for LLM models (this is critical to the role).
• Experience testing across multiple client surfaces: iOS apps, plugins, and Chrome extensions.
• Hands-on experience building automated test suites with frameworks such as pytest, Selenium, Playwright, Cypress, or similar.
• Proven experience integrating test automation into CI/CD systems (GitHub Actions, Jenkins, CircleCI, GitLab CI, or similar).
• Strong skills in data manipulation, test data preparation, and SQL.
• Bachelor's degree or higher in Computer Science, Engineering, or a related field.