QA Engineer, AI

About the role

We are seeking a QA Engineer (AI) to oversee quality assurance for our AI-first product. This platform, which relies heavily on prompt engineering and Large Language Models (LLMs), requires robust testing to ensure reliability across its various components.

Responsibilities

• Design and build automated end-to-end test suites using tools like Playwright or Selenium to cover frontend, backend, and API flows across our event-based system.
• Develop and maintain testing strategies specifically for LLM-based features and AI-driven workflows.
• Identify what should be automated, build the frameworks, and maintain them over time.
• Create structured test cases for prompt-based features.
• Develop and maintain test datasets for AI evaluation.
• Define evaluation criteria for AI outputs.
• Run regression testing when prompts or models change.
• Track quality metrics over time (output quality, failure rate, latency issues).
• Serve as the team’s go-to expert on automated and end-to-end testing practices.

Qualifications

• 5+ years of experience in QA / software testing, preferably for LLM-based products.
• Strong hands-on experience with automation frameworks such as Playwright or Selenium.
• Proficiency in Python (automation, LLM evaluation, scripting).
• Experience testing APIs and full-stack systems.
• Genuine curiosity about AI and a solid understanding of its evolution.
• Experience thinking in edge cases and breaking systems intentionally.
• Experience working with LLM-based products and prompt engineering workflows.

Pay

The Pay Range For This Role Is 130,000 - 160,000 USD per year (Hybrid, San Mateo, CA)