QA Engineer (SDET) — AI, Data & Platform Quality

What You'll Do

• Programmatic QA of Core Features
  • Extend our scenario test runner — a proprietary harness that captures real production personalization requests and replays them across environments, asserting on expected algorithms and content selection.
  • Grow it into automated regression across every client.
  • Write automated tests in Python with pytest across our tiers — unit, integration, HTTP, and end-to-end.
  • Build headless Playwright end-to-end tests to verify how personalized content and tracking render on real client pages.
  • Harden the pre-deploy quality gate and pre-commit checks that block bad changes automatically.
• Testing & Standardizing LLMs and Agents
  • Design evals for non-deterministic AI features — our conversational analytics assistant, AI content builders, and generative SEO — measuring correctness, grounding, and regression across prompt and model versions.
  • Test the tool-calling and agentic layers — that function-calling loops pick the right tools and guardrails hold on adversarial input.
  • Validate our agent/MCP interface — contract conformance, rate limiting, authorization, and safe failure.
  • Help set our standards for shipping AI — catching hallucinations and drift, and benchmarking prompt/model changes before clients see them.
• Data Quality Engineering
  • Build automated data-health checks that flag stale rollups, incomplete coverage, and broken aggregations before they hit a client dashboard.
  • Validate data pipelines end-to-end — rollups, funnel/rate/financial ingestion, and BigQuery — with drift detection across environments.
  • Guard model inputs so the signals our ML depends on stay accurate and complete.
• Reliability & Performance
  • Track platform performance — response times, JS load, and page speed — and help keep it fast.
  • Stand up quality dashboards — uptime, coverage, data-health, and eval scores.
• Collaboration & Bug Lifecycle
  • Work in the codebase alongside engineers to diagnose issues across development, release, and deployment.
  • Drive the bug lifecycle — reproduce, capture with a failing test, and verify the fix.

What We're Looking For

• 3+ years in QA/SDET or test automation with a code-first approach.
• Strong Python — you write clean test code and can read the app you're testing. pytest (preferred) and browser automation (Playwright or Selenium).
• API and contract testing experience.
• A genuine interest in testing AI — comfortable with non-determinism, evals, and prompts.
• Data-savvy — strong SQL, and the instinct to validate pipelines and reconcile data.
• Building automated quality gates into the deploy and release process.

Nice to Have

• Testing or evaluating LLM applications — evals, prompt regression, tool-calling agents, or MCP.
• Data or analytics QA — BigQuery or ETL/rollup validation.
• Django, MySQL, or Celery experience.
• Security testing with SAST/DAST tooling.
• Familiarity with machine learning.
• Financial industry, personalization, or CMS/marketing-platform experience.
• Familiarity with AWS.
• SaaS startup experience on a fast-moving, multi-tenant platform.