Staff Quality Engineer

About the role

The Staff Quality Engineer owns quality for the Unboxed platform. This is a senior individual contributor role with technical authority over how quality is defined, measured, and enforced across every surface shipped, including work produced by teams outside QA. Quality spans four surfaces: a robust learning management system, AI-generated roleplay and practice, coaching workflows, and client-facing analytics and reporting.

The role is hands-on: designing test architecture, writing and reviewing automation, running exploratory testing on high-risk changes, and diagnosing defects personally. The difference between this role and a senior testing role is reach—this person sets standards that other engineers follow, changes how the platform is built to ensure testability, and holds release authority on behalf of the company.

A significant portion of the role focuses on quality for AI-generated output. Unboxed delivers AI scoring, feedback, and roleplay to enterprise clients in regulated industries. The Staff Quality Engineer owns the program that validates system performance, accuracy, and fairness across client populations.

The role includes direct management, mentorship, and development of two QA Engineers, as well as setting quality standards for contract engineering and QA partners.

Responsibilities

• Quality Architecture and Strategy
  • Own the test architecture for the platform, including framework selection, automation strategy, test data management, and environment strategy
  • Set the quality standard for the organization: what gets tested, at what layer, to what depth, and what coverage gaps are consciously accepted
  • Author and maintain the defect severity rubric, definitions of done, and test conventions used across in-house and contract teams
  • Stay hands-on in the highest-risk areas of the product, including exploratory testing, complex defect diagnosis, and automation for critical paths
• Release Authority
  • Hold go/no-go authority for releases against a published, evidence-based standard, and communicate the basis for each decision
  • Own release verification and the quality signal that supports it, ensuring ship decisions rest on data rather than confidence
  • Escalate accepted risk explicitly, with exposure named, rather than allowing it to pass unrecorded
• AI Quality and Fairness
  • Own how quality is measured for AI-generated scoring, feedback, and roleplay, using evaluation approaches built for output variability rather than exact-match expectations
  • Build and maintain the evaluation harness: golden sets, rubric-based scoring, regression gates on model, prompt, and scoring-logic changes, and drift monitoring in production
  • Own the fairness and bias testing program for AI scoring, including subgroup analysis, monitoring for differential performance across learner populations, interpreting results, and escalating findings
  • Partner with measurement and psychometrics expertise on methodology and thresholds
  • Produce evidence artifacts enterprise clients and their compliance functions rely on to trust AI-generated output, and translate client trust concerns into verifiable acceptance criteria
• Quality Outcomes and Reporting
  • Own the outcome metrics for platform quality, including Defect Escape Rate, Critical Path Coverage, and MTTD/MTTR, and report on them directly to leadership on a regular cadence
  • Use those metrics to drive change in how software is built, not only to describe how testing performed
  • Provide leadership with a clear read on quality trends, risk concentration, and where investment is warranted
• Engineering Influence
  • Change how engineering builds, so quality moves upstream by driving testability into design, acceptance criteria into stories, and automated checks into CI
  • Review designs and pull requests for quality risk and raise the bar through direct technical influence rather than process enforcement
  • Earn adoption of quality practices across teams that do not report to this role
• Sourcing and Partner Strategy
  • Set the strategy for how testing capacity is sourced, including what stays in-house, what goes to contract partners, and where automation replaces manual effort
  • Define the standards, conventions, and acceptance criteria contract engineering and QA teams operate within, and review their output against the same bar as in-house work
  • Serve as the single point of quality accountability across in-house and contract contributions
• Mentorship and Development
  • Manage, mentor, and develop two QA Engineers, owning their prioritization, technical growth, and performance with a coaching-first approach
  • Grow each engineer's ability to reason about quality independently, so team judgment scales without requiring this role in every decision
  • Raise the quality skill level of the broader engineering organization through review, pairing, and documented practice

Requirements

• 8+ years of quality or software engineering experience in B2B SaaS, with demonstrated ownership of quality outcomes rather than test execution alone
• A track record of staying hands-on at a senior level, with systems or practices they designed that outlived their direct involvement
• Evidence of organization-level impact: a quality practice, framework, or standard they introduced that changed how a company built software
• Experience testing AI-driven, ML-driven, or data-driven features, or clear demonstrated ability to reason about quality under output variability
• Experience mentoring engineers and setting standards for contract, offshore, or vendor teams
• Strong automation and CI/CD background with modern testing frameworks
• Familiarity with enterprise quality expectations, including regulated or compliance-sensitive clients
• Experience with Agile/Scrum methodologies and tools such as JIRA
• Bachelor's degree in a relevant field, or equivalent practical experience

Skills

• Deep hands-on quality engineering skill: test design, exploratory testing, regression strategy, and defect diagnosis in B2B SaaS
• Test architecture judgment: the ability to design an automation strategy that holds up over years and to know where automation pays off versus where it becomes a maintenance burden
• Practical experience testing AI/ML or otherwise non-deterministic systems, including evaluation of scored output where there is no single correct answer
• Statistical literacy sufficient to read a subgroup analysis, distinguish signal from noise, and interpret fairness results responsibly
• Demonstrated influence without authority: the ability to change how other engineers work through credibility and evidence
• Fluency with quality metrics and the ability to use them to drive decisions, not just report them
• Ability to communicate quality risk credibly to technical and non-technical audiences, including enterprise clients

Benefits

• Competitive salary and benefits
• Ample paid time off
• Dynamic and convenient office location with unlimited snacks, casual dress code, covered parking, and on-site gym
• Open communication and a commitment to fostering teamwork across the organization

Pay

$115,000 - $130,000 annually