Agentic Software Test Engineer
About the Role
The Agentic Software Test Engineer — TRACE sits on the Customer Driven Solutions (C.D.S.) Team and runs the agentic testing organization for TRACE — the building energy simulation platform that powers HVAC load calculations, equipment sizing, and ASHRAE-compliant reporting for engineers worldwide. You direct a team of AI testing agents across a multi-repo .NET / Blazor codebase, working shoulder-to-shoulder with the TRACE software developers as their dedicated QA partner. You're the captain of the agent crew and the quality voice the developers trust — coaching the agents, raising the bar, and putting your name on the work that ships. This is a hands-on engineering seat; you direct the agent crew; you partner with the humans.
What You'll Do
- Direct the agentic testing team — define agent personas, dispatch patterns, and human-in-the-loop checkpoints for QA across TRACE.
- Author the prompts, context contracts, and knowledge bases that drive the agents through test plan generation, regression authoring, defect triage, exploratory testing, and accessibility sweeps.
- Build evaluation harnesses that measure agent output quality — false-positive rates on bug reports, regression coverage drift, flaky-test detection, eval-vs-production divergence — and tighten the loop sprint over sprint.
- Curate what the agents produce — test plans, automated scripts, exploratory findings, regression results — and ship the best of it into the engineering pipeline with your stamp on it.
- Hunt bugs through manual, automated, and agent-driven methods, and verify every agent-surfaced finding before it reaches the development team.
- Own TRACE's quality posture as the SME on HVAC simulation correctness, ASHRAE compliance reporting, EnergyPlus integration behavior, and cross-repo regression risk.
- Partner with developers and Product Owners on test coverage decisions, release readiness, and quality strategy.
- Run regression, functional, and data tests across the .NET 10.0 backend, Blazor frontend, PostgreSQL schema, and EnergyPlus calculation engine.
- Design adversarial verification workflows — parallel agents that try to refute a test result before it's trusted.
- Curate the agent learning loop — capture recurring failure patterns, encode them into spec rules, and feed them back into agent knowledge bases so the same defect class is caught earlier next sprint.
- Translate customer-impact reports into agent-runnable test scenarios and explain quality posture to non-technical stakeholders without hand-waving.
What You Bring
- 5+ years in software quality assurance, with a real track record of shipping tested software.
- 1+ year hands-on with LLM-based agents, AI coding assistants, or multi-agent systems in a production or near-production workflow.
- Strong test-design instincts across unit, integration, system, functional, and exploratory layers; comfortable owning test plans, defect lifecycle, regression strategy, and release gating.
- Hands-on experience with AI coding agents (Claude Code, Cursor, Copilot Workspace, or equivalent) and the Anthropic / OpenAI APIs.
- Working knowledge of prompt engineering, structured outputs (JSON Schema), tool use, and multi-agent orchestration (parallel/pipeline/judge-panel/adversarial-verify).
- Comfort designing evals and treating eval results as a first-class quality metric.
- Bias toward human-in-the-loop verification — never trusting agent output blindly.
- Ability to read and modify production code in C#, Python, or JavaScript — not just test code.
- Experience with GitHub Actions testing pipelines.
- Working knowledge of Jira (or equivalent) and testing tools like Playwright, BUnit, xUnit / NUnit.
- Strong written and oral communication; able to explain agent behavior, eval results, and quality posture in plain language.
Preferred
- Bachelor's Degree in Engineering, Computer Science, or related discipline (or equivalent experience — we care more about what you've shipped than where you studied).
- Familiarity with C#, .NET 10.0, Blazor, ASP.NET Core, Razor, plus the usual web stack (JavaScript / TypeScript / CSS / HTML).
- Working knowledge of PostgreSQL and EF Core, including database-aware test design (cascade-delete chains, FK integrity, schema migration testing).
- Background in building energy simulation, EnergyPlus, ASHRAE 140 software testing, ASHRAE 90.1 / 62.1 / 169 compliance reporting, or HVAC equipment sizing workflows.
- Agile / Scrum experience.
Working Style
This is an on-site role so you can pair tightly with the TRACE engineering team and the agent crew you're directing. Expect a fast feedback loop, candid technical conversation, and a team that takes both craft and outcomes seriously.
Pay
Annual Base Salary Range: $97,406.66 - $156,200.00. Compensation Type: Salary. Incentive Eligible: No. Sales Commission Eligible: No. Actual compensation will depend on elements such as seniority, merit, geographic location, education, experience, travel requirements, and union designation.
Benefits
- Benefits kick in on DAY ONE for you and your family, including health insurance and holistic wellness programs that include generous incentives.
- Family building benefits include fertility coverage and adoption/surrogacy assistance.
- Paid time off includes up to 15 vacation days, paid holidays, sick leave, and additional options to support volunteer and parental leave.
- 401K match up, educational and training opportunities through company programs along with tuition assistance and student debt support.