Software QA Analyst

About the role

This role contributes to the development of an agentic platform by testing services and evaluating agent output.

Responsibilities

• Test the services built, including control plane APIs, intake registry, agent dispatch flows, and webhook integrations, at the API and integration level.
• Review agent output, such as code review comments, generated tests, and draft MRs, assessing accuracy, relevance, and quality against the codebase and context.
• Create and maintain test cases and evaluation criteria, including API contracts, end-to-end flows, and agent output quality for each agent type (reviewer, tester, builder).
• Identify failure patterns across the stack, such as broken API responses, dropped state, hallucinations, missed issues, and bad diffs, and log them clearly.
• Work with the platform team to translate findings into fixes, guideline improvements, and guardrail changes, owning tickets end-to-end from test plan to verified improvement.
• Sit in on design reviews and help define what "done well" looks like before work ships.

Requirements

• 1-3 years of professional experience with software testing, QA tooling, or API test automation (Postman, pytest, httpx, or similar).
• Attention to detail and the ability to read code across at least one language (Python or TypeScript preferred).
• Comfortable with Git, HTTP/REST APIs, basic SQL, and Linux shell.
• Writes things down clearly — test cases, failure patterns, improvement proposals.
• Asks questions and communicates what's wrong without waiting to be asked.

Qualifications

• Familiarity with LLMs or AI agents (optional).
• Python or TypeScript (optional).
• Ci/Cd exposure (optional).
• AWS basics (ECS, CloudWatch, Secrets Manager) (optional).

Skills

• Software testing, QA tooling, or API test automation (Postman, pytest, httpx, or similar).
• Attention to detail and ability to read code across at least one language (Python or TypeScript preferred).
• Git, HTTP/REST APIs, basic SQL, and Linux shell.
• Writing clear test cases and failure patterns.
• Communicating effectively about issues and proposing solutions.

Benefits

• Paid AWS certification if desired.
• Comprehensive health, dental, and vision insurance plan.
• Basic and supplemental life insurance.
• Short and long-term disability coverage.
• Employee Assistance Program and wellness programs.
• 401K plan with matching contributions.