Senior Software Engineer in Test (AI Agentic Systems)

Outcome Evaluation

Outcome Evaluation (The "What")

• Golden Set Governance: Build and maintain a versioned library of "Grounding Data" results by working with senior claims examiners to define "Ground Truth."
• Model-as-a-Judge Automation: Design automated "LLM-grading-LLM" workflows using custom rubrics to score factual grounding and policy compliance.
• Semantic Assertion Framework: Develop testing libraries that move beyond string matching to validate semantic equivalence and numerical accuracy in agent outputs.

To be successful in this role, you'll need:

• Required Skills (The Core Bar)
• Python SDET Expertise: Expert in Python and pytest, specifically building custom mocking frameworks for external APIs (Vertex AI/ADK).
• AI/LLM Observability: Hands-on experience with Vertex AI Experiments, Auto-SxS, and Cloud Logging for trace analysis.
• Data Literacy: Expert-level SQL (BigQuery) and Pandas skills to "diff" massive datasets and identify adjudication discrepancies.
• Prompt Engineering for QA: Ability to analyze "System Instructions" and refine prompts based on failed test cases to close logic gaps.
• Architectural Testing: Experience testing multi-layer systems involving RAG (Vertex AI Search), state management (LangGraph), and function calling.

Trajectory Evaluation

Outcome Evaluation (The "How")

• Function-Call Auditing: Use Vertex AI traces to programmatically verify that mandatory tools (via MCP) were invoked with correct arguments.
• Orchestration Logic Validation: Assert that agents respect defined priorities across the four architectural layers: Data & Knowledge, Orchestration, Agentic Reasoning, and Tooling.
• Reasoning Trace Auditing: Ensure every autonomous decision is traceable to a specific SOP sentence and a live API data point.

Continuous Automated Regression

To be successful in this role, you'll need:

• Required Skills (The Core Bar)
• CI/CD Integration: Every prompt or model update in Vertex AI Prompt Management must trigger an automated regression run.
• Auto-SxS: Own the automated pairwise comparison process to detect logic drift between "New" and "Production" agent versions.
• Mocking & Resilience: Build a Vertex AI/ADK mocking layer to simulate model responses, allowing for thousands of logic tests in seconds with zero API costs.

Why Join Us?

• Mission-driven culture that values innovation, collaboration, and a commitment to excellence in healthcare
• Impactful projects that shape the future of our organization
• Opportunities for professional development through internal mobility opportunities, mentorship programs, and courses tailored to your interests
• Flexible work arrangements and a supportive work-life balance

Pay and Benefits

Pay Range: $99,200 USD - $124,000 USD

Benefits: health insurance, 401k, and paid time off

Learn more about our benefits at https://jobs.collectivehealth.com/benefits/