Product Manager, AI Evaluation Lead

About the role

Own AI evaluation end to end as a dedicated program, with the mandate to create a defined, repeatable, and scalable system ("Eval Ops"). Design the evaluation methodology, build the systems and partnerships that scale it, and pull the right expertise in at the right time.

Responsibilities

• Define the evaluation methodology — test sets and benchmarks spanning the dimensions that matter for our product (accuracy/factuality, instruction-following, robustness, multi-turn conversation behavior, and domain-specific edge cases), plus regression testing to catch when model or prompt changes break previously working behavior.
• Build canonical evaluation data sets — reference projects spanning varied size, complexity, document volume, and project phase (preconstruction, execution, closeout), each with proportionate question/query banks.
• Define rubrics and own ground truth — author clear, anchored scoring rubrics and calibration sets, and stand up the human-validation workflow that confirms whether answers are actually correct and keeps reviewers aligned.
• Stand up a tiered, scalable eval pipeline — combine automated evaluation (e.g., LLM-as-a-judge) for fast iteration with expert human review for calibration and ground truth, so scarce SME time is spent where it matters most.
• Design the eval workbench and contributor workflow — define the end-to-end tooling and process SMEs use to contribute (building on our existing internal eval tooling), so the system runs efficiently at scale.
• Build and coordinate a flexible SME network — recruit, vet, onboard, schedule, and quality-control subject matter experts: Trimble SMEs where available, Document Crunch SMEs and customers where appropriate, and external/contract experts as needed.
• Design the incentive model for the network — determine what motivates busy experts (especially Trimble SMEs) to lend scarce time, and build the structure that attracts and retains them.
• Partner with Product and Engineering/QA — work with product managers to understand the features being built and the evals they require, and partner with engineering (including our existing eval tooling) to operationalize the process.
• Establish a customer feedback loop — over time, incorporate real customer queries and chat history into evaluation data sets rather than relying solely on synthesized or SME-created data.
• Measure and report quality — define metrics and dashboards that track evaluation coverage and answer quality over time, and help the company articulate, internally and to increasingly sophisticated customers, how we know our AI is high quality.

Requirements

Strong people- and network-coordination skills — proven ability to navigate a large organization, recruit and motivate contributors, and orchestrate distributed work across many stakeholders.

Hands-on fluency with AI / LLM evaluation — not just familiarity. You should be able to describe and write an eval and articulate what good eval coverage looks like, and have enough depth to design the system to requirements. You can lean on our engineering/QA and SME partners, but you need enough fluency to know what to ask them.

Excellent communication and stakeholder management across Product, Engineering/QA, and SMEs; able to translate between technical and domain audiences.

Strong project- and resource-management skills, comfortable with the ebb-and-flow of evaluation work (a heavier ramp to establish a baseline, then flexing with demand).

Detail-oriented and quality-obsessed, with sound judgment about coverage, prioritization, and when "scoped-down but shipped" beats "comprehensive but stalled."

Qualifications

Demonstrated program and process ownership — a track record of taking an ambiguous, cross-functional problem, defining a process, rallying people around it, and driving it to measurable outcomes.

Technical fluency — enough comfort with tooling to partner with engineering on (or extend) an internal evaluation workbench; light scripting or "vibe-coding" is a plus, not a requirement.

Experience building or operating an expert network, annotation/labeling program, or human-feedback pipeline (e.g., RLHF-style review or a quality-operations function).

Exposure to construction, AECO, or construction technology — enough domain awareness to queue work intelligently (deep construction expertise is not required; we can source it).

Experience in B2B SaaS or AI product environments.

A track record designing metrics and dashboards to track evaluation coverage and quality over time.

Skills

• Program and process ownership
• Strong people- and network-coordination skills
• Network-building and incentive design
• Hands-on fluency with AI / LLM evaluation
• Technical fluency
• Experience building or operating an expert network
• Domain awareness
• Experience in B2B SaaS or AI product environments
• Track record designing metrics and dashboards

Benefits

Not specified

Pay

$99,600.00–$137,000.00

Schedule

Remote United States of America

Minimal Travel Requirement