Senior AI Engineer, Evals
What You Will Own
Forge throughput and the truth-quality gate. The number that matters: verified claims per week that survive adversarial challenge. You own both sides of it — loop throughput and the quality gates that keep scale from diluting truth. Like every outcome here, it is falsifiable: it ships with an evidence test a stranger could run.
The verification harness, company-wide. Every agent loop in the company — engineering, research, content — ships against verification you designed: oracle-separated verifier agents, guardrails, evals on agent behavior.
Regression corpora as the spec. Here the corpus is the contract: curated cases, paraphrase variants, adversarial attempts, golden sets. When behavior is disputed, the corpus decides. You own its growth, its honesty, and its bite.
The model-routing law. The standing rules for which model class runs which work, at what verification depth, with what fallback. Every company running agents will need this within two years; you'll have built ours from the inside.
Who You Are
You're a builder with a high technical bar whose leverage is judgment and taste — a product engineer who happens to live in verification. You decide what "verified" should mean and how you'll know it worked, and you write the spec the agents execute. Your leverage is judgment, not keystrokes. You interrogate every green checkmark. A passing eval is a claim, and claims get challenged — you ask what the test could not have caught before you ask what it confirmed. You form working models of complex systems on your own, notice where your model is wrong, and update fast. You write clearly, because clear writing is evidence of clear thought. You move between verification architecture and harness code without getting stuck at either altitude — a confidence-tier scheme in the morning, the verifier fleet that enforces it by evening. You treat agents as leverage you verify.
Who this isn't for
- This role is wrong if you mainly optimize for leaderboard scores, or if you trust vendor-reported evals.
- It's wrong if your instinct when output is weak is to massage the wording rather than redesign the verification, and wrong if you'd rather publish findings than ship gates.
- It's wrong if your code is whatever the model handed you and you couldn't say why it's right, or if you treat agents as autocomplete you trust rather than leverage you verify — and wrong if you're comfortable letting an agent grade its own work.
- It's wrong if you think of yourself as a heads-down coder who wants tickets and defined scope rather than a builder who decides what to build.
- And it's the wrong place if you need defined process, fixed scope, and a program plan with milestones; verification here is steered live, not run as a project.
How We Evaluate
- No traditional AI engineering interviews. Written artifact. Send us a system you built and the hardest failure you personally diagnosed in it — what broke, how you found it, and what you changed.
- Video screen. Brief and async: 5-6 questions, about 15 minutes, whenever works for you.
- Calls with company stakeholders. Short conversations with key members of the team.
- Conversation with the founder. Chemistry and comprehension.
- Paid work trial. One week of paid, real evals work in our real environment — live loops, live corpora, real verdicts. We watch how you ground yourself, whether you write the spec before the build, how you verify what your agents produce, and whether your self-assessment is honest.
Compensation & Ownership
- Total first-year comp: $375,000 – $450,000 (base + ownership + profit sharing).
- Base: $250,000 – $300,000, top of market for senior AI engineering.
- Profits Interest Units (PIUs) — Class B Membership Interests at $0 strike, real ownership day one, capital-gains treatment; annual pro-rata profit sharing from free cash flow; annual tender liquidity; 100% family premium coverage; effectively unlimited token budget, steered by ROI, never capped.