Machine Learning Engineer – Evals

About the role

Build the measuring stick for AI memory and identity. Our client, a research-driven AI startup, builds the memory and identity layer for the agentic world. Their product works, and it's growing more sophisticated by the week. It's a multi-agent system with real black-box behaviour, and the team needs one person to answer a hard question: are our identity representations actually getting better?

There's no ground truth to lean on. You'll define what "better" means for representations that change over time, then build the machinery that measures it. If you enjoy constructing measurements from nothing, digging through traces to find what's really broken, and shipping the fix yourself, this role is built for you.

Responsibilities

• Design the evals. Define the scores and the meaning of "better" for an evolving entity representation, and keep that definition current as the product and methods move.
• Build the pipelines and harnesses. Data ingestion, labelling, versioning, reruns and LLM judges: the infrastructure that lets the team ask a new question this week and get an answer this week.
• Run them and find the insight. Read traces and results, surface what's actually broken (not just what's easy to measure) and propose fixes that lead to higher-fidelity representations.
• Build simulation agents at scale. Iteration speed is capped by how many entities can be modelled and measured at once. You'll raise that ceiling.
• Own the loop end to end. The question, the pipeline, the rerun, the write-up. No handoffs, no waiting on someone else's sprint.

Requirements

• A track record of designing, building and running LLM evals end to end
• 3+ years designing and owning LLM evaluation
• Production-quality Python: tests, maintenance and a steady release cadence
• Hands-on experience with PyTorch, Hugging Face and Weights & Biases
• Data pipeline tooling (Hydra, Kafka, SQL) is a plus

Qualifications

• Work on memory, identity, personalization, agent evaluation or agentic optimization
• Shipped research: first or co-author at NeurIPS, ICML, ICLR or equivalent, or open-source work of comparable quality
• A degree in computer science, mathematics, statistics or a related quantitative field

Pay

$220K–$300K + competitive equity

Schedule

Full-time, on-site in New York, NY, five days a week

Additional information

Candidates must be authorized to work in the US without visa sponsorship (US citizens or green card holders). One seat available.