Research Engineer, Policy Evaluation

About the role

Orbifold AI is building the infrastructure layer for Physical AI. As intelligent systems move beyond language into the physical world, they require a fundamentally new understanding of physics, action, and interaction. We partner with leading robotics and world model research teams to advance the foundations of embodied intelligence, enabling intelligent systems to perceive, understand, and operate in the real world. The standards we set, and the infrastructure we build to scale them, will define the next frontier of robotics and Physical AI.

You will build the missing evaluation infrastructure: take a partner’s trained policy, run it against thousands of it against thousands of real scenes drawn from our corpus, and return not a score but a ranked, repeatable list of the scenes, objects and grasps that break it. Then you close the loop: every failure you characterize becomes a data specification, and the data we collect against it becomes the next evaluation. This is highly applied work. You will be measured on whether a partner’s real‑world policy improved, not on a benchmark number in isolation.

Responsibilities

• Design evaluation methodology end to end: task suites, held‑out slices, scoring, and statistical rigor that survives a skeptical research lead’s scrutiny.
• Build fine‑grained failure taxonomies: occlusion, transparency, deformables, long‑horizon, contact‑rich, distribution shift, and edge‑case discovery and long‑tall probing.
• Train and fine‑tune reference policies (π0‑class, OpenVLA, ACT, diffusion policy) on curated data to demonstrate dataset impact on metrics.
• Build automated critics and judges that approximate human evaluation at scale and correlate with downstream model behavior.
• Run evaluations on real hardware where it matters, and know precisely when a real‑robot trial is worth its cost versus a corpus replay.
• Work directly with partner research teams: reproduce their setup, run their model against our data, and be the person who tells them the truth about what you find.
• Close the loop: translate every evaluation finding into a concrete collection specification, and back again.
• Take a partner’s trained policy, run it against thousands of real scenes from our corpus, and return a ranked, repeatable list of the scenes, objects, and grasps that break it.
• Characterize each failure to create data specifications that drive the next data collection cycle.
• Measure success by real‑world policy improvement, not isolated benchmark numbers.

Requirements

• PhD or equivalent research experience in robot learning, imitation learning, or a closely related field, with first‑author publications or shipped work.
• Trained and debugged real policies on real robots; able to distinguish a policy failure from a robot miscalibration.
• Real opinions about how embodied models should be evaluated, with the statistical literacy to defend them.
• Self‑driven and high agency, with experience in fast‑paced applied research or startup environments.
• We index on the quality of the work rather than years served; a recent PhD with strong first‑author publications in a directly relevant area is