World Model Evaluation Lead

Responsibilities

• Drive Benchmarking & Evaluation - Design and implement rigorous evaluation methodologies and benchmarks for measuring model effectiveness, reliability, alignment, and safety
• Lead evaluation of model performance, ranging from offline experiments to full production model testing
• Build & Scale Evaluation Frameworks - Design and oversee the pipelines, dashboards, and tools that automate model evaluation
• Design and oversee tools for A/B model testing, regression testing, and production model performance
• Lead Evaluation Strategy - Develop and implement strategies for evaluating physical AI models that can scale across a broad range of real-world use cases, sensor types, and edge cases
• Plan, run, and oversee evaluations, across internal teams and external customers
• Drive edge case discovery, red-teaming, safety, privacy, and risk evaluation - feeding back knowledge to key stakeholders in research and engineering teams

Requirements

• Extensive expertise in evaluating AI and machine learning models, ideally in physical AI or a related AI field
• Experience in designing, implementing, and refining evaluation metrics
• Deep understanding of machine learning, AI, and generative models
• Excellent python and software engineering skills
• Experience designing and building scaleable data pipelines and evaluation tools
• Experience collaborating closely with key stakeholders from research, engineering, and product teams
• Strong communication and documentation skills, with a bias for creating detailed evaluation reports that help drive model performance
• Startup-ready mindset with the ability to thrive in high-velocity, high-ambiguity environments

Qualifications

• Experience evaluating real-world, real-time algorithms
• Experience evaluating a broad range of sensor types, such as cameras, LIDAR, physical sensors, RF sensors, and beyond
• A strong scientific approach to evaluation and understanding model performance
• Experience in evaluating production algorithms
• Experience building and curating data campaigns to create extensive test datasets
• Experience managing internal teams and/or external vendors