Machine Learning Evaluation Specialist

About the role

What if your years of hard-earned research expertise could directly shape the future of AI? We're looking for domain experts with deep machine learning knowledge to design evaluation challenges that push state-of-the-art AI systems to their limits. This isn't reviewing chatbot responses or rating simple outputs. You'll be crafting complex, nuanced problems that only a true domain expert could construct — and that only the most capable AI systems could hope to solve. Organization: Alignerr, Type: Hourly Contract, Location: Remote, Commitment: 10–40 hours/week.

Responsibilities

• Design original, expert-level machine learning problems grounded in your area of specialization
• Build evaluation tasks that go far beyond standard ML pipelines — requiring genuine, deep domain knowledge
• Draw from your own research experience to craft challenges that rigorously stress-test highly capable LLMs
• Define precise problem statements, evaluation criteria, and gold-standard solutions
• Assess AI-generated solutions for correctness, creativity, and methodological soundness
• Document problem difficulty levels, required knowledge, and expected failure modes
• Collaborate asynchronously with a global team of researchers and engineers

Requirements

• You hold a graduate-level degree (MS or PhD preferred) in a scientific or technical field that intersects with machine learning
• You have strong working knowledge of ML methods — model selection, feature engineering, evaluation metrics, and pipeline design
• You're deeply familiar with active, open research problems in your field
• You can identify precisely where general ML knowledge breaks down and specialized domain insight becomes essential
• You have experience conducting or publishing original research (highly valued)
• You communicate complex ideas clearly and precisely in writing
• You're self-motivated and thrive working independently on intellectually demanding tasks

Pay

Top-tier compensation — among the highest rates available for contract research work