Retail SME - AI Evaluation Expert

About the role

Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark, General Catalyst, Peter Thiel, Adam D'Angelo, Larry Summers, and Jack Dorsey.

This is a contract position offering $60–$80/hour, fully remote.

Responsibilities

• Guide research and engineering teams to close knowledge gaps in retail merchandising, category management, and operations reasoning.
• Design challenging, domain-relevant retail tasks and write accurate, well-reasoned solutions grounded in real retail practice.
• Evaluate AI model outputs against structured rubrics and provide clear, written feedback on correctness, judgment, and reasoning quality.
• Develop and refine evaluation guidelines and scoring rubrics specific to retail tasks.
• Collaborate with other subject matter experts to ensure consistency and accuracy in training data.

Requirements

• 8+ years of dedicated professional experience in retail (e.g., merchandising, category management, retail operations, buying/planning) at a recognized, top-tier organization (e.g., Amazon, Walmart, Target, Nike, Costco, Home Depot, or equivalent).
• Prior hands-on experience evaluating LLM/AI model outputs against rubrics or structured scoring criteria.
• Demonstrable career progression (e.g., Category Manager → Senior Manager → Director of Merchandising).
• Ability to engage reliably for at least 35 hours/week during weekdays.
• Strong verbal and written communication skills, problem-solving skills, and interpersonal skills.