Computational Linguist, AI Evaluation

About the role

You will be a vital member of our ML Data Operations Team – which leads the full spectrum of video-language data collection, labeling operations, and model quality measurement. This role comes with high ownership and includes responsibilities such as defining dataset and evaluation requirements in consultation with our research and product teams; designing and building data pipelines; and coordinating with our vendor partners that execute at scale. You will also be responsible for automating as much of the repetitive partnership and annotation-quality-evaluation work as possible.

Responsibilities

• Evaluation Strategy & Design: Define and build evaluation protocols for video-language model quality, working with Research and Product teams to translate ambiguous quality questions into measurable, repeatable benchmarks or evaluation flows.
• Data-to-Insight Pipelines: Turn raw customer and usage data into structured signal, building pipelines and analyses that surface where our models are underperforming and working with XFN partners to translate these insights into concrete improvement plans.
• Labeling Operations: Design and execute video-language data collection and labeling projects, automating repetitive processes so the team can focus on higher-leverage work.
• Vendor & Partner Collaboration: Coordinate with vendor and outsourcing partners executing at scale, keeping quality high through clear guidelines and feedback loops.
• Cross-Functional Partnership: Work closely with Engineering and Research teams to align on top-priority data and evaluation needs, communicating findings through dashboards and reports that drive decisions.

Requirements

• Direct experience building or running model evaluation pipelines (benchmark design, human eval frameworks, automated scoring systems), and translating ambiguous quality questions into measurable criteria.
• 5+ years of experience working in an AI focused data operations organization.
• A proven track record designing and executing large scale data projects, including gathering, labeling, and post-processing data.
• The ability to analyze messy and complex data, identify overarching patterns, and distill your findings into crisp annotation guidelines or other accessible documentation.
• Proficiency with Python, agentic coding, or other popular industry tools for automation.
• Excellent communication and project management skills, and the ability to support several projects simultaneously.
• A foundational understanding of and interest in LLMs/VLMs and multimodal AI.
• Conviction that data is the key ingredient for the performance of AI models.

Preferred Qualifications

• Experience in data collection and labeling for multimodal language models.
• Experience working with research scientists and engineers.
• Experience planning and operating new data tools and labeling systems.
• Expertise or interest in video-centric domains, such as sports, advertising, and content creation.