Model Evaluation & Data Quality Lead

Key Responsibilities

• Model Evaluation: Design and build robust model evaluation frameworks, automating repetitive processes while maintaining efficiency and depth in obtaining metrics and feedback.
• Portfolio Monitoring: Manage resource allocation and timelines, adjusting direction flexibly based on real-time information across data streams.
• External Partner Collaboration: Enhance dataset and process quality through seamless collaboration with vendors and outsourcing partners.
• Data Quality & Tooling Advancement: Establish labeling guidelines, monitor data quality, and improve tools/infrastructure to build a sustainable data operations framework.
• Internal Collaboration: Partner with Engineering and AI Model teams to align on data needs, design analytical tools (reports/dashboards), and communicate project progress.

Qualifications

• 5+ years of experience in an AI-focused data operations organization.
• Proven track record designing and executing large-scale data or evaluation projects (gathering, labeling, post-processing).
• Ability to analyze complex data, identify patterns, and distill findings into crisp annotation guidelines or quality reports.
• Proficiency with Python, LLMs, or other industry tools for automation.
• Excellent communication and project management skills.
• Foundational understanding and interest in LLMs, VLMs, and multimodal AI.
• Strong belief that data is key to AI model performance and assessment.

Tech Stack

• Development & Analysis: Python (pandas, Jupyter, etc.)
• Data Management & Visualization: Amazon S3, Framework-agnostic visualization tools.
• Project Management: Linear, Notion.

Compensation & Benefits

• Open and inclusive work culture.
• Work closely with a mission-driven team on cutting-edge AI technology.
• Full health, dental, and vision benefits.
• Flexible PTO and parental leave policy.
• Office closed the week of Christmas and New Years.