Data Annotator

What You’ll Do

• Content Moderation QA: Review and evaluate content moderation decisions made by vendors to ensure accuracy, consistency, and adherence to policies/guidelines
• Golden Set Creation: Develop and maintain golden sets of content to test and calibrate vendor performance against our quality standards
• Vendor Calibration: Partner with vendor sites to identify gaps, provide feedback, and implement calibration plans to improve performance
• Process Improvement: Spot opportunities to improve moderation workflows, tools, and processes; collaborate with internal teams to drive changes
• Data Analysis: Analyze QA metrics and trends to surface insights that inform quality strategy and vendor performance
• Cross-functional Communication: Work closely with Ops, Policy, and Leadership to ensure alignment on quality initiatives
• Training & Development: Stay current on policies, procedures, and industry best practices through ongoing training

What We’re Looking For

• Experience in content moderation, Trust & Safety, data labeling/annotation, QA, or policy enforcement
• Strong attention to detail with ability to apply nuanced policies consistently
• Experience with golden set creation, calibration, or vendor/BPO management is a strong plus
• Comfortable analyzing data and communicating findings – Excel/Sheets, basic metrics
• Excellent written communication for documenting policy rationale and feedback
• Exposure to AI/ML data operations, LLM/RLHF, or annotation tools is a bonus
• Ability to thrive in a fast-paced environment with multiple priorities