AI Rater Guidelines Writer (Linguist / Instructional Designer)

Key Responsibilities

• Translate complex and ambiguous program requirements into clear, concise, and easy-to-follow rater guidelines
• Design comprehensive evaluation rubrics and scoring frameworks that enable consistent decision-making across a variety of AI tasks
• Review existing documentation to identify ambiguity, inconsistencies, contradictions, and coverage gaps, then revise guidelines to improve clarity and usability
• Convert subject-matter requirements from domains such as finance, retail, insurance, legal, and sports into structured evaluation instructions that non-domain raters can apply confidently
• Collaborate with research teams, program managers, and subject matter experts to maintain consistency across multiple guideline sets
• Develop documentation that addresses edge cases, exceptions, and complex evaluation scenarios while minimizing reviewer escalation

Required Qualifications

• Minimum 3 years of professional experience in Linguistics, Instructional Design, Technical Writing, AI Content Development, or a closely related field
• Direct experience creating, refining, or maintaining evaluation guidelines, rating rubrics, or reviewer instructions within Generative AI, RLHF, human evaluation, or AI data annotation environments
• Demonstrated ability to work across multiple subject areas and convert domain-specific knowledge into clear, structured documentation
• Proven experience resolving ambiguity and improving written specifications with measurable before-and-after improvements
• Strong analytical thinking with exceptional attention to detail
• Demonstrated career progression and professional growth
• Able to commit reliably to 35+ hours per week during weekdays
• Outstanding written communication skills with the ability to explain nuanced concepts in a precise, consistent, and easy-to-understand manner

Preferred Qualifications

• Experience supporting AI model training, evaluation, RLHF, or large-scale annotation programs
• Background working with cross-functional teams including researchers, engineers, product managers, and subject matter experts
• Familiarity with structured documentation standards, quality assurance methodologies, and guideline governance
• Experience designing documentation that supports scalable, high-quality human evaluation processes