AI Quality Expert | $30/hr Remote

About the Role

This is a short-term contractor assignment lasting 4 months, requiring full-time availability with 30 or 40 hours per week and at least 4 hours of overlap with PST working hours. The position is remote.

Responsibilities

• Design and execute creative multi-turn conversational prompts, typically involving 1–5 turns, based on personal context and experiences.
• Evaluate AI responses based on the intent and requirements of the starting prompt.
• Assess whether personalization is accurate, relevant, and appropriately applied.
• Analyze responses for Grounding issues, including unsupported claims, incorrect personalization, flawed inferences, and hallucinations.
• Compare and stack-rank two AI model responses side-by-side based on helpfulness, usability, naturalness, and overall quality.

Requirements

• Minimum 1 year of relevant experience in data annotation, AI quality evaluation, content moderation, or a related analytical role is strongly preferred.
• Strong English proficiency, with the ability to read and write English at a high level.
• Exceptional analytical thinking and ability to evaluate nuanced and ambiguous AI responses.
• Experience or demonstrated ability in creative prompt engineering and multi-turn conversational testing.
• Strong understanding of personalization concepts, including identifying incorrect personalization, poor inferences, unsupported claims, and forced connections.