AI Experience Evaluator | $30/hr Remote

About the role

LLM AI Quality Analyst (Personalization) English – Short-Term Contractor Assignment (4 months) – Remote.

Responsibilities

• Design and execute creative multi-turn conversational prompts, typically involving 1 5 turns, based on personal context and experiences.
• Evaluate AI responses based on the intent and requirements of the starting prompt.
• Assess whether personalization is accurate, relevant, and appropriately applied.
• Analyze responses for Grounding issues, including unsupported claims, incorrect personalization, flawed inferences, and hallucinations.
• Evaluate Integration quality to ensure personal information is incorporated naturally without unnecessary or robotic overnarration.
• Compare and stack-rank two AI model responses side-by-side based on helpfulness, usability, naturalness, and overall quality.

Requirements

• Minimum 1 year of relevant experience in data annotation, AI quality evaluation, content moderation, or a related analytical role is strongly preferred.
• Strong English proficiency, with the ability to read and write English at a high level.
• Exceptional analytical thinking and ability to evaluate nuanced and ambiguous AI responses.
• Experience or demonstrated ability in creative prompt engineering and multi-turn conversational testing.
• Strong understanding of personalization concepts, including identifying incorrect personalization, poor inferences, unsupported claims, and forced connections.

Pay

$30/hour

Schedule

Full-time availability, with 30 or 40 hours per week and at least 4 hours overlap with PST