AI Response Evaluator | $30/hr Remote

About the role

Position: LLM AI Quality Analyst (Personalization) English. Type: Short‑Term Contractor Assignment (4 months). Location: Remote.

Responsibilities

• Design and execute creative multi‑turn conversational prompts, typically involving 1‑5 turns, based on personal context and experiences.
• Evaluate AI responses based on the intent and requirements of the starting prompt.
• Assess whether personalization is accurate, relevant, and appropriately applied.
• Analyze responses for Grounding issues, including unsupported claims, incorrect personalization, flawed inferences, and hallucinations.
• Evaluate Integration quality to ensure personal information is incorporated naturally without unnecessary or robotic overnarration.
• Compare and stack‑rank two AI model responses side‑by‑side based on helpfulness, usability, naturalness, and overall quality.

Requirements

• Minimum 1 year of relevant experience in data annotation, AI quality evaluation, content moderation, or a related analytical role is strongly preferred.
• Strong English proficiency, with the ability to read and write English at a high level.
• Exceptional analytical thinking and ability to evaluate nuanced and ambiguous AI responses.
• Experience or demonstrated ability in creative prompt engineering and multi‑turn conversational testing.
• Strong understanding of personalization concepts, including identifying incorrect personalization, poor inferences, unsupported claims, and forced connections.

Pay

$30/hour

Schedule

Full‑time availability, with 30 or 40 hours per week and at least 4 hours overlap with PST.