LLM Evaluator | $20/hr Remote

Short-term contractor assignment (4 months) – up to 40 hours per week with PST overlap. Remote role.

Responsibilities

• Design and evaluate multi-turn prompts using personal context and experiences.
• Assess AI responses for personalization, grounding, integration, and helpfulness.
• Compare model responses side-by-side and evaluate their overall quality.
• Identify inaccurate personalization, unsupported inferences, and other response issues.
• Provide clear written rationales and detailed feedback on model performance.
• Review the use of relevant information in responses and follow project data-handling procedures.

Requirements

• Strong English proficiency, including reading and writing.
• Strong analytical thinking and attention to detail.
• Ability to evaluate nuanced and ambiguous AI responses.
• Ability to create and assess multi-turn conversational prompts.
• Strong written communication skills and ability to provide structured feedback.
• Ability to work independently in a remote environment.
• Reliable computer and internet connection.
• Willingness to use a personal Google account and required data sources for the evaluation process.
• Full-time availability in the local time zone for applicable locations.

Pay

$15–$20 per hour.

Schedule

Up to 40 hours per week with PST overlap.