AI QA Trainer - LLM Evaluation - Freelance Project

Responsibilities

• Converse with advanced language models on real-world scenarios and evaluation prompts
• Verify factual accuracy and logical soundness
• Design and run test plans and regression suites
• Build clear rubrics and pass/fail criteria
• Capture reproducible error traces with root-cause hypotheses
• Suggest improvements to prompt engineering, guardrails, and evaluation metrics
• Partner on adversarial red-teaming, automation (Python/SQL), and dashboarding to track quality deltas over time

Requirements

• Bachelor’s, master’s, or PhD in computer science, data science, computational linguistics, statistics, or a related field
• Shipped QA for ML/AI systems
• Safety/red-team experience
• Test automation frameworks (e.g., PyTest)
• Hands-on work with LLM eval tooling (e.g., OpenAI Evals, RAG evaluators, W&B)

Qualifications

• Evaluation rubric design
• Adversarial testing/red-teaming
• Regression testing at scale
• Bias/fairness auditing
• Grounding verification
• Prompt and system-prompt engineering
• Test automation (Python/SQL)
• High-signal bug reporting

Skills

• Clear, metacognitive communication

Benefits

Pay range: $6-$65 per hour

Schedule

Remote

Employment Type

Contract