STEM Careers Canada

About The Role

This remote, full-time role supports AI/ML training workflows through high-quality data labeling, RLHF-style preference ranking, prompt evaluation, and QA evaluation. You will apply clear rubrics and annotation guidelines to improve training data quality and measurable model performance.

Key Responsibilities

• Perform high-accuracy data labeling for text, image, and multimodal datasets
• Execute RLHF evaluation (preference comparisons, rubric-based scoring)
• Run prompt evaluation and response grading for LLM evaluation
• Apply content safety labeling and policy-aware categorization
• Maintain annotation guidelines compliance and propose guideline improvements
• Conduct QA evaluation, audits, and disagreement resolution to ensure training data quality
• Document edge cases clearly and support distributed, asynchronous collaboration

Required Qualifications

• Mid-Senior experience in evaluation, annotation programs, or engineering-adjacent data operations
• Strong consistency applying detailed rubrics with low error rates
• Familiarity with LLM training pipelines and preference-based grading (RLHF)
• Working knowledge of NLP concepts (e.g., named entity recognition)
• Exposure to computer vision annotation workflows preferred

Compensation

Competitive base pay: $30–$50 per hour. Remote, full-time; async-first collaboration.