Remote Data Labeling Specialist (New York)

What You Will Do

• Perform data labeling and data annotation for NLP and computer vision tasks using provided tools and schemas
• Execute QA evaluation workflows (spot checks, inter-annotator agreement reviews, review queues, systematic error tagging)
• Complete RLHF evaluation including preference ranking, pairwise comparisons, and rubric-based scoring for LLM responses
• Conduct prompt evaluation to grade instruction-following, reasoning quality, and safety behaviors
• Maintain annotation guidelines compliance by escalating ambiguous cases and documenting edge-case rationales
• Contribute to training data quality via feedback loops, consistency audits, and taxonomy refinement

Core Workstreams (project-dependent)

• Named entity recognition (NER) and entity linking
• Computer vision annotation (bounding boxes, polygons/segmentation, keypoints, attribute tagging)
• Content safety labeling (harassment, self-harm, hate, violence, adult content)
• LLM evaluation (helpfulness, factuality, refusal quality, conversational coherence)
• Data curation (deduplication flags, language identification, domain tagging)

Requirements

• Professional experience in data labeling, annotation operations, or evaluation workflows
• Strong attention to detail and consistent application of rubrics and taxonomies
• Familiarity with QA evaluation concepts (sampling, agreement checks, review processes)
• Comfort working with ambiguity and writing clear decision rationales
• Able to meet throughput and quality targets while following versioned guidelines

Compensation

Base pay range: $30–$50/hour (USD, HOURLY), depending on project scope and demonstrated quality metrics.

This role is remote within the US and optimized for New York-based job seekers.