Senior AI Data Annotation Specialist (Portland)

About The Role You will lead end-to-end data annotation workflows for LLM and multimodal training pipelines, ensuring training data quality and consistent labeling across projects. You will drive annotation guidelines compliance, perform QA evaluation and audit sampling, and collaborate with engineers and researchers to resolve edge cases that impact model behavior. Responsibilities Own dataset-level quality strategy, including gold sets, inter-annotator agreement, and error taxonomiesExecute RLHF and large language model evaluation tasks such as pairwise ranking and rubric-based scoringPerform QA evaluation on labeled data, identify systematic issues, and drive corrective actionsImprove annotation guidelines, ambiguity handling, and escalation workflows for complex edge casesCoordinate with AI/ML engineering stakeholders on data schemas, prompts, and evaluation metricsSupport content safety labeling and policy-aligned decisions for safety-critical datasets Qualifications Professional experience in AI data annotation, data labeling operations, or model evaluation programsStrong command of sampling methods, audits, and training data quality measurementFamiliarity with RLHF, prompt evaluation, and LLM evaluation workflowsExposure to NLP tasks such as named entity recognition and classification taxonomiesAbility to document decisions clearly and maintain traceability for dataset changes Workplace Remote, aligned to Portland-area candidates. Company: Rex.zone