Senior AI Data Annotation Jobs in Austin

About the role

Senior AI data annotation jobs focused on producing high-quality training data for AI/ML systems. You will support LLM training pipelines through RLHF, prompt evaluation, QA evaluation, named entity recognition, computer vision annotation, and content safety labeling. This is a remote, full-time role open to candidates in the US with Austin-aligned hiring intent.

Responsibilities

• Perform expert AI data annotation and data labeling for NLP, LLM training, and computer vision datasets
• Run RLHF workflows: preference ranking, pairwise comparisons, critique writing, and rubric-based scoring
• Execute prompt/response evaluation to identify failure modes, hallucinations, and unsafe outputs
• Apply content safety labeling (toxicity, self-harm, hate, harassment, sexual content, policy violations)
• Conduct QA evaluation and audits to ensure annotation guideline compliance and consistent labeling
• Create decision logs for ambiguous cases and propose guideline refinements
• Monitor training data quality metrics and sampling strategies to improve model performance
• Collaborate with engineering and AI/ML stakeholders to integrate labeled data into LLM training pipelines

Requirements

• Mid-senior experience with annotation, labeling, or evaluation workflows
• Strong judgment and consistency on edge cases and nuanced content
• Experience with QA evaluation, error analysis, and training data quality improvement
• Comfort with LLM outputs, prompt evaluation, and RLHF ranking/scoring frameworks
• Clear written communication and documentation for auditability

Nice to Have

• Named entity recognition and taxonomy/policy-driven labeling experience
• Computer vision annotation tooling (polygons/segmentation) experience
• Familiarity with calibration exercises and inter-annotator agreement concepts