Remote AI Engineer (United States)

Key Responsibilities

• Build and maintain LLM evaluation pipelines (rubric-based grading, pairwise preference tests)
• Integrate RLHF signals and feedback loops for safer, higher-quality outputs
• Define and iterate on annotation guidelines compliance checks and quality gates
• Develop prompt evaluation and regression suites to detect quality drops across releases
• Support NLP (e.g., named entity recognition) and CV (evidence of working with computer vision annotation) workflows
• Implement content safety labeling strategies and audit-friendly reporting
• Create dashboards/monitoring for training data quality, evaluation metrics, and production health
• Improve tooling and automation for QA evaluation, labeling workflows, and review operations

Required Qualifications

• Mid-Senior experience shipping AI/ML systems in production
• Strong Python and ability to build reliable data/evaluation pipelines
• Hands-on experience with model evaluation, prompt evaluation, and experiment design
• Understanding of RLHF concepts and preference modeling workflows
• Ability to turn ambiguous quality issues into measurable metrics and engineering tasks
• Comfort collaborating across product, data operations, and QA