Senior AI QA Engineer (Manual & Automation)

Responsibilities

• Monitor and audit live sports broadcasts (NBA, MLB, NFL, NHL) to ensure the AI/Inference Service correctly tracks, frames, and labels major moments (e.g., touchdowns, home runs, buzzer-beaters) in real-time
• Work directly with video frames, timecodes, transcriptions, captions, and JSON outputs to ensure pinpoint alignment between AI detections and actual broadcast moments
• Act as the human-in-the-loop expert to catch AI hallucinations, misinterpretations of complex sports rules, or edge-case errors. Collaborate directly with AWS and core engineering teams to detail bugs and validate model resolutions
• Review AI-generated labels/tags to ensure they align with sports context and meet advertising industry compliance standards (e.g., IAB guidelines), ensuring content is brand-safe for monetization
• Design and execute automation scripts to compare AI inference outputs against customer-provided ground truth data at scale
• Keep meticulous records of inference service performance, track accuracy metrics, log defects, and help streamline iterative testing processes for ongoing model updates

Requirements

• Hybrid QA Experience: Proven experience in both manual and automation testing, ideally in video-focused or AI-driven environments
• Automation Scripting: Hands-on automation scripting skills (e.g., Python, Selenium, or similar testing frameworks) to validate JSON outputs and data payloads against ground truth datasets
• SDLC & QA Methodologies: Strong understanding of testing lifecycles, including detailed bug logging, defect triage, regression testing, and quality reporting
• LLM & CV Familiarity: Solid understanding of testing LLMs (workflows, prompt/response validation) and conceptual familiarity with computer vision and transcription analysis
• Media & Video Literacy: Experience working with video frame analysis, timecodes, subtitles/captions, and metadata structures
• Sports Domain Knowledge: A deep understanding of major sports leagues (NFL, NBA, MLB, NHL), including gameplay rules, terminology, and key metrics
• Communication: Strong verbal and written communication skills to act as a bridge between technical development teams and business stakeholders
• Analytical Mindset: An iterative, meticulous approach to data validation, checking model outputs for accuracy, bias, and context

Qualifications

• Nice to have: Prior experience with specialized video testing tools, media processing pipelines, or video player frameworks
• Professional background in sports analytics, sports media, or digital broadcasting technology

Benefits

This Remote Position Cannot be Performed in New York City.

Pay

$100 - $105 per hour

Schedule

Remote Position