Data Quality Engineer, AI Business

About the role

The Data Quality Engineer within Prolific AI Data Services will oversee the quality design for managed service studies, ensuring the data delivered is trustworthy, authentic, and scalable. This role involves designing and implementing quality measurement systems, building and deploying automated checks, and leading calibration sessions.

Responsibilities

• Own end-to-end quality design for Prolific managed service studies, including rubrics, acceptance criteria, defect taxonomies, severity models, and clear definitions of done.
• Define, implement, and maintain quality measurement systems, such as sampling plans, golden sets, calibration protocols, agreement targets, adjudication workflows, and drift detection.
• Build and deploy automated quality checks and launch gates using Python and SQL, including schema and format validation, completeness checks, anomaly detection, consistency testing, and label distribution monitoring.
• Partner with Product and Engineering to embed in-study quality controls and authenticity checks into workflows, tooling, and escalation paths.
• Write and continuously improve guidelines and training materials to align participants, reviewers, and internal teams on evolving quality standards.
• Investigate quality and integrity issues end to end, running root-cause analysis across guidelines, UX, screening, training, and operations, and driving corrective and preventive actions (CAPAs).
• Build dashboards and operating cadences to track defect rates, rework, throughput versus quality trade-offs, integrity events, and SLA adherence.
• Lead calibration sessions and coach QA leads and reviewers to improve decision consistency, rubric application, and overall quality judgement.
• Translate one-off quality fixes into repeatable, scalable playbooks across customers, programs, and study types.

Requirements

• 5+ years of experience in quality engineering, data or annotation quality, analytics engineering, trust and integrity, or ML/LLM evaluation operations.
• Strong proficiency in Python and SQL, with comfort applying statistical concepts like sampling strategies, confidence levels, and agreement metrics.
• A proven track record of turning ambiguous or messy quality problems into clear metrics, automated checks, and durable process improvements.
• Strong quality systems thinking, able to translate complex edge cases into clear rules, tests, rubrics, and governance mechanisms.
• Hands-on experience instrumenting workflows and implementing pragmatic automation that catches quality and integrity issues early.
• Demonstrated ability to influence cross-functional teams (Product, Engineering, Operations, Client teams) and drive change without direct authority.
• Strong customer empathy, understanding what “useful, trustworthy data” means for research, AI training, and evaluation use cases.

Benefits

• Competitive salary
• Equity
• Benefits
• Remote working