Strategic Projects Lead

About the role

Pareto builds human training data pipelines for frontier AI labs. As a Strategic Projects Lead, you will own the architecture, execution, and continuous improvement of complex data collection and evaluation workflows.

Responsibilities

• Design end-to-end data collection and evaluation pipelines for RLVR, RLHF, SFT, red-teaming, and model evaluation workflows.

• This includes expert sampling strategy, annotation schema, rubric structure, inter-rater calibration, and QA system design.

• You'll be expected to prototype novel workflows quickly, identify architectural risks before launch, and make tradeoff decisions with confidence.

• Build, test, and iterate on AI agents that automate pipeline tasks — quality gate review, expert matching, output flagging, throughput anomaly detection.

• Define data quality standards across annotation, evaluation, and expert output review.

• Evaluate new approaches — model-assisted annotation, structured output formats, automated calibration methods — and integrate them into active pipelines where they improve quality or efficiency.

• Engage directly with AI researchers, TPMs, and PMs at our client organizations.

• Translate research-driven requirements into operational workflows.

• Communicate pipeline performance clearly, escalate technical risks early, and contribute to project scoping and pricing decisions.

• Stay current with developments in LLM post-training, evaluation methodology, and data tooling.

Requirements

• Proficiency in Python and SQL for data manipulation, pipeline monitoring, and quality analysis.

• Hands-on experience with at least one agentic or LLM workflow framework (LangChain, DSPy, AutoGen, direct tool-use via API, or equivalent).

• Demonstrated ownership of a data or ML pipeline from scoping through delivery — including quality design, not just throughput tracking.

• Strong written communication: you'll write technical guidelines and rubrics that distributed expert workers follow accurately, and you'll brief senior researchers on pipeline performance.

• Comfort operating with ambiguity in a fast-moving environment where model requirements shift and client priorities evolve.

• Direct experience with RL environment data pipelines, evaluation framework design, and red-teaming workflows.

• Background in data engineering, ML research support or equivalent.

• Experience designing or operating agentic systems in a production or near-production context.

• Familiarity with inter-rater reliability methods, calibration set design, and annotation quality frameworks.

• Prior client-facing or technical program management experience in an AI/ML-adjacent context.

• Prior experience on scoping or driving projects with fuzzy upfront specs or evolving requirements.

Qualifications

• Strong background in software engineering, data science, or ML research.

Benefits

• Compensation Range: $140K - $180K