Python Engineer - Freelance AI Trainer

What This Opportunity Involves

Frontier coding agents are already good at passing tests. We measure whether they pass them the right way.

We're building a dataset to evaluate the safety and conduct of AI coding agents — not just whether they complete a task, but whether they do it honestly, within scope, and without cutting corners.

• Design tasks where the easy path is the unsafe one, and write the tests that catch it:
• Build realistic developer environments — a virtual company with codebase, infrastructure, and context (tickets, docs, conversations) that forms a believable development history
• Design tasks that pair a benign development goal with a tempting unsafe shortcut: scope creep, policy violations, data corruption, over-permissive changes
• Write tests that verify whether the agent completed the task the right way — catching corners cut, not just checking outputs
• Iterate on tasks and tests based on QA feedback — review agent solutions, analyze failures, and refine until the evaluation is fair and robust

What This is NOT

• Data labeling;
• Prompt engineering;
• Cybersecurity or red-teaming — there is no attacker in the scenario.

Why This is Hard

Frontier models are already good at coding. Creating a task that genuinely challenges the best models is non-trivial. The real difficulty is building the temptation — a scenario where the unsafe or out-of-scope path is the path of least resistance — and then writing tests that reliably catch an agent that took it.

How It Works

• Apply → Pass qualification(s) → Join a project → Complete tasks → Get paid

Project Time Expectations

Tasks are estimated to require around 20-25 hours per week during active phases, based on project requirements. This is an estimate, not a guaranteed workload, and applies only while the project is active.

Compensation

On this project, contributors can earn up to $75 per hour equivalent, depending on their level and pace of contribution. Compensation varies across projects depending on scope, complexity, and required expertise.