Software Engineer - AI Agent Evaluation (Remote)
Mindrift · New York, NY · Yesterday
RemoteRemoteOTHR$50/hrPart-time
What this opportunity involves
- Create realistic developer environments - a virtual company with codebase, infrastructure, and context (tickets, docs, conversations)
- Craft tasks from intermediate states of these environments - define what "solved" means and ensure the task is solvable by an AI agent
- Write tests that verify agent solutions - accept all valid approaches and reject incorrect ones, neither too strict nor too lenient
- Iterate on tasks and tests based on QA feedback - review agent solutions, analyze failures, and refine until the evaluation is fair and robust
What This Is NOT
- Data labeling
- Prompt engineering
- Writing code from scratch - the agent writes most of the code; you guide and evaluate
What We Look For
- 5+ years in software development
- Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis
- Experience writing tests (functional, integration)
- English proficiency - B2+
How It Works
- Apply
- Pass qualification(s)
- Join a project
- Complete tasks
- Get paid
Compensation
- Up to $50/hr equivalent, depending on level and pace
- Tasks are estimated at :20 hours each; you set your own schedule