AI Agent Evaluation Engineer - Remote

Key Responsibilities

• Create reinforcement learning environments for software engineering tasks.
• Design tasks involving bug fixing, feature development, refactoring, and performance optimization.
• Build deterministic verification systems and golden reference solutions.
• Evaluate AI agents' ability to reason through complex codebases and use MCP tools effectively.
• Develop realistic, reproducible software engineering scenarios.
• Ensure tasks accurately measure coding ability, problem-solving, and tool usage.

Required Skills

• Strong proficiency in Python 3, Java, Rust, C++, or TypeScript.
• Strong understanding of algorithms and data structures.
• Experience with bug fixing and debugging complex software issues.
• Proven experience in feature implementation and codebase refactoring.
• Strong knowledge of performance optimization and tuning.
• Excellent written and verbal communication skills.
• Strong attention to detail.

Preferred Qualifications

• Experience working with large-scale or distributed codebases.
• Familiarity with AI/ML systems (not required).
• Experience with rigorous code reviews and software engineering best practices.
• Experience working effectively in remote or cross-functional teams.

Hiring Process

• Submit an application and screening questions.
• Complete an AI interview (~30 minutes).
• Complete a technical assessment, if required.
• Hiring Manager review.

Compensation

Compensation is output-based, with payment provided per task that meets project specifications. Minimum weekly submission requirements may apply.

Availability

Selected experts should be prepared to begin their first tasks within 24-48 hours of completing onboarding.