Member of Technical Staff, Evals

About the role

Evals builds the internal platform that teams across Magic use to evaluate the performance of internal and external models. The team supports pre-training, post-training, data, inference, and product, and sits on the critical path of many of the company's most important decisions.

Responsibilities

• Build and maintain the internal evals platform used across Magic
• Implement and validate eval tasks for pre-training, post-training, reinforcement learning, inference, and product systems
• Develop infrastructure for running large-scale evaluations
• Build systems to measure dataset quality and identify opportunities to improve training data
• Improve evaluation correctness, reproducibility, and reliability
• Audit and improve upon public benchmarks, evaluation methodologies, and open-source implementations
• Partner with research, data, inference, and product teams to define metrics that accurately reflect model quality
• Build tooling and frameworks that enable teams across Magic to make decisions based on trustworthy measurements

Requirements

Experience building production systems, internal platforms, or developer infrastructure

Experience working with machine learning systems, evaluation frameworks, data infrastructure, or research tooling

Track record of owning technical projects end-to-end

Skepticism toward results that cannot be reproduced, validated, or explained

Ability to reason critically about benchmarks, metrics, and experimental methodology

Experience designing, implementing, or operating systems that run at scale

Comfortable navigating ambiguity and determining whether a measurement is actually capturing the behavior it claims to measure

Excitement about helping researchers and engineers make better decisions through trustworthy measurements

Qualifications

Experience with Python, TensorFlow, PyTorch, or similar ML frameworks

Experience with databases and data management

Experience with distributed systems and cloud computing

Experience with version control systems such as Git

Experience with CI/CD pipelines

Skills

Strong problem-solving skills

Excellent communication and collaboration skills

Ability to work independently and as part of a team

Benefits

Annual salary range between $200K - $550K depending on experience

Equity is a significant part of total compensation, in addition to salary

401(k) plan with 6% salary matching

Generous health, dental, and vision insurance for you and your dependents

Unlimited paid time off

Visa sponsorship and relocation support for candidates moving to San Francisco

A small, fast-moving, highly collaborative team working on frontier AI systems

Magic strives to be the place where high-potential individuals can do their best work

We value quick learning and grit just as much as skill and experience

Our culture:

• Integrity. Words and actions should be aligned
• Hands-on. At Magic, everyone is building
• Teamwork. We move as one team, not N individuals
• Focus. Safely deploy AGI. Everything else is noise
• Quality. Magic should feel like magic