Member of Technical Staff, Evals

About the role

Evals builds the internal platform that teams across Magic use to evaluate the performance of internal and external models. The team supports pre-training, post-training, data, inference, and product, and sits on the critical path of many of the company's most important decisions.

Responsibilities

• Build and maintain the internal evals platform used across Magic
• Implement and validate eval tasks for pre-training, post-training, reinforcement learning, inference, and product systems
• Develop infrastructure for running large-scale evaluations
• Build systems to measure dataset quality and identify opportunities to improve training data
• Improve evaluation correctness, reproducibility, and reliability
• Audit and improve upon public benchmarks, evaluation methodologies, and open-source implementations
• Partner with research, data, inference, and product teams to define metrics that accurately reflect model quality
• Build tooling and frameworks that enable teams across Magic to make decisions based on trustworthy measurements

Requirements

Experience building production systems, internal platforms, or developer infrastructure

Experience working with machine learning systems, evaluation frameworks, data infrastructure, or research tooling

Track record of owning technical projects end-to-end

Skepticism toward results that cannot be reproduced, validated, or explained

Ability to reason critically about benchmarks, metrics, and experimental methodology

Experience designing, implementing, or operating systems that run at scale

Comfortable navigating ambiguity and determining whether a measurement is actually capturing the behavior it claims to measure

Excitement about helping researchers and engineers make better decisions through trustworthy measurements

Qualifications

Experience with Python, TensorFlow, PyTorch, or similar ML frameworks

Experience with databases, data processing pipelines, and distributed systems

Strong understanding of machine learning concepts such as model evaluation, hyperparameter tuning, and transfer learning

Experience with version control systems like Git

Excellent communication and collaboration skills

Skills

• Python
• Machine Learning
• Data Processing
• Distributed Systems
• Version Control

Benefits

Annual salary range between $200K - $550K depending on experience

Equity is a significant part of total compensation, in addition to salary

401(k) plan with 6% salary matching

Generous health, dental, and vision insurance for you and your dependents

Unlimited paid time off

Visa sponsorship and relocation support for candidates moving to San Francisco

Pay

Annual salary range between $200K - $550K depending on experience

Schedule

Full-time