Member of Technical Staff - Platform

About the role

We are looking for an exceptional engineer to join our team. You will own the platform that runs our benchmarks, spanning everything needed to evaluate Large Language Models (LLMs) at scale.

Responsibilities

• Build distributed systems to run evaluations across multiple models, benchmarks, and machines at scale
• Deploy cloud infrastructure using Infrastructure as Code (IaC), including deployment pipelines, servers, logging, monitoring, etc.
• Contribute to the internal and external libraries we maintain, including our public model library
• Develop full-stack features for our platform using React/TypeScript on the frontend and Python/Django on the backend
• Perform code and architecture reviews for other members of the team
• Help establish engineering best practices across the organization
• Collaborate closely with the research team to ensure our infrastructure meets their needs

Requirements

• Technical: Strong engineering fundamentals, significant experience in Python, familiarity with LLMs, and experience with cloud infrastructure and IaC
• Non-technical: Team collaboration, strong communication skills, comfort with ambiguity, and iteration speed

What You'll Do

• Build distributed systems to run evaluations across multiple models, benchmarks, and machines at scale
• Deploy cloud infrastructure using IaC, including deployment pipelines, servers, logging, monitoring, etc.
• Contribute to the internal and external libraries we maintain, including our public model library
• Develop full-stack features for our platform using React/TypeScript on the frontend and Python/Django on the backend
• Perform code and architecture reviews for other members of the team
• Help establish engineering best practices across the organization
• Collaborate closely with the research team to ensure our infrastructure meets their needs

Non-technical

• Team collaboration: Experience working in development sprints, Git workflows, and pull request reviews
• Communication: Strong ability to provide and receive feedback effectively (spoken and written)
• Comfort with ambiguity: Ability to break down fuzzy problems into clear and actionable steps
• Iteration speed: Tenacity to develop and iterate quickly

Location

We are an in-person team based in San Francisco. We will support your relocation or transportation as needed.

Nice to Haves

• Experience with frontend development, ideally React
• Experience with Django, FastAPI, or other Python-based HTTP servers
• Experience working with AWS infrastructure, including IaC
• Experience at early-stage startups or your own company
• Interest in AI/ML systems and evaluation

What We Offer

• Highly competitive salary and meaningful ownership
• Relocation and transportation support
• Health/dental insurance coverage
• Lunch and dinner provided, free snacks/coffee/drinks
• 401K plan
• Unlimited PTO

About Us

Founded by researchers from Stanford, we raised a $5M seed from top institutional and angel investors in the valley. Our team includes Stanford PhDs, ex-Jane Street quants, and the first designer at Snorkel. Our tech stack is Python for most things, with Django for the backend and React for the frontend, all built on AWS using CDK for IaC. Our platform serves startups, enterprises, and research labs measuring model performance, and we work with major foundation model labs, financial institutions, and hospital systems worldwide.