Evaluation Engineer

About the role

Elicit is seeking a software engineer to lead the development of its auto-evaluation systems. The ideal candidate will work on improving the speed, usability, and accuracy of Elicit’s evaluation processes, particularly in the context of pharmaceutical decision-making.

Responsibilities

• Build a comprehensive, fast, and user-friendly core auto-eval platform
• Ensure evaluations are accurate and reliable through statistical testing and confidence intervals
• Mentor the evals engineering intern
• Work closely with the evals team to build and improve specific evals
• Understand and improve how people interact with the eval system
• Learn about user needs to ensure evals measure what matters

Requirements

• At least 3 years of experience as a professional software engineer
• Aptitude and interest in evaluating how Elicit supports pharma decision-making
• Knowledge of statistics and advanced Python (asyncio/trio, parallel processing)
• Experience with front-end technologies and strong UX sensibility
• Experience building developer tools and data pipelines
• Knowledge of pharma/biomed and experience evaluating ML systems
• Experience building language-model-based systems

Qualifications

• PhD in Computer Science or related field preferred
• Experience with TypeScript
• Knowledge of pharma/biomed
• Experience evaluating ML systems
• Experience building language-model-based systems

Benefits

• Flexible work environment
• Health, dental, vision, and life insurance
• Flexible vacation policy
• Wellbeing stipend
• 401K with a 6% employer match
• New Mac and workstation setup budget
• AI experimentation and learning budget
• Team administrative assistant

Pay

We target starting ranges of: Career (L3): $140-185k + equity, Senior (L4): $175-230k + equity.