Applied Data Scientist, LLM Evaluation

About the role

This role will build the LLM evaluation function from scratch. You will define what "good" means for generated content, build the infrastructure to measure it, and create the experimental framework to ship changes with confidence. You will own the LLM evaluation strategy, define quality metrics, build evaluation datasets, establish benchmarking and experimentation infrastructure, develop automated quality signals, and quantify tradeoffs to inform decisions.

Responsibilities

• Define what "good" means for generated content, build the infrastructure to measure it, and create the experimental framework to ship changes with confidence.
• Own the LLM evaluation strategy, define quality metrics, build evaluation datasets, establish benchmarking and experimentation infrastructure, develop automated quality signals, and quantify tradeoffs to inform decisions.

Qualifications

• Education: Bachelor's, Master's, or PhD in Statistics, Machine Learning, Data Science, Computational Linguistics, or a related quantitative field.
• Minimum 3-5 years in applied science, ML engineering, or data science roles with a focus on evaluation, NLP, or generative AI. 7+ years experience preferred.
• Required Technical Skills:
  • Strong statistical foundations (experimental design, hypothesis testing, etc.).
  • Experience designing and running evaluations for LLM or NLP systems.
  • Proficient in Python and the scientific/data stack (pandas, NumPy, scipy, sklearn).
  • Comfortable working in Jupyter notebooks and automating pipelines.
  • Experience with LLM-as-judge approaches, inter-annotator agreement, and rubric design.
  • Familiarity with practical challenges of non-deterministic systems.
  • Strong data storytelling skills.

Benefits

• Competitive Compensation Packages (Cash & Equity)
• Flexible Work Culture
• Unlimited Time Off + 12 Paid Company Holidays
• Health, Dental, & Vision Insurance
• Life Insurance & FSA Accounts
• 401(k) Retirement Accounts
• Quarterly Team Offsites