Senior Research Engineer, Model Evaluation

Evaluation

As models continue to become superhuman in many real-world use cases, we must continue to develop new techniques to accurately measure our models' performance on frontier capabilities.

Responsibilities

• Develop evaluation benchmarks, datasets, and environments for measuring the bleeding edge of model capabilities
• Conduct research to push the state-of-the-art in LLM evaluation methods, including training LLM judges; improving evaluation efficiency; and scalably building high-quality datasets
• Build scalable tools for investigating and understanding evaluation results that are used by all members of technical staff at Cohere, as well as leadership and our CEO
• Learn from and work with the best researchers and engineers in the field

Requirements

You enjoy pushing the limits of what LLMs are capable of, and you have built high-quality evaluation resources to measure those capabilities (datasets, simulators, environments, etc.).

You have a track record of developing new methods and/or data to evaluate LLMs, e.g. publications at top-tier conferences, popular benchmarks, etc.

You have deep experience building with and around LLMs, and you have built tools for analyzing and understanding their performance.

You have strong software engineering skills.

Qualifications

Full-Time Employees at Cohere Enjoy These Perks:

• A weekly lunch stipend of $75/£75 or equivalent in your local currency for lunch.
• Full health and dental benefits, including a separate budget for mental health.
• RRSP matching, 401K, Pension Scheme.
• 100% Parental Leave top-up for up to 6 months, for either parent.
• Annual enrichment benefits: Arts & culture, fitness/wellness, quality time, and a workspace improvement credit.
• Education & learning stipend for conferences, courses, and coaching.
• 6 weeks of paid vacation (30 working days!).
• Budget for traveling to other offices if you are remote, plus an annual company offsite.

Benefits

• A weekly lunch stipend of $75/£75 or equivalent in your local currency for lunch.
• Full health and dental benefits, including a separate budget for mental health.
• RRSP matching, 401K, Pension Scheme.
• 100% Parental Leave top-up for up to 6 months, for either parent.
• Annual enrichment benefits: Arts & culture, fitness/wellness, quality time, and a workspace improvement credit.
• Education & learning stipend for conferences, courses, and coaching.
• 6 weeks of paid vacation (30 working days!).
• Budget for traveling to other offices if you are remote, plus an annual company offsite.

Pay

• Competitive salary based on experience and qualifications.

Schedule

• Full-time position.