Member of Technical Staff, Pre-Training Data

Why this role?

As a Machine Learning Engineer specializing in pretraining data, you will play a pivotal role in developing the data pipeline that underpins Cohere’s advanced language models. In this role, you will conduct data ablations to assess data quality and experiment with data mixtures to enhance model performance. You will develop robust data modeling techniques to ensure datasets are structured and formatted for optimal training efficiency. You will research and implement innovative data curation methods, leveraging Cohere’s infrastructure to drive advancements in natural language processing. You will collaborate with cross-functional teams, including researchers and engineers, to ensure data pipelines meet the demands of cutting-edge language models.

What you'll do:

• Conduct data ablations to assess data quality and experiment with data mixtures to enhance model performance.
• Develop robust data modeling techniques to ensure datasets are structured and formatted for optimal training efficiency.
• Research and implement innovative data curation methods, leveraging Cohere’s infrastructure to drive advancements in natural language processing.
• Collaborate with cross-functional teams, including researchers and engineers, to ensure data pipelines meet the demands of cutting-edge language models.

Requirements:

• Strong software engineering skills, with proficiency in Python and experience building data pipelines.
• Familiarity with curriculum learning, data mixing, and data attribution.
• Experience working with large-scale datasets, including web data, code data, and multilingual corpora.
• Knowledge of data quality assessment techniques and experimentation with data mixtures.
• A passion for bridging research and engineering to solve complex data-related challenges in AI model training.

Bonus:

• Paper at top-tier venues (such as NeurIPS, ICML, ICLR, AIStats, MLSys, JMLR, AAAI, Nature, COLING, ACL, EMNLP).

Full-Time Employees At Cohere Enjoy These Perks:

• A weekly lunch stipend of $75/£75 or equivalent in your local currency for lunch.
• Full health and dental benefits, including a separate budget for mental health.
• RRSP matching, 401K, Pension Scheme.
• 100% Parental Leave top-up for up to 6 months, for either parent.
• Annen enrichment benefits: Arts & culture, fitness/wellness, quality time, and a workspace improvement credit.
• Education & learning stipend for conferences, courses, and coaching.
• 6 weeks of paid vacation (30 working days!).
• Budget for traveling to other offices if you are remote, plus an annual company offsite.