Lead Member of Technical Staff, Inference Infrastructure

The Lead Member of Technical Staff at Cohere is responsible for developing, deploying, and operating the AI platform delivering Cohere’s large language models through easy to use API endpoints. The role involves providing technical leadership across multiple teams, driving the architecture and strategy for deploying optimized NLP models to production in low latency, high throughput, and high availability environments. Key responsibilities include:

• Leading the design and implementation of scalable and reliable machine learning systems
• Setting technical direction and helping shape the next generation of AI platforms powering advanced NLP applications
• Providing technical guidance and mentorship to engineers to raise the technical bar across the team
• Serving as a key point of contact for customers, leading the design of customized deployments to meet their specific needs

About the role

The ideal candidate will have 8+ years of engineering experience running production infrastructure at a large scale, with a track record of technical leadership. They should demonstrate experience leading the architecture and design of large, highly available distributed systems with Kubernetes and GPU workloads on those clusters. Extensive experience across GCP, Azure, AWS, OCI, and multi-cloud on-prem / hybrid serving environments, with the ability to guide strategic infrastructure decisions is also required.

Responsibilities

• Provide technical leadership across multiple teams, driving the architecture and strategy for deploying optimized NLP models to production in low latency, high throughput, and high availability environments
• Lead the design, deployment, support, and troubleshooting of complex Linux-based computing environments at scale
• Own compute/storage/network resource and cost management at an organizational level, including optimisation strategies
• Collaborate with cross-functional teams to build mission-critical systems
• Guide the design and implementation of complex technical solutions

Requirements

• Demonstrated experience leading the architecture and design of large, highly available distributed systems with Kubernetes and GPU workloads on those clusters
• Extensive experience across GCP, Azure, AWS, OCI, and multi-cloud on-prem / hybrid serving environments, with the ability to guide strategic infrastructure decisions
• Proven ability to lead the design, deployment, support, and troubleshooting of complex Linux-based computing environments at scale
• Experience owning compute/storage/network resource and cost management at an organizational level, including optimisation strategies
• Exceptional collaboration and communication skills, with experience mentoring engineers and leading cross-functional initiatives to build mission-critical systems
• Strong expertise in the computational characteristics of accelerators (GPUs, TPUs, and/or custom accelerators), and how to leverage them to drive latency and throughput improvements at scale
• Deep knowledge of distributed systems, with experience establishing patterns and practices across engineering teams
• Proficiency in Golang, C++, or other languages designed for high-performance scalable servers, with the ability to set coding standards and conduct senior-level technical reviews

Qualifications

• 8+ years of engineering experience running production infrastructure at a large scale
• Track record of technical leadership
• Deep expertise with Kubernetes dev and production coding and support, including setting team-wide standards and best practices
• Extensive experience across GCP, Azure, AWS, OCI, and multi-cloud on-prem / hybrid serving environments
• Proven ability to lead the design, deployment, support, and troubleshooting of complex Linux-based computing environments at scale
• Experience owning compute/storage/network resource and cost management at an organizational level, including optimisation strategies
• Exceptional collaboration and communication skills, with experience mentoring engineers and leading cross-functional initiatives to build mission-critical systems
• Strong expertise in the computational characteristics of accelerators (GPUs, TPUs, and/or custom accelerators), and how to leverage them to drive latency and throughput improvements at scale
• Deep knowledge of distributed systems, with experience establishing patterns and practices across engineering teams
• Proficiency in Golang, C++, or other languages designed for high-performance scalable servers, with the ability to set coding standards and conduct senior-level technical reviews

Skills

• Technical leadership
• Architecture and design of large, highly available distributed systems
• Kubernetes and GPU workloads
• Multi-cloud environments
• Linux-based computing environments
• Compute/storage/network resource and cost management
• Collaboration and communication skills
• Mentoring engineers
• Designing and implementing complex technical solutions
• Understanding of computational characteristics of accelerators
• Proficiency in Golang, C++, or other languages designed for high-performance scalable servers

Benefits

• Weekly lunch stipend
• Full health and dental benefits
• RSP matching, 401K, Pension Scheme
• Parental leave top-up
• Annual enrichment benefits
• Education & learning stipend
• 6 weeks of paid vacation (30 working days)
• Budget for traveling to other offices if you are remote, plus an annual company offsite