Principal AI Research Scientist, Research Director - AI Scaling

About the role

The Databricks AI Scaling team focuses on pushing the boundaries of large language model (LLM) training and inference efficiency beyond what is required to support existing models. The team explores novel avenues for scaling and efficiency improvements across algorithms, systems, and infrastructure, requiring researchers who can both drive independent research agendas and dive deep into low-level implementation details with engineering partners.

Responsibilities

• Lead and grow a multidisciplinary research team focused on foundational and applied AI problems, with a particular emphasis on LLM scaling, efficiency, and systems performance.
• Define the scaling research roadmap in alignment with Databricks' strategic objectives, prioritizing advances in foundation model efficiency and large-scale training and inference.
• Drive algorithmic innovations for large-scale neural network training and inference, including novel optimizers, low-precision techniques, and model adaptation methods, and guide your team in rigorous empirical validation against state-of-the-art approaches.
• Optimize end-to-end ML systems for distributed training and RL, memory efficiency, and compute efficiency through close collaboration with core systems and platform teams, ensuring that research ideas translate into performant, reliable infrastructure.
• Partner with product and engineering to translate research breakthroughs, especially around scaling and efficiency, into customer-im impacting capabilities in the Databricks AI platform.
• Foster a culture of scientific excellence and openness, including high-quality research practices, reproducible experimentation, and effective internal knowledge sharing across Databricks AI.
• Represent Databricks AI research externally through top-tier publications, conference talks, and collaborations with academia and the open-source community, with a focus on optimization and efficiency for large-scale models.
• Mentor and develop talent, providing both technical guidance (research agendas, experimentation, implementation) and career development support for research scientists and engineers.

Qualifications

• Proven ability to lead a research team to develop novel techniques for foundation model efficiency and related topics, with a strong track record of industry impact.
• Deep expertise in at least one of: generative AI, LLMs, distributed ML systems, model optimization, or responsible AI, with a strong emphasis on scaling and efficiency for large-scale neural networks.
• Hands-on leadership - strong programming skills and demonstrated ability to write high-quality, efficient code in Python and PyTorch for research implementation and experimentation.
• Demonstrated ability to translate research innovation into scalable product capabilities in partnership with product and engineering teams.
• Excellent communication, leadership, and stakeholder management skills, with experience influencing cross-functional roadmaps and aligning research with business impact.

Benefits

At Databricks, we strive to provide comprehensive benefits and perks that meet the needs of all of our employees. For specific details on the benefits offered in your region click here.