Sr. Engineering Manager - Pipelines Engine

We are the Lakeflow Engineering team at Databricks, responsible for the Databricks ETL product line: Jobs, Spark Declarative Pipelines, and Genie Code for Data Engineering. We run one of the world's largest data engineering platforms, processing exabytes of data daily for tens of thousands of customers.

About the role

We're seeking a dedicated Engineering Leader to spearhead the Pipelines Engine engineering team. This team builds next-generation Runtime ETL features and ensures Lakeflow Pipelines delivers state-of-the-art performance for ETL workloads. You will also lead the development of Agentic Data Engineering infrastructure, creating the next-generation engine to power agentic pipeline authoring, execution, and maintenance.

Responsibilities

• Lead an engineering team building next-generation ETL features for the Databricks Lakeflow platform.
• Oversee sustained recruitment of top-tier talent and upskill the existing team.
• Build processes to implement product vision and strategy aligned with organizational goals and priorities.
• Develop software that is high quality, easy to operate, and reliable.
• Manage technical debt, including long-term technical architecture decisions, while balancing the product roadmap.
• Ensure the team builds high-quality, reliable infrastructure services with a focus on testing, quality, and Service Level Agreements.

Requirements

• Minimum 3 years of experience managing top-tier engineering teams.
• 5+ years of experience building data infrastructure systems such as Apache Spark™ or database internals.
• Passion for database systems, storage systems, distributed systems, or performance optimization.
• Experience working with product management and directly with customers to understand their needs.
• Proven ability to attract, hire, and coach engineers to meet Databricks hiring standards.
• Experience up-leveling teams through hiring top talent and growing existing team members.
• Background in complex technical domains, such as distributed data systems or database internals.

Benefits

At Databricks, we strive to provide comprehensive benefits and perks that meet the needs of all our employees. For specific details on the benefits offered in your region, visit our benefits page.

Pay

Databricks is committed to fair and equitable compensation practices. The pay range for this role is $228,600—$317,233 USD. Actual compensation is based on factors such as skills, experience, relevant certifications, training, and work location. The total compensation package may also include eligibility for an annual performance bonus, equity, and benefits.

About Databricks

Databricks is the data and AI company. More than 10,000 organizations worldwide—including Comcast, Condé Nast, Grammarly, and over 50% of the Fortune 500—rely on the Databricks Data Intelligence Platform to unify and democratize data, analytics, and AI. Founded by the original creators of Lakehouse, Apache Spark™, Delta Lake, and MLflow, Databricks is headquartered in San Francisco with offices around the globe.