Azure Databricks Lead Engineer

About the role

We’re seeking an experienced Azure Databricks Lead Engineer to help shape and manage a modern enterprise data platform. This role is ideal for a hands-on data engineer who enjoys building scalable solutions directly in Databricks, leading migration efforts, and driving automation across data workflows.

Responsibilities

• Design, build, and optimize end-to-end data pipelines within Azure Databricks and Delta Lake.
• Develop solutions using PySpark, SQL, or Scala to support core data transformations and platform engineering.
• Build and maintain production data workflows using Databricks Workflows and Jobs.
• Lead migration efforts from legacy data platforms into Databricks-based solutions.
• Refactor and convert existing business logic and ETL processes into scalable, code-driven Databricks notebooks.
• Implement data governance practices using Unity Catalog, including access, security, schema management, and lineage.
• Support CI/CD and DevOps automation for Databricks workspace deployments and jobs using Azure DevOps and YAML-based pipelines.
• Collaborate with stakeholders to ensure data solutions are reliable, maintainable, and aligned with business needs.

Skills

• Strong hands-on experience with Azure Databricks in an enterprise environment.
• Deep knowledge of Delta Lake and modern data engineering practices.
• Proficiency in PySpark, SQL, and/or Scala.
• Experience with Unity Catalog and Databricks governance capabilities.
• Working knowledge of CI/CD practices and DevOps automation.
• Experience building or supporting data platform migrations, especially from legacy systems.
• Ability to translate business and ETL logic into clear, maintainable code.
• Strong communication and collaboration skills.
• Understanding of Azure fundamentals.

Preferred Skills

• Databricks certifications, such as Data Engineer Associate or Professional.
• Microsoft Azure Data Engineer Associate (DP-203) certification.
• Microsoft Azure Fundamentals (AZ-900) or equivalent foundational Azure knowledge.
• Experience converting Alteryx workflows into PySpark or SQL solutions.
• Background supporting migration initiatives from Synapse into Databricks.
• Experience working in hybrid team environments and coordinating across multiple stakeholders.