Databricks Developer

Responsibilities

• Lead the design and development of end-to-end data pipelines on Azure, leveraging Databricks, Azure Data Lake, and ADLS Gen 2.
• Implement scalable ETL/ELT pipelines for structured, semi-structured, APIs, and unstructured data.
• Collaborate with Data Scientists, Business Analysts, and other stakeholders to translate business requirements into robust data solutions.
• Optimize data workflows for performance, reliability, and cost-efficiency in cloud environments.
• Establish and enforce data quality, governance, and validation standards.
• Provide guidance on data modeling, schema design, and metadata management to support analytics and AI workloads.
• Monitor and troubleshoot production data pipelines and recommend performance improvements.
• Evaluate emerging Azure and Databricks technologies and propose enhancements to our data architecture.

Requirements

• Proven experience leading data engineering teams and managing large-scale data pipelines.
• Expertise in Azure Databricks, Azure Data Lake, and ADLS Gen 2.
• Strong understanding of data warehousing, data modeling, and SQL.
• Experience with ETL/ELT tools and technologies.
• Knowledge of cloud-based data architectures and best practices.
• Ability to communicate complex technical concepts to non-technical stakeholders.
• Experience with data quality, governance, and validation standards.
• Proficiency in Python, Scala, or similar programming languages.