Sr Databricks Data Engineer

About the role

Join Deloitte's AI & Engineering practice and help organizations transform enterprise technology platforms, modernize data environments, and unlock value through innovation. As a Databricks Engineer in our AI & Data practice, you will design, build, and optimize cloud-based data engineering solutions that support large-scale transformation.

Responsibilities

• Champion Best Practices: Establish, document, and promote best-in-class approaches for data architecture, integration, and modelling.
• Pipeline Ownership: Oversee the design, development, and maintenance of robust data pipelines and data architectures that support large-scale, enterprise data needs.
• Drive Excellence: Initiate and manage efforts to improve data quality, operational efficiency, and process scalability.
• Team and Technology Lead: Evaluate, pilot, and integrate new big data and analytics technologies, ensuring the organization remains at the cutting edge.
• Lead, coach, and develop teams of data engineers and architects, fostering technical growth and effective project delivery.
• Data Governance: Consult on, design, and implement governance, security, and compliance strategies tailored to modern cloud data ecosystems.
• Communication: Communicate technical concepts and business value to diverse stakeholders, including executives, business leads, and technology teams.
• DevOps and Automation: Oversee the implementation of CI/CD practices with tools such as Azure DevOps, AWS Code Pipeline, Jenkins, TFS, or PowerShell for streamlined deployments and operations.

Qualifications

• Bachelor's degree in Computer Science, Engineering, or a related field
• 5+ years of hands-on experience in data engineering with a focus on Databricks on Amazon Web Services (AWS), Microsoft Azure, or Google Cloud Platform (GCP)
• Experience with Lakehouse architecture, Apache Spark, Delta Lake, cloud-native databases, storage solutions, and distributed compute platforms
• Experience with data warehousing, third normal form (3NF), dimensional modeling, enterprise data lakes, incremental data loads, and metadata-driven ingestion and data quality frameworks using PySpark
• Experience leading complex, cross-functional data projects and technical teams, including experience with Delta Live Tables, Autoloader, Structured Streaming, Databricks Workflows, Apache Airflow, Unity Catalog, automated continuous integration and continuous deployment (CI/CD) pipelines, and performance optimization of data engineering pipelines, code, and compute resources