Lead Data Engineer

Position Overview

We are seeking a Lead Data Engineer to join a rapidly growing Data & Analytics organization responsible for delivering enterprise-scale data products, analytics solutions, and cloud-based data platforms. This role will lead the design, development, and optimization of modern data architectures and reusable data pipelines supporting critical business initiatives across multiple lines of business.

Key Responsibilities

• Data Architecture & Engineering
• Design and implement end-to-end data architectures from source systems through analytics and reporting consumption layers.
• Architect, develop, and optimize cloud-based data products and reusable data pipelines.
• Build scalable ETL/ELT frameworks using Azure Databricks and modern big data technologies.
• Design and maintain enterprise data warehouses, data lakes, and cloud-native analytics platforms.
• Develop high-performance data ingestion, transformation, and curation processes for structured and unstructured data.
• Technical Leadership
• Lead and mentor a team of data engineers while establishing development standards and best practices.
• Drive adoption of reusable frameworks and automation to improve efficiency and scalability.
• Partner with architecture, engineering, BI, and data science teams to deliver enterprise data solutions.
• Collaborate with stakeholders to translate business requirements into technical designs and implementation plans.
• Cloud & Platform Optimization
• Leverage Azure Databricks, Azure Data Factory, Azure Functions, Cosmos DB, and related cloud services to deliver scalable solutions.
• Optimize Spark workloads, ETL processes, and cloud infrastructure for performance and cost efficiency.
• Implement and enhance CI/CD pipelines, deployment automation, and DevOps best practices.
• Support data governance, data quality, and enterprise data management initiatives.
• Analytics & Stakeholder Engagement
• Analyze complex datasets using SQL and exploratory tools to identify trends, anomalies, and opportunities.
• Track and report on delivery KPIs, platform performance, and data quality metrics.
• Present technical solutions, business impact, and recommendations to leadership and business stakeholders.

Required Qualifications

• Bachelor's degree in Computer Science, Information Systems, Engineering, or a related field.
• 8+ years of data engineering, data solutions, or analytics platform development experience.
• 4+ years of hands-on experience building solutions within Azure and Databricks environments.
• Strong experience designing and deploying enterprise-scale cloud data platforms.
• Advanced proficiency with SQL, Spark, Python, and/or Scala.
• Hands-on experience with Azure Databricks, Azure Data Factory, Azure Functions, Cosmos DB, Delta Lake, and big data technologies.
• Experience building cloud-based data warehouses, data lakes, and analytics platforms supporting both batch and real-time workloads.
• Strong expertise in performance tuning, optimization, and troubleshooting large-scale data processing environments.
• Experience implementing CI/CD pipelines and modern software development practices.
• Excellent communication, leadership, and stakeholder management skills.

Preferred Qualifications

• Databricks and/or Microsoft Azure certifications.
• Experience with Event Hub and real-time streaming architectures.
• Expertise developing Azure Functions using Python or Node.js.
• Experience with Hive, HBase, partitioning, bucketing, and large-scale data optimization.
• Experience implementing enterprise data governance and data quality frameworks.
• Experience working within large enterprise environments supporting analytics, AI, and data science initiatives.

Ideal Candidate Profile

The ideal candidate combines deep cloud data engineering expertise with strong leadership capabilities. They have successfully built and optimized enterprise-scale Azure data platforms, excel at mentoring teams, and can bridge technical and business requirements while delivering high-quality, scalable data products.