Senior Data Engineer

About the Role

As an experienced Senior Data Engineer, you will work in a collaborative environment under Deloitte’s Project Delivery Talent Model (PDM), tailored for long-term, onsite client service delivery with minimal travel demands. This role allows you to share new ideas and collaborate on projects as a consultant.

Responsibilities

• Communicate regularly with Engagement Managers (Directors), project team members, and representatives from various functional and technical teams, escalating matters requiring additional attention.
• Design, develop, and optimize ETL/ELT pipelines using Azure Data Factory (ADF) and Databricks.
• Write and tune PySpark/Spark SQL notebooks for large-scale data transformation.
• Architect end-to-end data solutions across dev → UAT → prod environments using Unity Catalog.
• Lead and drive design discussions with client architects and other counterparts.
• Collaborate with different teams on data contracts and schema agreements.
• Lead design and optimization of high-volume data pipelines.
• Define and enforce data engineering standards, including naming conventions, partitioning strategies, cluster configurations, and Spark tuning.
• Drive performance optimization through AQE tuning, liquid clustering, broadcast joins, and shuffle partition management.
• Design Databricks cluster policies, autoscaling configurations, and cost optimization strategies.
• Conduct root cause analysis on production incidents and implement permanent fixes.
• Mentor junior and mid-level engineers through code reviews and pair programming.
• Evaluate new technologies and recommend adoption (e.g., DABs, DLT, Auto Loader, Serverless Compute, event hubs).

About the Team

AI & Engineering leverages cutting-edge engineering capabilities to build, deploy, and operate integrated sector solutions in software, data, AI, network, and hybrid cloud infrastructure. These solutions transform mission-critical operations and enable clients to stay ahead with the latest advancements by modernizing technology and data platforms. Delivery models are tailored to meet each client’s unique requirements.

Requirements

• Python, PySpark, Spark SQL, SQL Server
• Azure (ADF, ADLS Gen2, Key Vault, Azure Monitor)
• Databricks (Delta Lake, Unity Catalog, Workflows)
• Apache Airflow
• Git / Azure DevOps
• Deep Spark internals (DAG optimization, spill analysis, skew handling)
• Delta Lake advanced features (time travel, deletion vectors, predictive I/O)
• Unity Catalog governance (row/column security, external locations, system tables)
• IaC - Terraform, Azure ARM templates
• Bachelor’s degree, preferably in Computer Science, Information Technology, Computer Engineering, or related IT discipline; or equivalent experience
• Limited immigration sponsorship may be available
• Ability to travel 10%, on average, based on work and client needs

Pay

The wage range for this role is $95,000 to $150,000, taking into account factors such as skill sets, experience, training, licensure, certifications, and business needs. Compensation decisions depend on individual circumstances.

Benefits

This position is aligned with the Project Delivery Talent Model. To view the associated benefit package, please reference this document.