Senior Data Engineer
About the Role
As an experienced Senior Data Engineer, you will work in a collaborative environment under Deloitte’s Project Delivery Talent Model (PDM), tailored for long-term, onsite client service delivery with minimal travel demands. This role allows you to share new ideas and collaborate on projects as a consultant.
Responsibilities
- Communicate regularly with Engagement Managers (Directors), project team members, and representatives from various functional and/or technical teams, escalating matters requiring additional attention.
- Design, develop, and optimize ETL/ELT pipelines using Azure Data Factory (ADF) and Databricks.
- Write and tune PySpark/Spark SQL notebooks for large-scale data transformation.
- Architect end-to-end data solutions across dev → UAT → prod environments using Unity Catalog.
- Lead and drive design discussions with client architects and other counterparts.
- Collaborate with different teams on data contracts and schema agreements.
- Lead design and optimization of high-volume data pipelines.
- Define and enforce data engineering standards, including naming conventions, partitioning strategies, cluster configurations, and Spark tuning.
- Drive performance optimization through AQE tuning, liquid clustering, broadcast joins, and shuffle partition management.
- Design Databricks cluster policies, autoscaling configurations, and cost optimization strategies.
- Conduct root cause analysis on production incidents and implement permanent fixes.
- Mentor junior and mid-level engineers through code reviews and pair programming.
- Evaluate new technologies and recommend adoption (e.g., DABs, DLT, Auto Loader, Serverless Compute, event hubs).
About the Team
AI & Engineering leverages cutting-edge engineering capabilities to build, deploy, and operate integrated sector solutions in software, data, AI, network, and hybrid cloud infrastructure. These solutions transform mission-critical operations and enable clients to stay ahead with the latest advancements by modernizing technology and data platforms. Delivery models are tailored to meet each client’s unique requirements.
Requirements
- Python, PySpark, Spark SQL, SQL Server
- Azure (ADF, ADLS Gen2, Key Vault, Azure Monitor)
- Databricks (Delta Lake, Unity Catalog, Workflows)
- Apache Airflow
- Git / Azure DevOps
- Deep Spark internals (DAG optimization, spill analysis, skew handling)
- Delta Lake advanced features (time travel, deletion vectors, predictive I/O)
- Unity Catalog governance (row/column security, external locations, system tables)
- Infrastructure as Code (IaC) - Terraform, Azure ARM templates
- Bachelor’s degree, preferably in Computer Science, Information Technology, Computer Engineering, or related IT discipline; or equivalent experience
- Limited immigration sponsorship may be available
- Ability to travel 10%, on average, based on work and client needs
Pay
The wage range for this role is $95,000 to $150,000, taking into account factors such as skill sets, experience, training, licensure, certifications, and business needs. Compensation decisions depend on individual circumstances.
Benefits
This position is aligned with the Project/Center benefits package. For details, please reference this document.