Sr. Azure Data Engineer

About The Role

Job Number 1
Candidate should be hands-on with Azure Databricks and Unity Catalog.

Key Responsibilities

• Design, develop, and optimize ETL/ELT pipelines using Azure Databricks (PySpark).
• Build scalable data ingestion workflows from various structured and unstructured sources.
• Implement transformation logic, data cleansing, enrichment, and validation frameworks.
• Work with Delta Lake to build medallion architecture (Bronze/Silver/Gold layers).
• Develop reusable Databricks notebooks and jobs for production data workflows.
• Build and orchestrate pipelines using Azure Data Factory (ADF).
• Integrate Databricks with other Azure services—ADLS, Azure SQL, Event Hub, Key Vault, Synapse.
• Optimize compute environments (clusters, pools, autoscaling).
• Implement DevOps processes using Git, CICD, Azure DevOps.
• Optimize PySpark jobs for performance and cost efficiency.
• Implement best practices for data governance, security, and access control.
• Troubleshoot production issues and perform root-cause analysis.
• Conduct code reviews ensuring coding standards and data quality.
• Collaborate with Data Architects to define architecture and design patterns.
• Prepare technical documents, solution diagrams, and runbooks.
• Collaborate with business stakeholders to understand requirements and translate them into technical solutions.

Mandatory Skills

• Azure Databricks – notebooks, jobs, workflows, Delta Lake.
• PySpark – dataframes, Spark SQL, optimization & debugging.
• Azure Data Factory (ADF) – triggers, pipelines, integration runtime.
• Data Lake Storage (ADLS Gen2) – folder structures, partitioning, security.
• CI/CD – Git (branching strategies), Azure DevOps pipelines.
• SQL – strong proficiency in writing optimized queries.

Good-to-Have Skills

• Azure Synapse Analytics.
• Azure Event Hub / Kafka.
• Azure Functions.
• Streaming pipelines (Structured Streaming).
• Experience with data modelling.
• Knowledge of Lakehouse architecture.

Behavioral & Soft Skills

• Strong analytical and problem-solving skills.
• Ability to work independently and in cross-functional teams.
• Good communication skills for stakeholder interaction.
• Comfortable working in Agile/Scrum models.

Technical Skills

• Azure Databricks / Delta Lake / PySpark.
• Azure Data Factory / Azure Synapse Analytics.
• Data Lake Storage (ADLS Gen2) – Folder Structures, Partitioning, Security.
• CI/CD – Git (Branching Strategies), Azure DevOps Pipelines.
• SQL / Azure SQL Database.
• Event Hub / Kafka.
• Azure Functions.
• Structured Streaming.
• Lakehouse Architecture.

Education

• Bachelor's Degree in Computer Science, Software Engineering, Information Systems, Data Science, Electrical Engineering, Computer Engineering.
• Preferred: Master's in Data Science, Master's in Computer Science, Master's in Information Systems, MBA (Technology Management).

Industry Experience

• Banking / Financial Services.
• Cloud Data Engineering.
• Enterprise Analytics.
• Azure Cloud Platform.
• Data Warehouse & Lakehouse Solutions.