Azure Data lead

Key Responsibilities

• Analyze existing Python OOP applications and redesign single-node processing logic for distributed Spark execution.
• Design, develop, and deploy enterprise-scale data pipelines on Azure Databricks; build reusable PySpark frameworks and utility modules.
• Implement Delta Lake solutions using the Bronze Silver Gold architecture.
• Build robust ETL/ELT pipelines with Azure Data Factory, ADLS Gen2, and Azure Synapse Analytics.
• Implement data quality, reconciliation, validation, and monitoring frameworks.
• Optimize Spark jobs (partitioning, bucketing, caching, broadcast joins, Adaptive Query Execution, Delta optimization) and benchmark converted applications against original Python implementations.

Core Skills

• Python (expert)
• OOP and advanced Python design patterns
• PySpark, Spark SQL, and SQL
• Azure Databricks, Azure Data Factory (ADF), MS SQL, Oracle PL/SQL

Experience & Expected Outcome

Senior data engineering leader with proven delivery of large-scale Databricks modernization programs. Expected outcome: existing Python applications converted into scalable, cost-efficient, enterprise-grade data solutions on Azure Databricks with proven performance parity.