DataBricks Data Engineer

About the role

Prodapt is the largest specialized player in the Connectedness industry. As an AI-first strategic technology partner, Prodapt provides consulting, business reengineering, and managed services for the largest telecom and tech enterprises building networks and digital experiences of tomorrow. A ServiceNow-invested company, Prodapt has been recognized by Gartner as a Large, Telecom-Native, Regional IT Service Provider. A "Great Place To Work® Certified™" company, Prodapt employs over 5,000 technology and domain experts across the Americas, Europe, India, Africa, & Japan. Prodapt is part of the 130-year-old business conglomerate The Jhaver Group, which employs over 32,000 people across 80+ locations globally.

We are seeking an experienced Azure Databricks Engineer to join our team in Dallas, TX. The ideal candidate has strong hands-on experience in designing and building scalable data engineering solutions using Azure Databricks, PySpark, Python, SQL, and Azure Data Services including leading the modernization and migration of existing Python object-oriented applications into scalable PySpark and Spark SQL data-processing solutions on Azure Databricks.

Responsibilities

• Design, develop, and implement scalable data pipelines using Azure Databricks
• Build and optimize ETL/ELT pipelines using PySpark, Python, and SQL
• Develop data solutions using the Medallion Architecture (Bronze, Silver, and Gold layers)
• Work with Delta Lake, Delta tables, and advanced Databricks optimization techniques
• Integrate Databricks with Azure services such as ADLS Gen2, Azure Data Factory, Azure Synapse, and Azure Key Vault
• Develop and manage Databricks Workflows, Jobs, and Notebooks
• Implement data quality, monitoring, error handling, and performance optimization
• Collaborate with Data Architects, Data Engineers, Data Scientists, and business stakeholders
• Establish best practices for CI/CD, Git/version control, code reviews, and automated deployments
• Lead technical discussions and provide guidance to junior engineers
• Analyze existing Python OOP applications and redesign single-node processing logic for distributed Spark execution
• Design, develop, and deploy enterprise-scale data pipelines on Azure Databricks; build reusable PySpark frameworks and utility modules
• Implement Delta Lake solutions using the Bronze, Silver, Gold architecture
• Build robust ETL/ELT pipelines with Azure Data Factory, ADLS Gen2, and Azure Synapse Analytics
• Implement data quality, reconciliation, validation, and monitoring frameworks
• Optimize Spark jobs using partitioning, bucketing, caching, broadcast joins, Adaptive Query Execution, and Delta optimization

Requirements

• 10+ years of overall Data Engineering experience, including designing and implementing ETL/ELT pipelines with Azure Data Factory and other Azure services
• Bachelor's degree in Computer Science, Engineering, or a related field; OR equivalent combination of education and relevant experience
• Strong hands-on experience with Azure Databricks; 4+ years of experience across Azure services and Databricks (ADLS, ADF, Azure DevOps, etc.)
• 7+ years of Python development experience, with the ability to design and build reusable libraries
• 4+ years of experience with Snowflake or SQL (No-SQL experience is a plus)
• Expert-level knowledge of PySpark and Spark SQL
• Strong programming experience in Python and SQL
• Experience with Delta Lake and Lakehouse architecture
• Strong experience with Azure Data Lake Storage (ADLS Gen2)
• Experience with Azure Data Factory (ADF) and other Azure data services
• Strong understanding of ETL/ELT, data modeling, and large-scale data pipelines
• Experience with performance tuning and optimization in Databricks/Spark
• Experience with Git, CI/CD, and DevOps practices

Preferred Skills

• Databricks certifications
• Experience with Unity Catalog and Databricks governance/security
• Experience with Terraform or Infrastructure as Code
• Knowledge of Azure DevOps
• Experience designing APIs and integrating with React JS within a cloud platform