Azure Databricks Data Engineer

About the role

We are looking for a highly motivated Data Engineer with strong expertise in Databricks and modern data engineering practices. The ideal candidate should have hands-on experience in building scalable data pipelines, notebooks, implementing CI/CD processes, and working with Delta Lake technologies. The candidate must possess excellent Python programming skills, a solid understanding of Unity Catalog, and the ability to collaborate effectively within large cross-functional teams.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines and Notebooks using Databricks
• Build and optimize data processing solutions using PySpark and Python
• Implement and manage data governance, security, and access controls using Unity Catalog
• Work with Delta Lake features including Delta Sharing, Time Travel, Change Data Feed (CDF), and data optimization techniques
• Develop and maintain CI/CD pipelines for Databricks deployments using industry-standard DevOps practices
• Perform performance tuning and optimization of Databricks workloads
• Collaborate with architects, business analysts, and stakeholders to understand data requirements and deliver solutions
• Support production deployments, troubleshooting, monitoring, and performance improvements
• Follow coding standards, documentation, and best practices for software development lifecycle (SDLC)

Requirements

• Databricks: Strong understanding of Databricks architecture and ecosystem
• Hands-on experience with Delta Lake, Delta Sharing, Unity Catalog, Structured Streaming, Workflow orchestration, performance tuning, and optimization
• Experience with Medallion Architecture (Bronze, Silver, Gold layers)
• Programming: Strong hands-on coding experience in Python
• Experience with PySpark for large-scale data processing
• Good understanding of SQL and query optimization
• DevOps/CI/CD: Experience implementing CI/CD processes for Databricks workloads
• Familiarity with Git-based source control and branching strategies
• Experience with deployment automation tools and DevOps pipelines
• Data Engineering Concepts: ETL/ELT design patterns, data modeling, data warehousing concepts, data quality monitoring, and observability frameworks

Nice to have

• AWS Cloud: AWS Glue, S3, IAM, Lambda
• Azure Cloud: Azure Data Factory (ADF), Azure Storage, Azure Databricks, Key Vault
• Soft Skills: Excellent verbal and written communication, strong problem-solving and analytical abilities, ability to work effectively in a large global team environment, strong stakeholder management and collaboration, self-driven and proactive with minimal supervision

Benefits

• Comprehensive Medical Plan covering Medical, Dental, Vision
• Short-Term and Long-Term Disability Coverage
• 401(k) Plan with Company match
• Life Insurance
• Vacation Time, Sick Leave, Paid Holidays
• Paid Paternity and Maternity Leave

Pay

The range displayed reflects the minimum and maximum salary target for the position across all US locations. Within the range, individual pay is determined by work location, job level, and additional factors including job-related skills, experience, and relevant education or training. Depending on the position, other forms of compensation may be provided, such as an annual performance-based bonus, sales incentive pay, and other forms of bonus or variable compensation.

Actual compensation within the range will be dependent upon the individual's skills, experience, performance, and internal equity.