Senior Databricks Data Engineer

About the role

This is a contract remote position for a highly skilled Data Engineer with extensive experience in Databricks technologies.

Responsibilities

• Build and manage Delta Tables in Databricks for efficient data storage and processing.
• Develop Declarative Pipelines for comprehensive data workflows using PySpark.
• Deploy and automate data solutions using Databricks Asset Bundles (DAB).
• Implement Unity Catalog for robust data governance and metadata management.
• Manage Databricks security by setting up permission policies and access controls.
• Design and enforce cluster policies and governance best practices.
• Write Python code that adheres to Clean Code principles for scalability and efficiency.
• Apply problem-solving skills to design, develop, and optimize scalable data engineering solutions.

Requirements

• 5+ years of hands-on experience with Databricks Data Engineering.
• Strong expertise in PySpark for large-scale data processing.
• Experience with Unity Catalog for data governance and metadata management.
• Experience with Databricks security, including permission management and access controls.
• Deep knowledge of cluster policies and governance best practices.
• Excellent Python programming skills for building scalable and efficient data solutions.
• Strong understanding of Object-Oriented Programming (OOP) principles.
• Experience applying Clean Code practices to develop maintainable and high-quality codebases.
• Strong problem-solving skills with the ability to design, develop, and optimize scalable data engineering solutions.