Azure Databricks Data Engineer

Temporary full-time role based in Atlanta, GA (hybrid: 3 days/week in office). Estimated duration: 16 months. Hourly rate: $50.00–$55.00.

About the Role

Support and evolve the Azure Databricks Lakehouse architecture of an established Data Warehouse team—setting design standards, governing the platform, unlocking full Databricks/Databricks AI capabilities, and optimizing solutions for performance, cost, security, and scalability.

Responsibilities

• Act as the technical design authority for an existing Databricks ETL team, collaborating with business, development, Databricks, and DevOps teams.
• Support and evolve the production Lakehouse architecture: Delta medallion layers, Databricks SQL Warehouses, Unity Catalog security, and data lineage.
• Define and enforce design standards, reusable patterns, and best practices across the existing pipeline estate and all new development.
• Design new ingestion, streaming, and batch pipelines; review team designs against the medallion (bronze/silver/gold) architecture.
• Drive continuous optimization of the running platform: cluster sizing, autoscaling, Photon usage, query performance, Delta Liquid Clustering/Z-Ordering, VACUUM strategies, and DBU cost governance.
• Translate business questions into analytical models, KPIs, and reusable semantic layers.
• Mature CI/CD (Azure DevOps/GitHub), observability, and security (RBAC, row/column-level policies) across existing pipelines.
• Evaluate and drive adoption of new Databricks capabilities (Lakeflow, Delta Live Tables, serverless compute, AI/BI) where they add value.
• Conduct architectural reviews, code audits, and mentoring sessions to raise the team’s engineering bar and ensure adherence to standards and scalability.
• Guide the team in version control, automated testing, and release discipline for Databricks notebooks, jobs, and workflows.

Requirements

• Bachelor’s degree in Computer Science, Information Technology, or related field.
• 5+ years designing, implementing, and optimizing Azure Databricks Lakehouse solutions, including ETL pipelines.
• 5+ years Azure data engineering experience using ADLS Gen2, Databricks SQL, Python, Event Hub, Key Vault, and CI/CD.
• 5+ years data architecture & data modeling, including dimensional modeling, ETL design, and code reviews.
• Strong hands-on knowledge of Azure Databricks and the Azure platform, with proven experience operating a production Lakehouse at scale.
• Proficiency in Databricks SQL, Python, and other relevant programming languages.
• Deep Azure knowledge: ADLS Gen2, Key Vault, Networking (VNet/Private Endpoints), Event Hub.
• Deep understanding of advanced data warehouse concepts (dimensional modeling, SCD, surrogate key management).
• Experience with Unity Catalog security & lineage.
• Strong business problem framing and stakeholder communication skills.
• Experience establishing engineering standards, performing design/code reviews, and mentoring data engineers.
• Excellent problem-solving and communication skills.
• Ability to work independently and collaboratively in a fast-paced environment.

Preferred Qualifications

• Databricks and Azure certifications.

Skills

• Azure
• Databricks
• ETL
• Python
• Snowflake
• SQL