Azure Databricks Engineer

Location: Dallas, TX (onsite) – 1-year duration

About the role

We are seeking an Azure Databricks Platform Support Engineer to ensure the reliability, performance, and security of our Databricks environment. The role focuses on day-to-day platform administration, production troubleshooting, and enabling data engineering and analytics teams across the organisation.

Responsibilities

• Administer and support Azure Databricks workspaces, compute, jobs, notebooks, and SQL Warehouses.
• Troubleshoot and resolve production incidents by analysing job logs, cluster events, error messages, and audit information.
• Manage Unity Catalog governance and ADLS Gen2 integration, including permissions and access controls.
• Configure and maintain compute policies, instance pools, autoscaling, auto-termination, and cost controls.
• Support workspace networking and connectivity, including VNet injection, private endpoints, and Private Link.
• Maintain operational documentation, runbooks, and change records for the platform.

Requirements

• Hands-on experience with Azure Databricks platform support or administration.
• Good understanding of Databricks workspaces, compute, jobs, notebooks, SQL Warehouses, libraries, and permissions.
• Experience troubleshooting Databricks production issues.
• Working knowledge of Unity Catalog and ADLS Gen2 integration.
• Understanding of Azure networking basics: VNet, subnet, NSG, route table, NAT Gateway, firewall, private endpoint, Private Link, and DNS.
• Familiarity with compute policies, instance pools, runtime versions, autoscaling, auto-termination, and cost controls.
• Ability to read job logs, cluster events, error messages, and audit information.

Preferred Skills

• Strong communication and documentation skills.
• Experience with VNet-injected Databricks workspaces and secure cluster connectivity.
• Experience with Azure Key Vault, Azure Monitor, Log Analytics, and diagnostic logs.
• Knowledge of PySpark, Spark SQL, Delta Lake, and Medallion Architecture.
• Experience with Databricks CLI, REST APIs, Terraform, Bicep, Azure DevOps, or GitHub.
• ITIL-style Incident, Problem, and Change Management experience.

Tools & Technologies

• Azure Databricks
• Unity Catalog
• Delta Lake
• Apache Spark / PySpark / Spark SQL
• ADLS Gen2
• Azure Key Vault
• Azure Monitor / Log Analytics
• Azure Networking
• Databricks CLI / REST APIs
• Azure DevOps / GitHub