SENIOR AZURE DATABRICKS ENGINEER

About the role

Prepare the Databricks Serverless environment for the team, set up job clusters, and ensure all needed libraries are present. Work with Databricks and CDO to ensure all AI capabilities are enabled, and migrate existing Databricks jobs and workflows.

Responsibilities

• Resolve Azure networking, proxy, firewall, and authentication issues in Databricks and web applications
• Optimize performance, cost, and scalability of Databricks workloads
• Enhance monitoring, alerting, and job reliability improvements by creating a UI that tracks all jobs start, completion, duration, and status
• Maintain and enhance DataSphere health check framework
• Support Unity Catalog, access controls, and environment governance
• Troubleshoot production issues and failed jobs
• Create Genie Spaces for each Domain as the jobs are migrated

Requirements

• 10+ years overall experience
• Strong Databricks experience, including workflows, jobs, clusters, serverless, and performance optimization
• Azure cloud experience, including networking, proxy, firewall, authentication, and application deployments
• Python, SQL, ETL development, and migration experience with Databricks and Snowflake
• Production data pipeline support, monitoring, alerting, and job reliability
• Unity Catalog, access controls, governance, and data quality frameworks

Skills

• At least one contractor must have React, TypeScript/JavaScript, API integration, and custom Azure web app development experience
• Preferred: Databricks AI capabilities, AI/LLM applications, health check automation, dashboards, SSO/OAuth/Entra ID, and operational UI development

Schedule

Onsite 5 days per week.

Interview

3 rounds, with the final round face to face.