Senior Databricks Administrator

About the role

We are seeking a Senior Databricks Administrator with 15+ years of overall IT experience and 5+ years of hands-on Databricks Administration experience to lead the administration, governance, security, performance, and cost optimization of an AWS Databricks platform. This is a platform administration and governance role, not a development position.

Responsibilities

• Ensure platform stability, security, compliance, performance, and operational excellence.
• Manage enterprise-scale Databricks environments on AWS, including Unity Catalog Administration and Governance, RBAC, SSO, SCIM, and Security Administration.
• Administer Databricks Workspaces, including Workspace Provisioning and Environment Governance.
• Administer Databricks Clusters, including Cluster Policies and Compute Governance, User and Access Management, and Platform Upgrades and Runtime Management.
• Manage multi-workspace administration, including Hive Metastore and Hive Tables Administration.
• Optimize Databricks cost management, pricing, and financial operations.
• Monitor, audit, and ensure compliance with platform policies and procedures.
• Support engineering teams by troubleshooting and analyzing job failures, performing root-cause analysis, and providing guidance on data engineering workflows.

Requirements

• 15+ years of overall IT experience with 5+ years in Databricks Administration.
• Strong hands-on expertise in Databricks Workspace Administration, Unity Catalog Administration and Governance, RBAC, SSO, SCIM, and Security Administration.
• Experience with Databricks Cluster Administration (Interactive, Job Clusters, SQL Warehouses, Serverless).
• Expertise in Databricks Cost Management, Pricing, and Financial Operations Optimization.
• Knowledge of Databricks Platform Monitoring, Auditing, and Compliance.
• Experience with AWS, including VPC, Security Groups, and Networking, Secrets Management, and Cloud Governance and Security Best Practices.
• Experience managing enterprise-scale AWS-based Data & Analytics environments.
• Technical understanding of Databricks Lakehouse Architecture, Apache Spark, Delta Lake, Hive Metastore, and Hive Tables.

Qualifications

• Hands-on Databricks Administrator, not a Databricks Developer.
• Expert in Unity Catalog, Security, RBAC, Governance, and Cost Optimization.
• Strong experience managing clusters, SQL Warehouses, and workspace administration.
• Deep knowledge of Hive Metastore and Hive tables.
• Able to independently own and govern the Databricks platform on AWS.
• Capable of troubleshooting and debugging platform and workload issues while working closely with Data Engineering, Cloud, Security, and Infrastructure teams.

Preferred Skills

• Experience with Hive Metastore to Unity Catalog migrations.
• Strong understanding of Databricks pricing models and consumption optimization.
• Exposure to Databricks Genie AI / GenAI capabilities and their governance within Unity Catalog.
• Experience implementing enterprise security, governance, and compliance controls.
• Databricks certifications are a plus.