Azure Databricks Admin

About the role

Cognizant is seeking a highly skilled Azure Databricks Administrator to join our Cloud, Data & AI practice. This role is responsible for the administration, governance, security, optimization, and operational support of enterprise-scale Azure Databricks environments.

Responsibilities

• Azure Databricks Administration & Optimization
• Design, deploy, and administer Azure Databricks workspaces utilizing VNet injection, private endpoints, and secure cluster connectivity.
• Configure and maintain cluster policies, Databricks Runtime versions, autoscaling, and workload management standards.
• Optimize platform performance through cluster tuning, Spark configuration management, and Delta Lake best practices.
• Perform capacity planning and resource forecasting to support enterprise growth.
• Drive cost optimization initiatives including right-sizing, auto-termination policies, reserved instances, and spot instance utilization.
• Troubleshoot and resolve complex platform, cluster, job execution, and connectivity issues.
• Data Governance & Unity Catalog
• Design, implement, and administer Unity Catalog metastores, catalogs, schemas, external locations, and volumes.
• Establish and enforce fine-grained access controls across enterprise data assets.
• Support data lineage tracking, auditing, and governance requirements.
• Govern managed and external data assets while promoting best practices for data management.
• Security & Identity Management
• Integrate Azure Databricks with Microsoft Entra ID for authentication and Single Sign-On (SSO).
• Configure and manage SCIM provisioning for automated user and group lifecycle management.
• Implement secure secret management using Azure Key Vault-backed secret scopes.
• Define and manage access controls across Unity Catalog, Databricks workspaces, and Azure Data Lake Storage Gen2.
• Support security audits, compliance reviews, and governance initiatives.
• Networking & Connectivity
• Architect and maintain secure networking solutions utilizing VNets, NSGs, private endpoints, and firewall controls.
• Ensure secure connectivity between Databricks, Azure services, and on-premises environments.
• Manage DNS resolution, routing configurations, and network security best practices.
• Data Integration & Automation
• Design and implement Azure Data Factory (ADF) integration patterns with Azure Databricks.
• Support orchestration of enterprise data pipelines and notebook execution workflows.
• Partner with DevOps teams to establish and maintain CI/CD pipelines.
• Utilize Databricks REST APIs and automation frameworks to streamline platform administration.
• Storage Management
• Provide guidance on Azure Data Lake Storage Gen2 architecture and configuration.
• Implement secure access patterns using service principals, credential passthrough, and Microsoft Entra ID integration.
• Troubleshoot storage connectivity and access management issues.
• Operations & Platform Reliability
• Implement monitoring and alerting using Azure Monitor, Log Analytics, and Databricks native monitoring tools.
• Maintain service availability, platform health, resource utilization, and job performance.
• Support disaster recovery, business continuity, and resiliency planning efforts.
• Plan and coordinate Databricks Runtime upgrades and platform maintenance activities.
• Communicate service impacts and operational risks to stakeholders.
• Documentation & Enablement
• Develop and maintain architecture documentation, diagrams, standards, and operational runbooks.
• Establish best practices for platform governance, security, cost optimization, and performance management.
• Provide technical guidance, training, and knowledge transfer to internal teams.
• Stakeholder Engagement
• Collaborate with data engineering, data science, infrastructure, networking, security, and operations teams.
• Act as the subject matter expert for Databricks platform administration and governance.
• Participate in architecture reviews and provide technical leadership for strategic initiatives.

Requirements

• Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field, or equivalent work experience.
• 5+ years of experience supporting cloud-based data and analytics platforms.
• Hands-on experience administering Azure Databricks in enterprise environments.
• Strong experience with:
• Azure Databricks
• Azure Data Factory (ADF)
• Azure Data Lake Storage Gen2 (ADLS Gen2)
• Azure Key Vault
• Azure Networking (VNets, NSGs, Private Endpoints)
• Azure Monitor and Log Analytics
• Experience implementing and managing Unity Catalog.
• Strong understanding of cloud security, RBAC, IAM, encryption, and governance principles.
• Experience with SCIM-based identity provisioning.
• Proficiency in Python, SQL, PowerShell, and Azure CLI.
• Experience supporting CI/CD initiatives using Azure DevOps, GitHub Actions, or similar platforms.
• Excellent analytical, troubleshooting, and problem-solving skills.
• Strong verbal and written communication skills.

Preferred Qualifications

• Experience supporting large-scale enterprise Data & AI platforms.
• Knowledge of Spark architecture, Delta Lake optimization, and Databricks APIs.
• Experience with Infrastructure as Code (IaC) tools and cloud automation frameworks.
• Experience working within regulated or highly governed environments.
• Microsoft Certified: Azure Solutions Architect Expert.
• Databricks Certified Data Engineer Associate or Professional certification.