Data Engineering Lead / DevOps Lead

Data Engineering Lead / DevOps Lead – Data Intelligence Platform Enablement Framework

About the role

The Data Engineering Lead / DevOps Lead is responsible for building and operationalizing the core foundation of the Data Intelligence Platform Enablement Framework. This role leads the implementation of governance-as-code, infrastructure-as-code, CI/CD automation, workspace configuration standards, platform monitoring, and observability capabilities to ensure a scalable, secure, and enterprise-ready data platform. The ideal candidate is a hands-on technical leader with deep expertise in Databricks, Azure, DevOps, platform engineering, and automation. This individual will establish engineering standards, automate platform operations, and enable self-service deployment capabilities that accelerate delivery while maintaining governance, reliability, and operational excellence.

Responsibilities

• Platform Engineering & Foundation Build
  • Lead the implementation of the Data Intelligence Platform core infrastructure and enablement framework.
  • Design and build reusable platform services, deployment patterns, and engineering standards.
  • Develop scalable platform capabilities supporting analytics, data products, AI, and self-service development.
  • Establish repeatable engineering practices that support enterprise adoption and growth.
• Governance-as-Code
  • Design and implement governance-as-code frameworks across Databricks and Azure environments.
  • Automate provisioning and enforcement of security, access controls, Unity Catalog policies, data classifications, and governance standards.
  • Partner with Security, Architecture, and Data Governance teams to operationalize governance requirements through automation.
  • Ensure platform controls are consistently deployed across environments.
• Infrastructure-as-Code (IaC)
  • Lead infrastructure automation initiatives using Infrastructure-as-Code methodologies.
  • Develop and maintain Terraform modules and deployment frameworks for Databricks and Azure services.
  • Automate environment provisioning, configuration management, and platform scaling.
  • Standardize deployment patterns to improve consistency, reliability, and speed.
• CI/CD & Release Automation
  • Design and implement enterprise CI/CD pipelines supporting data engineering, analytics, and platform workloads.
  • Establish automated deployment frameworks for code, infrastructure, notebooks, workflows, and platform configurations.
  • Enable version control, automated testing, release management, and deployment governance.
  • Improve developer productivity through automation and self-service platform capabilities.
• Databricks Workspace Administration & Enablement
  • Define workspace architecture, environment standards, and deployment strategies.
  • Manage workspace configuration, cluster policies, compute governance, secret management, and environment isolation.
  • Develop standards for development, testing, and production environments.
  • Ensure platform configurations align with security, governance, and operational requirements.
• Monitoring, Observability & Reliability Engineering
  • Implement platform-wide monitoring, observability, and operational health capabilities.
  • Define and manage logging, alerting, dashboards, performance monitoring, and incident response processes.
  • Establish proactive monitoring for platform health, usage, performance, security, and cost management.
  • Support Site Reliability Engineering (SRE) practices and operational excellence initiatives.
• Technical Leadership & Engineering Excellence
  • Provide technical leadership and mentorship to platform engineers, data engineers, and DevOps resources.
  • Establish engineering standards, deployment patterns, and operational best practices.
  • Lead troubleshooting of complex platform, infrastructure, and automation challenges.
  • Drive continuous improvement through automation, standardization, and platform innovation.

Requirements

• Bachelor's degree in Computer Science, Information Technology, Engineering, or related field.
• 8+ years of experience in Data Engineering, Platform Engineering, DevOps, or Cloud Infrastructure.
• 5+ years of hands-on experience with Azure cloud technologies.
• 3+ years of experience supporting Databricks platform deployments and administration.
• Strong experience with Terraform, Infrastructure-as-Code, CI/CD automation, and DevOps practices.
• Deep understanding of cloud security, identity management, networking, and platform governance.
• Experience implementing monitoring, observability, and operational support frameworks.
• Proven ability to lead technical initiatives and drive engineering best practices.

Preferred Qualifications

• Databricks Certified Data Engineer or Databricks Platform certifications.
• Microsoft Azure DevOps Engineer Expert certification.
• Experience with Unity Catalog, Delta Lake, Azure Data Lake Storage, Azure Key Vault, and Azure Monitor.
• Experience implementing Governance-as-Code and Policy-as-Code frameworks.
• Knowledge of GitHub Actions, Azure DevOps Pipelines, Jenkins, or similar automation platforms.
• Experience supporting enterprise Data Intelligence, Data Product, or AI platforms.

Benefits

• Supplemental benefits including medical, dental, vision, life, and disability insurance plans.
• Employee Stock Purchase Program (ESPP) and 401K with company match after 12 months of tenure.
• Health Savings Account (HSA) on the HDHP plan.
• SupportLinc Employee Assistance Program (EAP) with up to 8 free counseling sessions.
• Corporate discount savings program and other discounts.
• On-demand training program, access to certification prep, and a library of technical and leadership courses/books/seminars after 6+ months of tenure.
• Certification discounts and perks to associations including CompTIA and IIBA.
• Dedicated customer service team for benefits and resources, and a certified Career Coach.