Senior Platform, DevOps Engineer

About the role

The Senior Platform / DevOps Engineer plays a critical role in designing, automating, and operating enterprise-grade Databricks platforms in support of data and AI initiatives. This role partners closely with engineering, data, and cloud teams to ensure scalable, secure, and cost-optimized platform services. The engineer operates with a high degree of autonomy, contributing deep technical expertise while supporting mission-critical workloads.

Responsibilities

• Automate and manage Databricks workspaces, clusters, Unity Catalog, identity, networking, and secret scopes using Infrastructure-as-Code tools such as Terraform.
• Design and implement CI/CD pipelines using GitHub Actions and Azure DevOps to support data and platform deployments.
• Establish and enforce platform guardrails including cluster policies, cost controls, logging, alerting, drift detection, and secure networking standards.
• Ensure platform health, scalability, reliability, and cost optimization across Databricks environments.
• Provide tier-3 operational support, troubleshoot complex incidents, and drive root-cause resolution.
• Collaborate with data engineering, analytics, and AI teams to enable efficient development and deployment of data solutions.
• Contribute to platform standards, documentation, and continuous improvement initiatives.

Requirements

• 5 years of experience in designing, developing, and deploying solutions on the Databricks platform with strong understanding of its architecture and capabilities.
• Proficiency in Python, including PySpark, and SQL; experience with Scala or Java is a plus.
• Strong understanding of cloud platforms such as AWS, Azure, or GCP, including compute, storage, identity and access management, and networking services.
• Hands-on experience with data warehousing, data lake, and Lakehouse architectures, including Delta Lake and Medallion Architecture concepts.
• Proven experience building and maintaining ETL and ELT pipelines in enterprise environments.
• Experience using Git-based version control and CI/CD practices for data and platform deployments.
• Strong problem-solving, critical thinking, and communication skills, with the ability to operate independently in complex technical environments.

Qualifications

• Master's degree in Computer Science, Information Technology, or related field, or equivalent practical experience.
• Hands-on experience with Databricks, including managing clusters, workflows, and notebooks.
• Experience with Kubernetes, Docker, and container orchestration tools.
• Knowledge of data governance principles and best practices.
• Experience with monitoring and logging tools such as Prometheus, Grafana, and ELK Stack.
• Experience with security best practices, including encryption, access controls, and compliance requirements.

Skills

• Strong understanding of cloud platforms such as AWS, Azure, or GCP.
• Experience with data warehousing, data lake, and Lakehouse architectures.
• Hands-on experience with Databricks, including managing clusters, workflows, and notebooks.
• Experience with Kubernetes, Docker, and container orchestration tools.
• Knowledge of data governance principles and best practices.
• Experience with monitoring and logging tools such as Prometheus, Grafana, and ELK Stack.
• Experience with security best practices, including encryption, access controls, and compliance requirements.

Benefits

• Comprehensive benefits package including medical, dental, vision, and life insurance.
• Flexible work schedule and remote work options.
• Professional development opportunities and training programs.
• Employee recognition and rewards program.
• Discounted technology products and services from CDW.

Pay

Competitive salary based on experience and qualifications.

Schedule

Full-time position with standard business hours.