Sr. Data Platform Engineer

About the role

We are seeking a Senior Data Platform Engineer to support the administration, automation, governance, security, and optimization of our enterprise Databricks Lakehouse platform on AWS. This is a hands-on technical role responsible for maintaining and enhancing the Databricks platform, implementing infrastructure automation, enforcing compute and security standards, supporting cost-governance initiatives, and enabling reliable Data Engineering and Data Science workloads. The ideal candidate will have strong hands-on experience with Databricks platform administration, AWS cloud services, Terraform-based infrastructure automation, and DevOps practices. The individual will evaluate emerging Databricks capabilities through proof-of-concept implementations and contribute to reusable platform patterns and standards. This role will collaborate closely with Data Engineering, Data Science, Cloud Platform, Security, and Governance teams to ensure the Databricks environment remains secure, reliable, scalable, and cost-efficient.

Responsibilities

• Databricks Platform Administration – Support and maintain enterprise Databricks workspace architecture, including networking, security, and compute governance.
• Implement and maintain cluster policies, job policies, compute guardrails, and platform governance standards.
• Manage and optimize Databricks compute environments including job clusters, interactive clusters, and serverless compute.
• Implement Unity Catalog for centralized data governance and access management.
• Manage workspace configurations, platform integrations, and operational reliability.
• AWS Cloud Architecture for Databricks – Design and maintain secure and scalable Databricks infrastructure leveraging AWS services including: IAM roles and cross‑account access architecture, VPC networking, subnets, and secure connectivity, PrivateLink and secure endpoint configuration, S3 data lake architecture, AWS KMS encryption and key management, CloudWatch monitoring and logging integration; ensure the platform aligns with enterprise security, compliance, and networking standards.
• Infrastructure as Code & DevOps Automation – Build and maintain Databricks infrastructure using Terraform; implement infrastructure‑as‑code frameworks for workspace provisioning, cluster policies, and platform configurations; develop automation using Databricks CLI, Databricks REST APIs, Python and shell scripting; design and maintain CI/CD pipelines for Databricks workloads and platform infrastructure; integrate deployments with Git‑based development workflows (GitHub, GitLab, Azure DevOps); enable standardized environment promotion frameworks across Dev, Test, and Production environments.
• Platform Cost Governance & Optimization – Monitor and analyze DBU consumption and compute utilization across workloads; provide recommendations on cost optimizations; implement guardrails such as cluster policies, budget alerts, resource tagging, usage monitoring dashboards; identify and implement cost optimization strategies across compute configurations, job scheduling patterns, serverless vs classic compute usage, storage lifecycle management; build dashboards for cost transparency.
• Data Platform Enablement – Provide technical guidance and platform best practices for data engineering and data science workloads, including: Delta Lake architecture and performance optimization, Medallion architecture (Bronze/Silver/Gold), batch and streaming data ingestion, large‑scale Spark workloads; MLflow experimentation and model lifecycle management, Feature Store usage, machine learning pipelines and model deployment frameworks.
• Innovation & Proof of Concept Development – Evaluate and operationalize new Databricks capabilities by building Proof of Concepts (POCs) and support adoption into platform standards; areas of focus include: Databricks Serverless Compute, Delta Live Tables, Auto Loader, Lakehouse AI capabilities, Databricks Feature Store, Vector Search and LLM integrations, Lakehouse monitoring and observability frameworks; support and maintain reference architectures and reusable design patterns based on POC outcomes.

Required Technical Skills

• Databricks Platform – Databricks workspace administration, cluster and job policy design, Unity