Principal Software Engineer, Tech Lead - Databricks Platform

Responsibilities

• Own day-to-day administration of the Databricks environment on AWS, including workspace configuration, cluster policies, user and group management, access controls, and cost governance.
• Design and evolve the Cardlytics Databricks architecture; establish reference architectures and standards for batch, streaming, and ML workloads.
• Drive adoption of Unity Catalog for centralized RBAC, column-level security, data lineage, and metadata management; implement and maintain security best practices including encryption and audit logging.
• Monitor platform health, define SLAs, and lead performance tuning for Spark jobs, Delta tables, and SQL warehouses; own incident response and root-cause analysis.
• Serve as the primary technical partner for Analysts, ML Engineers, and Business users; develop self-service guides, onboarding materials, and best-practice documentation to reduce friction across all user groups.
• Own the CI/CD strategy for Databricks deployments using Terraform, GitHub Actions, and Databricks Asset Bundles; enforce DevOps practices across the team.
• Lead and mentor a small team of data/platform engineers; provide technical direction, drive engineering excellence, and manage the Databricks vendor relationship.

Requirements

• 7+ years of experience in data engineering or data platform roles, with at least 2 years in a technical lead or architecture capacity.
• Deep, hands-on expertise with the Databricks platform: Delta Lake, Spark (PySpark), Databricks SQL, Unity Catalog, Workflows, and cluster management on AWS.
• Proven experience designing and delivering Lakehouse architectures and large-scale ELT/ETL pipelines in production environments.
• Proficiency in Infrastructure as Code tools, preferably Terraform.
• Demonstrated experience supporting diverse stakeholder groups including Analysts, ML Engineers, and non-technical Business users.
• Strong understanding of data governance, data modeling, and cloud security best practices.
• Excellent communication and collaboration skills, with the ability to translate technical concepts for non-technical audiences.

Qualifications

• We primarily use macOS and Google Workspace (Docs, Sheets, Slides).
• Familiarity with or willingness to work within this environment is required.

Skills

• Experience with Databricks platform components such as Delta Lake, Spark (PySpark), Databricks SQL, Unity Catalog, Workflows, and cluster management on AWS.
• Hands-on experience with Infrastructure as Code tools like Terraform.
• Ability to design and deliver large-scale ELT/ETL pipelines in production environments.
• Experience with data governance, data modeling, and cloud security best practices.
• Strong communication and collaboration skills.

Benefits

• Flexible paid time off plus company holidays.
• Medical, dental, and vision insurance begins on your first day.
• 401(k) retirement plan with company match, plan also includes a student loan debt repayment option.
• Employee Stock Purchase Plan.
• Continuing education assistance.
• Lifestyle Spending Account for physical, emotional, and financial wellness (like gym memberships, home down payments, art classes, park passes, and more!)
• Complimentary Calm app subscriptions to support employee mental health and wellbeing.