Senior Databricks Data Engineer

Job Responsibilities

• Build and evolve an internal Azure Databricks Lakehouse platform using Delta Lake and standardized architectural patterns.
• Develop scalable ingestion and transformation frameworks for multiple source systems.
• Implement curated, trusted datasets and “data products” with clear contracts.
• Operationalize pipelines using Databricks Workflows (Jobs).
• Own performance and cost optimization across workloads.
• Implement and administer governance and security with Unity Catalog.
• Establish engineering standards for the platform.
• Partner with stakeholders to align platform roadmap, priorities, and adoption.
• Provide technical leadership across the team.

Qualifications

• 8+ years of data engineering experience, including 3+ years with Azure Databricks.
• Strong proficiency in Python (PySpark) and Spark SQL.
• Strong experience with Delta Lake.
• Experience productionizing workloads with Databricks Workflows (Jobs).
• Strong Azure fundamentals relevant to a data platform.
• Knowledge of data architecture and platform concerns.
• Demonstrated ability to work cross-functionally and communicate technical decisions.
• Demonstrated technical leadership and mentoring experience.
• Databricks Certified Data Engineer Professional certification preferred.
• Experience with CI/CD for Databricks assets and familiarity with Terraform preferred.
• Experience with streaming or near-real-time ingestion and/or Delta Live Tables preferred.
• Familiarity with Databricks SQL and serving data to BI tools is preferred.
• Private equity, credit, or Alternative Asset Management industry experience preferred.

Pay

Salary: $150,000 – $225,000, Plus Bonus