Databricks Data Engineer

Role Summary

The Databricks Data Engineer will design, build, and optimize scalable data pipelines supporting Claims Payment Integrity (PI) analytics across Medicare & Retirement, Community & States, and Employer & Individual businesses. The role focuses on developing governed lakehouse-based data assets, integrating claims and provider datasets, and ensuring high-quality data availability for PI, actuarial, audit, recovery, and financial analytics teams.

Key Responsibilities

• Data Engineering & Lakehouse Development
• Build scalable ETL/ELT pipelines in Databricks using PySpark, Spark SQL, Delta Live Tables, and workflows.
• Engineer curated datasets across bronze/silver/gold layers for claims, pricing, provider, RCM, and member data.
• Implement Delta Lake best practices including ACID transactions, schema evolution, CDC, and optimized storage formats.
• Automate ingestion/transformation of large datasets from claims systems, provider files, call center platforms, and EHR feeds.
• Data Quality & Governance
• Perform reconciliation and validation of claim-related financial datasets.
• Enforce PHI-compliant design patterns using Unity Catalog, governance guardrails, and cluster policies.
• Implement pipeline monitoring, logging, and Spark performance optimization.
• Platform & Collaboration
• Work with Data Analysts, Data Scientists, and PI SMEs to translate analytic requirements into production data assets.
• Support cluster optimization, table indexing (Z-ORDER), and cost-efficient lakehouse operations.
• Participate in Agile ceremonies and ensure timely delivery of engineering tasks.

Technical Skills Required

• Hands-on experience with Databricks (PySpark, SQL, Delta Lake, Jobs/Workflows).
• Strong Spark performance tuning experience.
• Experience engineering data for claims, provider, and membership domains.
• Strong understanding of healthcare data models and adjudication flows.
• Typically 5-8 years of Data Engineering experience in healthcare.
• Bachelor's degree (4-year).

Nice-to-Have Skills

• Experience with Call center data (member & provider interactions), Provider RCM datasets, and EHR/clinical data.
• Experience with DLT, CI/CD, and MLflow-integrated pipelines.
• Exposure to actuarial or PI forecasting workflows.