Data Engineer - Hybrid / Remote

Core Responsibilities

• Platform & Architecture: Architect and implement scalable data processing pipelines using Databricks Runtime (Apache Spark, Spark SQL, MLflow, Delta Lake)
• Delta Lake: Design and enforce a medallion (Bronze/Silver/Gold) architecture with schema evolution, Delta Live Tables (DLT), and robust error-handling patterns
• Ingestion Frameworks: Build high-performance ingestion frameworks for FHIR and HL7 message streams, X12 837/835 healthcare claims data, EHR/EMR source systems, batch, real-time, and event-driven data sources
• Azure Cloud Engineering: Develop and operate data pipelines leveraging Azure Data Lake Storage Gen2, Azure Data Factory or Synapse Pipelines, Azure Event Hubs and/or Service Bus for streaming ingestion, Azure SQL Database and Azure Synapse (Dedicated and Serverless pools), Azure Functions for lightweight orchestration and automation, Azure Monitor, Log Analytics, and Application Insights for observability
• Security: Implement enterprise-grade security including VNet integration and private endpoints, Secrets and key management using Azure Key Vault, Managed identities and least-privilege access controls
• Distributed Data Engineering: Develop optimized PySpark and/or Scala pipelines using advanced Spark techniques, Catalyst optimizer tuning, Cluster sizing and autoscaling strategies, Adaptive Query Execution (AQE), Efficient join strategies (broadcast vs. shuffle)
• Pipelines: Build and maintain high-volume batch ETL pipelines (100M+ records), low-latency streaming pipelines using Spark Structured Streaming
• CI/CD: Implement CI/CD for Databricks environments, including Git-integrated DEV/QA/PROD workspaces, automated job and workflow deployments, unit testing using pytest and Databricks testing frameworks
• Healthcare Data & Compliance: Design and implement secure PHI pipelines compliant with HIPAA Privacy and Security Rules, SOC 2 and HITRUST-aligned controls, build pipelines supporting healthcare data standards including FHIR R4 resources, HL7 v2/v3, X12 EDI transactions

Required Qualifications

• 5+ years of experience in modern data engineering roles
• Expert-level proficiency in PySpark and Spark SQL, Databricks (Jobs, Workflows, Repos, Delta Live Tables), Delta Lake architecture and transactional design patterns, Azure Data Factory or Azure Synapse Pipelines, cloud-native data security (RBAC, ABAC, privilege boundary enforcement)
• Strong experience working with healthcare data formats and standards: FHIR (JSON), HL7 v2/v3, X12 EDI claims data
• Deep understanding of distributed systems, data partitioning strategies, concurrency, and cluster resource tuning

Preferred Qualifications

• Experience implementing Unity Catalog at enterprise scale
• Familiarity with MLOps workflows and Databricks MLflow
• Experience using dbt with Databricks SQL
• Relevant certifications: Databricks Data Engineer Professional, Microsoft Azure DP-203, HL7 or FHIR certification (nice to have)

Benefits

• Comprehensive health, dental, and vision insurance
• Health Savings Account with an employer contribution
• Life Insurance
• PTO
• 401(k) retirement plan with a company match

Environmental/Working Conditions

Normal busy office environment with much telephone work. Possible long hours as needed.