Data Engineer

Responsibilities

• Pipeline Architecture: Design, implement, and maintain end-to-end data pipelines on Azure, ensuring high availability and low latency for healthcare claim and analytics processing.
• High-Performance Storage: Manage and optimize ClickHouse as our primary analytical engine, focusing on rapid data ingestion and lightning-fast query performance for large-scale datasets.
• ML Data Readiness: Structure data environments to support the full ML lifecycle, from feature engineering and training to real-time model inference.
• MLOps Integration: Collaborate with Data Scientists to implement automated CI/CD pipelines for model deployment, monitoring, and retraining.
• Rapid Acquisition: Develop scalable frameworks to ingest diverse healthcare data sources (EDI, claims, clinical notes) with high velocity.
• Security & Compliance: Ensure all data structures and processes adhere to HITRUST/HIPAA standards, collaborating with IT and the leads for technical efforts for HITRUST certification readiness.
• Cloud Expertise: 5+ years of experience in data engineering, with deep proficiency in Azure Data Factory, Azure Databricks, or Azure Synapse.
• OLAP Mastery: Proven experience managing and tuning ClickHouse (or similar columnar databases like Druid/Pinot) for massive datasets.
• Programming: Expert-level Python and SQL skills.
• ML Engineering: Familiarity with ML frameworks (PyTorch, TensorFlow) and MLOps tools (MLflow, Kubeflow, or Azure Machine Learning).
• Healthcare Domain: Prior experience with healthcare data formats (HL7, FHIR, 835/837) and a strong understanding of HITRUST/HIPAA security requirements.
• Scale-up Mindset: Ability to build "v1" processes while designing for 10x growth.
• Experience with Infrastructure as Code (Terraform, Bicep).
• Knowledge of stream processing (Kafka, Azure Event Hubs).
• Background in financial or payment integrity analytics.

Qualifications

• 5+ years of experience in data engineering.
• Deep proficiency in Azure Data Factory, Azure Databricks, or Azure Synapse.
• Proven experience managing and tuning ClickHouse (or similar columnar databases like Druid/Pinot) for massive datasets.
• Expert-level Python and SQL skills.
• Familiarity with ML frameworks (PyTorch, TensorFlow) and MLOps tools (MLflow, Kubeflow, or Azure Machine Learning).
• Prior experience with healthcare data formats (HL7, FHIR, 835/837).
• Strong understanding of HITRUST/HIPAA security requirements.
• Ability to build "v1" processes while designing for 10x growth.
• Experience with Infrastructure as Code (Terraform, Bicep).
• Knowledge of stream processing (Kafka, Azure Event Hubs).
• Background in financial or payment integrity analytics.

Skills

• Azure Data Factory
• Azure Databricks
• Azure Synapse
• ClickHouse
• Python
• SQL
• ML frameworks (PyTorch, TensorFlow)
• MLOps tools (MLflow, Kubeflow, Azure Machine Learning)
• Kafka
• Azure Event Hubs