Senior Data Engineer
CareScout · Richmond, VA · 3 mo ago
Information Technology$121k–$187k/yrFull-time
About The Role
We are seeking a highly skilled and experienced Senior Data Engineer to join our growing data and machine learning organization and help build the pipelines, models, and infrastructure that power our analytics, machine learning, and operational data needs.
In this role, you will work closely with analysts, data scientists, ML/AI engineers, and product teams to design and deliver reliable, scalable data workflows on our Databricks Lakehouse platform.
- Data Pipeline Engineering
- Design, build, and maintain scalable ETL/ELT pipelines using Spark, Python, SQL, and Databricks.
- Implement reliable ingestion frameworks for batch and streaming data sources.
- Ensure pipelines meet SLAs, data quality standards, and production-grade reliability.
- Implement reliable ingestion frameworks for batch and streaming data sources.
- Design, build, and maintain scalable ETL/ELT pipelines using Spark, Python, SQL, and Databricks.
- Lakehouse Modeling & Architecture
- Create robust data models across raw, curated, and semantic layers using Delta Lake.
- Develop dimensional models, star schemas, and domain-layer datasets for analytics and ML.
- Establish and maintain standards for schema design, metadata, and lineage.
- Develop dimensional models, star schemas, and domain-layer datasets for analytics and ML.
- Create robust data models across raw, curated, and semantic layers using Delta Lake.
- Data Quality & Observability
- Implement data validation, anomaly detection, SLAs, and documentation across pipelines.
- Build automated tests, monitoring, and alerting for freshness, completeness, and accuracy.
- Implement data validation, anomaly detection, SLAs, and documentation across pipelines.
- Collaboration & Cross-Functional Support
- Work closely with analysts to understand business KPIs and deliver high-quality curated datasets.
- Partner with ML engineers and data scientists to build reusable feature pipelines.
- Collaborate with data platform engineers to optimize compute, governance, and orchestration.
- Work closely with analysts to understand business KPIs and deliver high-quality curated datasets.
- Performance & Optimization
- Optimize Spark jobs, SQL queries, cluster configurations, and storage patterns for performance and cost.
- Security, Compliance & Governance
- Apply best practices for RBAC, data privacy, and PII handling using Unity Catalog.
- Continuous Learning
- Stay current on modern data engineering patterns, Lakehouse architecture, orchestration, and best practices.