Senior Data Engineer - Databricks
RADcube · Indianapolis, IN · 2 mo ago
On-siteInformation TechnologyFull-time
Key Responsibilities
- Design, develop, and maintain data ingestion pipelines for real-world data (RWD) sources
- Build and optimize data models within a Databricks Lakehouse architecture
- Ensure data governance, quality, and compliance standards are met
- Optimize cost, performance, and resource utilization across AWS and Databricks
- Collaborate with business and technical stakeholders to translate requirements into scalable solutions
- Provide technical guidance, coding standards, and best practices
- Support knowledge sharing through documentation and training sessions
- Deliver status updates and progress reports to senior leadership
Requirements
- Strong hands-on experience with AWS services (S3, Glue, Lambda, Redshift, etc.)
- Proven experience with Databricks (Delta Lake, notebooks, workflows)
- Expertise in SQL and Python
- Experience building end-to-end ETL/ELT data pipelines
- Strong understanding of data modeling techniques
- Experience with data governance, quality, and compliance frameworks
- Ability to work independently and in a collaborative team environment
- Databricks Certification
- Experience building and managing CI/CD pipelines
- Experience with workflow orchestration tools
- Exposure to data observability and lineage tools
- Experience with monitoring, logging, and observability frameworks
- Highly Preferred Skills:
- Experience in Healthcare / Pharma domain
- Exposure to RWD (Real World Data) or HEOR analytics
- Familiarity with Lakehouse architecture / Medallion (Bronze-Silver-Gold) models
- Experience optimizing cost and performance in cloud environments
- Strong communication and stakeholder management skills