Data Engineer
Blue Cross Blue Shield Association · Chicago, IL · 4 wk ago
HybridEngineeringFull-time
Design, build, and maintain
- Reliable, high-performance data pipelines for large-scale structured and unstructured healthcare data.
- Use PySpark and modern cloud-based tools (Databricks, AWS Glue, EMR, Snowflake) to transform and process data efficiently.
- Support ingestion, transformation, and validation processes that ensure data consistency, integrity, and availability.
Partner with teams
- Partner with Data Architects, Data Scientists, and Analysts to translate business needs into scalable engineering solutions.
- Collaborate with platform and DevOps teams to deploy, scale, and monitor data pipelines using Airflow and Kubernetes.
Implement and maintain
- Implement and maintain data validation frameworks to ensure pipeline accuracy and completeness.
- Contribute to best practices in version control, metadata management, and reproducibility.
Stay current and recommend improvements
- Stay current with emerging technologies in data engineering and cloud computing, recommending improvements to existing infrastructure.
- Identify automation opportunities to streamline ETL/ELT processes and reduce operational overhead.
Share knowledge and mentor
- Share knowledge and mentor junior team members on tools, techniques, and best practices.
Compliance and culture
- Support compliance with SOC 2, HIPAA, and GDPR by adhering to established data privacy and security practices.
- Promote a culture of collaboration, innovation, and continuous learning within the engineering organization.
Required Education, Certifications and Experience:
- Education: Bachelor’s or Master’s degree in Computer Science, Engineering, or related field.
- Experience: 5+ years of experience in data engineering, including building and managing pipelines in cloud-based environments.
- Knowledge Skills and Abilities:
- Experience with building and operationalizing the data foundations that support machine learning and generative AI use cases, including feature pipelines, training/inference data preparation, and retrieval-ready datasets (e.g., embeddings and vector stores).
- Familiarity with GenAI skills and adjacent tooling (foundation models, prompt engineering, RAG, embeddings/vector databases, and GenAI orchestration frameworks).
- Hands-on experience with AWS AI/ML and data services, including Amazon Bedrock, Bedrock Agent Core, SageMaker, Glue, and EMR.
- Experience designing and optimizing data architectures, including data foundations that support ML and GenAI workloads.
- Hands-on experience with workflow orchestration (Airflow) and containerization (Kubernetes).
- Hands-on technical expertise, cross-team collaboration, and contributing to architecture decisions.
- Proficiency in Python, SQL, and distributed data frameworks (PySpark, Databricks, AWS Glue, EMR).
- Working knowledge of cloud platforms (AWS or Azure) and data warehouses (Snowflake).
- Familiarity with NoSQL and relational databases, as well as data modeling best practices.
- Strong analytical, problem-solving, and communication skills.
- Understanding of compliance frameworks (SOC 2, HIPAA) and secure data management principles.
- Experience working with healthcare datasets or knowledge of healthcare standards (HIPAA, HL7, FHIR) preferred.
The posting range for this position is: $100,800.00 - $138,600.00