AWS Data Engineer

Responsibilities

• Lead and support the delivery of data platform modernization projects.
• Design and develop robust and scalable data pipelines leveraging AWS native services.
• Optimize ETL processes, ensuring efficient data transformation.
• Migrate workflows from on-premise to AWS cloud, ensuring data quality and consistency.
• Design automations and integrations to resolve data inconsistencies and quality issues.
• Perform system testing and validation to ensure successful integration and functionality.
• Implement security and compliance controls in the cloud environment.
• Ensure data quality pre- and post-migration through validation checks and addressing issues regarding completeness, consistency, and accuracy of data sets.
• Collaborate with data architects and lead developers to identify and document manual data movement workflows and design automation strategies.

Requirements

• 10+ years’ experience with a core data engineering skillset leveraging AWS native technologies (AWS Glue, Python, Snowflake).
• Experience in the design and development of robust and scalable data pipelines leveraging AWS native services.
• Proficiency in leveraging Snowflake for data transformations, optimization of ETL pipelines, and scalable data processing.
• Experience with streaming and batch data pipeline/engineering architectures.
• Familiarity with DataOps concepts and tooling for source control and setting up CI/CD pipelines on AWS.
• Hands-on experience with Databricks and a willingness to grow capabilities.
• Experience with data engineering and storage solutions (AWS Glue, EMR, Lambda, Redshift, S3).
• Strong problem-solving and analytical skills.
• Knowledge of Dataiku is needed.
• Graduate/Post-Graduate degree in Computer Science or a related field.
• AWS S3 (data storage, export, recall) Athena (querying data lakes) Data pipelines (batch & near-real-time) Integration with external systems (FHIR) Secure data handling (KMS, Macie) Cloud-native analytics.
• Multi-account, multi-region data architecture.
• BI integrations: Power BI, Tableau, QuickSight.