Senior Big Data Engineer - Cloudera Hadoop to AWS Migration

Job Summary

We are seeking a senior-level Big Data Engineer with hands-on experience migrating enterprise Cloudera Hadoop workloads to AWS. The ideal candidate has deep expertise in HDFS, Hive, Spark, Oozie, Python, shell scripting, and AWS services including EMR, Glue, Airflow, and Lambda. This role will lead migration planning, target architecture design, application replatforming, validation, and technical documentation.

Responsibilities

• Design and develop scalable, maintainable software and data-processing solutions.
• Analyze existing Cloudera Hadoop architecture, workflows, and data pipelines.
• Develop and execute migration strategies from on-premises Hadoop to AWS.
• Design AWS architectures using EMR, Glue, Airflow, Lambda, and related services.
• Oversee data transfer, application re-platforming, testing, and validation.
• Ensure solutions meet performance, scalability, security, and reliability requirements.
• Troubleshoot software, platform, and data-pipeline issues.
• Create technical documentation, migration plans, and operational runbooks.
• Collaborate with engineering, DevOps, QA, product, and business stakeholders.
• Mentor team members on software engineering and AWS big data practices.

Qualifications

• Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.
• 10+ years of experience with enterprise application design, Hadoop, and big data platforms.
• Hands-on experience with Cloudera Hadoop, HDFS, Hive, Spark, Oozie, and Python.
• Proven experience migrating or modernizing Hadoop workloads on AWS.
• Strong experience with Amazon EMR, AWS Glue, Airflow, Lambda, and cloud data architecture.
• Proficiency with Bash or similar shell scripting languages.
• Experience With Terraform Or Other Infrastructure-as-Code Tools.
• Strong understanding of cloud security, governance, compliance, and data protection.
• Experience with XML and JSON data formats.
• Familiarity with monitoring and logging tools.
• 5+ years of experience with tools such as Bitbucket and Artifactory.
• Strong communication, documentation, troubleshooting, and mentoring skills.

Preferred Qualifications

• AWS certification, such as AWS Certified Solutions Architect or a related data certification.
• Experience working with healthcare claims data, providers, members, and related healthcare entities.
• Previous experience leading large-scale Hadoop-to-AWS migration initiatives.