Python Hadoop Engineer

Roles & Responsibilities

• 5 to 10 years of experience in Data Engineering, Software Engineering, or related technical discipline.
• Strong proficiency in Python, and SQL for advanced data transformations.
• Hands-on experience designing and building ETL/ELT pipelines, data ingestion processes, and distributed data processing jobs.
• Practical experience working with distributed data tools such as Apache Spark, Databricks, or Hadoop ecosystems.
• Experience building and managing datasets in relational and/or cloud-based data platforms (Teradata, Snowflake, SQL Server, Azure/AWS/GCP).
• Solid understanding of data modeling, metadata, data quality controls, data lineage, and secure data management.
• Experience contributing to automated test suites, analyzing test failures, and supporting test-driven development.
• Knowledge of CI/CD pipelines, version control (Git), and automated deployment practices.
• Experience adhering to enterprise data governance, compliance, and operational risk frameworks.
• Ability to troubleshoot pipeline issues, performance bottlenecks, and data discrepancies.
• Strong communication skills and ability to collaborate across engineering, product, and business teams.
• Experience implementing monitoring and observability for data pipelines (logs, metrics, health checks).
• Advanced experience with performance tuning of SQL, Spark, or distributed data workflows.
• Knowledge of data security practices (encryption, masking, PII handling).
• Experience supporting analytical workloads, BI tools, or data science teams.
• Prior experience in a financial institution or other regulated industry.

Qualifications

• BACHELOR OF COMPUTER SCIENCE