Hadoop developer

Skills

Primary: PySpark, Hive, Python, SQL, Hadoop
Secondary: Unix, Agile, Base support

Responsibilities

• Design and build ETL/ELT pipelines, data ingestion processes, and distributed data processing jobs.
• Work with distributed data tools such as Apache Spark, Databricks, or Hadoop ecosystems.
• Build and manage datasets in relational and/or cloud-based data platforms (Teradata, Snowflake, SQL Server, Azure/AWS/GCP).
• Apply data modeling, metadata, data quality controls, data lineage, and secure data management practices.
• Contribute to automated test suites, analyze test failures, and support test-driven development.
• Implement CI/CD pipelines, version control (Git), and automated deployment practices.
• Adhere to enterprise data governance, compliance, and operational risk frameworks.
• Troubleshoot pipeline issues, performance bottlenecks, and data discrepancies.
• Collaborate across engineering, product, and business teams.
• Implement monitoring and observability for data pipelines (logs, metrics, health checks).
• Tune performance of SQL, Spark, or distributed data workflows.
• Apply data security practices (encryption, masking, PII handling).
• Support analytical workloads, BI tools, or data science teams.

Requirements

• 5 to 10 years of experience in Data Engineering, Software Engineering, or related technical discipline.
• Strong proficiency in Python and SQL for advanced data transformations.
• Practical experience with distributed data tools (Apache Spark, Databricks, Hadoop).
• Experience in a financial institution or other regulated industry.

Qualifications

Bachelor of Computer Science

Benefits

• Discretionary Annual Incentive
• Comprehensive Medical Coverage: Medical & Health, Dental & Vision
• Disability Planning & Insurance
• Pet Insurance Plans
• Maternal & Parental Leaves
• Auto & Home Insurance
• Identity Theft Protection
• Commuter Benefits
• Certification & Training Reimbursement
• Vacation, Time Off, Sick Leave & Holidays
• Legal Assistance
• 401K Plan
• Performance Bonus
• College Fund
• Student Loan Refinancing

Pay

$95,000 - $115,000 a year