Big Data Engineer

Responsibilities

• Design, develop, and maintain large-scale data processing pipelines using Big Data technologies (e.g., Hadoop, Spark, Python, Scala)
• Implement data ingestion, storage, transformation, and analysis solutions that are scalable, efficient, and reliable
• Stay current with industry trends and emerging Big Data technologies to continuously improve the data architecture
• Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions
• Optimize and enhance existing data pipelines for performance, scalability, and reliability
• Develop automated testing frameworks and implement continuous testing for data quality assurance
• Conduct unit, integration, and system testing to ensure the robustness and accuracy of data pipelines
• Write and maintain automated unit, integration, and end-to-end tests
• Monitor and troubleshoot data pipelines in production environments to identify and resolve issues

Requirements

• Strong software development background with the ability to contribute to technical discussions
• Extensive experience with Scrum, Kanban, and continuous improvement practices
• Experience with big data technologies such as Hadoop, Spark, Hive, and Trino
• Understanding of common challenges such as: Data skew and mitigation strategies, working with massive data volumes (petabyte scale), troubleshooting job failures related to resource constraints, bad data, and scalability issues
• Ability to provide real-world debugging and mitigation examples
• Proficiency in SQL including window functions, multi-table joins, and aggregations
• Experience handling edge cases such as NULLs, duplicates, and ordering
• Experience with Apache Spark, strong understanding of Spark architecture (executors, tasks, stages, DAG), and performance tuning techniques (partitioning, caching, broadcast joins)
• Experience optimizing jobs for large-scale datasets
• Experience with AWS services such as S3, EMR, Glue, Lambda, Athena, etc.
• Experience with CI/CD pipelines
• Experience writing test cases
• Experience with object-oriented and database technologies leading to enterprise-quality solutions
• Experience developing enterprise solutions in an iterative or Agile environment
• Extensive knowledge of test automation, build automation, and configuration management frameworks
• Strong written and verbal communication skills
• Proven ability to build effective working relationships and improve quality of work products
• Strong organizational skills with the ability to manage competing priorities
• Experience managing production data pipelines and ETL systems
• Experience with object-oriented programming languages such as Java, Scala, or Python

Qualifications

• Bachelor’s degree in Computer Science, Information Systems, or related discipline with at least five (5) years of related experience, or equivalent training and/or work experience
• Master’s degree and financial services industry experience preferred