Scala/Spark Developer - Big Data

About the role

Design and develop scalable data-processing applications using Scala and Apache Spark. Build, maintain, and optimize large-scale data/ETL pipelines. Work extensively with Hadoop ecosystem technologies, including Hive and Impala. Develop complex SQL queries and data transformations. Use Python for data processing, scripting, and automation. Analyze performance bottlenecks and optimize Spark jobs for scalability and performance. Perform data validation, troubleshooting, and production support. Work with large volumes of structured and semi-structured financial data. Collaborate with architects, analysts, and business teams to implement data solutions.

Responsibilities

• Design and develop scalable data-processing applications using Scala and Apache Spark.
• Build, maintain, and optimize large-scale data/ETL pipelines.
• Work extensively with Hadoop ecosystem technologies, including Hive and Impala.
• Develop complex SQL queries and data transformations.
• Use Python for data processing, scripting, and automation.
• Analyze performance bottlenecks and optimize Spark jobs for scalability and performance.
• Perform data validation, troubleshooting, and production support.
• Work with large volumes of structured and semi-structured financial data.
• Collaborate with architects, analysts, and business teams to implement data solutions.

Requirements

• Spark
• Scala
• Hadoop (Hive/Impala)
• SQL
• Python

Nice to Have

• Kafka
• Airflow
• Cloud platforms
• CI/CD
• Data Lake technologies

Preferred Qualifications

Banking/Financial Services experience, particularly in Trading, Capital Markets, Tax, Risk, or Regulatory data.