Specialist - Data Engineering

About the role

Design, develop, and maintain data pipelines using PySpark for large-scale data processing. Perform data transformations, aggregations, and validations using Spark SQL Data Frames. Write and optimize SQL queries for data extraction, reconciliation, and reporting. Conduct data quality checks, profiling, and consistency validations.

Responsibilities

• Support existing data pipelines by troubleshooting failures and performing root cause analysis.
• Work closely with data engineers, analysts, QA, and DevOps teams in Agile delivery models.
• Optimize performance of Spark jobs for efficiency and scalability.
• Maintain technical documentation workflows and runbooks.
• Follow enterprise standards for coding, deployment, and change management.

Requirements

• 7+ years of experience in Big Data Data Engineering roles.
• Strong hands-on experience with PySpark and Apache Spark.
• Solid understanding of SQL and relational database concepts.
• Experience working in Linux/UNIX environments.
• Understanding of ETL concepts and large-scale data processing.
• Strong analytical and debugging skills.
• Experience supporting production data pipelines.
• Exposure to Azure Databricks, ADF, Synapse, or similar cloud data platforms.
• Knowledge of orchestration tools like Airflow.
• Python scripting experience beyond Spark use cases.
• Familiarity with Agile DevOps delivery practices.
• Domain exposure in Banking, BFSI, or Analytics is advantageous.

Qualifications

• Strong problemsolving and ownership mindset.
• Ability to work collaboratively in cross-functional teams.
• Good communication and documentation skills.
• Ability to adapt quickly in dynamic project environments.

Skills

• PySpark
• Apache Spark
• SQL
• Linux/UNIX
• ETL concepts
• Azure Databricks
• ADF
• Synapse
• Orchestration tools (Airflow)
• Python scripting

Benefits

• Comprehensive Medical Plan
• Coverage for Medical, Dental, Vision, Short Term and Long-Term Disability
• 401(k) Plan with Company match
• Life Insurance
• Vacation Time, Sick Leave, Paid Holidays
• Paid Paternity and Maternity Leave