Lead Data Engineer

Job Summary

Lead and contribute to moderately complex technology initiatives, focusing on large-scale data processing frameworks aligning with enterprise strategies.

Responsibilities

• Build, optimize, and maintain robust and highly available data pipelines for advanced analytics and reporting.
• Analyze and resolve moderately complex business, operational, or technical issues involving multiple variables.
• Oversee data integration activities, including data modeling, data warehouse/analytics environment management, and development of integration/analysis scripts.
• Lead teams in delivering data engineering solutions, ensuring compliance with data policies and procedures.
• Collaborate with colleagues and management to resolve data engineering challenges and achieve organizational objectives.
• Conduct code reviews emphasizing testability and code coverage.
• Execute end-to-end data engineering projects in Agile/Scrum environments.
• Utilize deep expertise in Ab Initio, Spark (Core, SQL, DataFrames, Datasets), Python/scripting, GCP/cloud platforms, Teradata, Hadoop, Hive, Autosys, Airflow, BigQuery, SQL/PL-SQL, and distributed systems.
• Preferred: Experience leading teams/programs, familiarity with non-relational DBMS (MongoDB), real-time data streaming (Kafka), AI/ML pipelines (Apache Spark, Sparkling Water, H2O, LLMs), Agentic AI frameworks, statistical/machine learning model development, Unix/shell scripting, CI/CD pipelines, and cloud deployment best practices.

Qualifications

• Deep expertise in Ab Initio, Spark (Core, SQL, DataFrames, Datasets), Python/scripting, GCP/cloud platforms, Teradata, Hadoop, Hive, Autosys, Airflow, BigQuery, SQL/PL-SQL, and distributed systems.
• Preferred: Experience leading teams/programs, familiarity with non-relational DBMS (MongoDB), real-time data streaming (Kafka), AI/ML pipelines (Apache Spark, Sparkling Water, H2O, LLMs), Agentic AI frameworks, statistical/machine learning model development, Unix/shell scripting, CI/CD pipelines, and cloud deployment best practices.

Skills

• Ab Initio
• Spark (Core, SQL, DataFrames, Datasets)
• Python/scripting
• GCP/cloud platforms
• Teradata
• Hadoop
• Hive
• Autosys
• Airflow
• BigQuery
• SQL/PL-SQL
• Distributed systems
• MongoDB
• Real-time data streaming (Kafka)
• AI/ML pipelines (Apache Spark, Sparkling Water, H2O, LLMs)
• Agentic AI frameworks
• Statistical/machine learning model development
• Unix/shell scripting
• CI/CD pipelines
• Cloud deployment best practices