Lead Data Engineer (contract)

Responsibilities

• Lead moderately complex technology initiatives and contribute to enterprise-scale data processing frameworks
• Design, build, and maintain optimized, highly available data pipelines for analytics and reporting
• Analyze and solve complex business, operational, and technical challenges involving multiple variables
• Oversee data integration efforts, including:
• Data modeling
• Data warehouse and analytics environment management
• Development of data integration and analysis scripts
• Ensure adherence to data governance, policies, procedures, and compliance standards
• Troubleshoot and resolve complex data engineering issues while guiding teams to successful delivery
• Conduct code reviews with emphasis on scalability, testability, and coverage
• Collaborate with cross-functional teams and leadership to achieve strategic data initiatives

Qualifications

• Strong background in Data Engineering or equivalent experience
• Expertise in Ab Initio and Apache Spark (Core, SQL, DataFrames, Datasets)
• Proficiency in Python or scripting languages
• Experience with cloud platforms (GCP, AWS, or Azure)
• Strong SQL and data warehousing skills (Teradata, PL/SQL, Oracle)
• Experience with Hadoop ecosystem tools (Hive, Airflow, BigQuery, SparkFlow, Autosys)
• Deep understanding of distributed systems and Spark architecture
• Experience delivering end-to-end data engineering solutions in Agile environments
• Strong communication and collaboration skills
• Experience leading teams and large-scale initiatives
• Knowledge of NoSQL databases (MongoDB) and streaming (Kafka)
• Experience with ML/AI pipelines and tools (Spark, H2O, LLMs, Agentic AI, Vertex AI)
• Ability to optimize real-time, high-throughput data pipelines
• Familiarity with CI/CD, infrastructure as code, and cloud deployments
• Strong Unix and shell scripting skills
• Advanced ETL experience using Ab Initio, Airflow, SparkFlow, BigQuery