Lead Data Engineer - AZ, NC

About the role

This role involves consulting on complex initiatives with broad impact and large-scale planning for Data Engineering. The Lead Data Engineer will lead data engineering efforts across enterprise data platforms, focusing on ETL development, cloud migration, and large-scale data processing solutions. This role supports the enterprise data warehouse, enabling scalable data pipelines and advanced analytics capabilities.

Responsibilities

• Lead moderately complex initiatives within technology and contribute to large-scale data processing framework initiatives.
• Design, develop, and maintain ETL pipelines using Ab Initio and Spark.
• Build and optimize data pipelines on Google Cloud Platform (GCP), including BigQuery, Airflow, and SparkFlow.
• Oversee data integration work, including data modeling, data warehouse maintenance, and script development.
• Develop and execute complex SQL queries across large datasets, particularly with Teradata and Oracle.
• Support the migration of data workloads to GCP.
• Perform code reviews with a focus on performance, testability, and maintainability.
• Resolve moderately complex issues and lead teams to meet data engineering deliverables.
• Work within an Agile environment to deliver end-to-end data engineering solutions.
• Collaborate with cross-functional teams to support enterprise analytics initiatives.

Requirements

• Experience: 6+ years of Data Engineering experience, or equivalent demonstrated through a combination of work experience, training, military experience, or education.
• Technical Skills: 6+ years of experience with Ab Initio, Spark Core, Spark SQL, Data Frames, and Datasets. 3+ years of experience with Python or other scripting languages. 3+ years of data warehouse experience, including Teradata, SQL, and PL/SQL. Hands-on experience with a cloud platform such as GCP, AWS, or Azure. Strong experience with Teradata, Hadoop, Hive, Autosys, Airflow, BigQuery, and SparkFlow. Understanding of distributed systems, parallel processing, and Spark architecture. Experience with end-to-end data engineering projects in an Agile environment.

Preferred Qualifications

• Prior experience leading teams and managing major programs.
• Knowledge of non-relational DBMS platforms (e.g., MongoDB) and real-time data streaming via Kafka.
• Working knowledge of developing end-to-end ML/AI pipelines using tools like Apache Spark, Sparkling Waters, or H2O.
• Experience working with Large Language Models (LLMs) and Agentic AI frameworks.
• Proficiency in large-scale data processing, various file formats, Unix commands, and shell scripting.
• Familiarity with CI/CD pipelines, infrastructure as code, and cloud deployment best practices.

Benefits

Everforth Apex offers a range of supplemental benefits, including medical, dental, vision, life, disability, and other insurance plans that offer an optional layer of financial protection. We offer an ESPP (employee stock purchase program) and a 401K program which allows you to contribute typically within 30 days of starting, with a company match after 12 months of tenure. Everforth Apex also offers a HSA (Health Savings Account on the HDHP plan), a SupportLinc Employee Assistance Program (EAP) with up to 8 free counseling sessions, a corporate discount savings program and other discounts. In terms of professional development, Everforth Apex hosts an on-demand training program, provides access to certification prep and a library of technical and leadership courses/books/seminars once you have 6+ months of tenure, and certification discounts and other perks to associations that include CompTIA and IIBA. Everforth Apex has a dedicated customer service team for our Consultants that can address questions around benefits and other resources, as well as a certified Career Coach. You can access a full list of our benefits, programs, support teams and resources within our ‘Welcome Packet’ as well, which an Everforth Apex team member can provide.

Pay

$65/hr - $70/hr

Schedule

Hybrid (3 days onsite)