Lead Data Engineer

Key Skills Required

• Experience using and implementing database technologies, storage layers, modeling frameworks, and data integration tooling.
• Able to follow strategy and established guidelines, make recommendations for enhancements or changes, and participate in discussions on future strategic design.
• Experience in optimization and performance tuning in all aspects of the platform and technology (database, pipelines, SQL statements, resource scaling).

Application Architecture

• Understanding of possible integration options between systems and applications.
• Understanding of orchestration for applications; experience with Airflow would be a bonus.

Security

• Adequate understanding of security models and controls.
• Able to implement and troubleshoot controls in platform and technology.

Data Engineering

• Use of azure data factory especially with metadata driven pipelines.
• Strong understanding of ETL patterns especially within a data warehouse environment.
• Items like merge and upsert, handling slowly changing dimensions, snapshot fact tables, etc.

Data Quality

• Good understanding of testing lifecycle from unit testing, to system integration testing, to user acceptance testing.
• Able to review and critique test plans and recorded results.
• Able to direct developers on necessary test scenarios if plans are lacking.
• Able to execute data quality checks for data products within the data warehouse. May include technical checks such as duplication checks, orphan record checks, and invalid surrogate key checks. May also include business rule checks.

Soft Skills

• Collaboration
• Driving sessions for requirement gathering and review for both functional and technical requirements.
• Documentation
• Able to produce clear and concise documentation, appropriately tailored for audience, for policies, standards, and procedures.

Pay and Benefits

Our Organization has a national salary range of $85,000- $150,000. DHL Supply Chain offers multiple benefits including Medical, Dental, Vision, Prescription, Discounted Stock Purchase, General Bonus Plan and a generous PTO policy.

About the Role

We are seeking a skilled Data Engineering Lead to oversee technical data projects and develop efficient data pipelines within the organization. This role is essential for optimizing data architecture to meet the needs of cross-functional teams, ensuring rapid data ingestion into global platforms while upholding data integrity and accessibility. The successful candidate will collaborate with stakeholders to promote data-driven decision-making and enhance data initiatives. As a subject matter expert, the Data Engineering Lead will mentor team members at all levels, fostering their development and ensuring adherence to best practices in data engineering.

Responsibilities

• Lead technical data engineering projects, ensuring rapid data ingestion into regional and global platforms.
• Guide the development, testing, and maintenance of large-scale data pipelines while adhering to best practices and quality standards.
• Support software developers, database architects, data analysts, and data scientists in enhancing data accessibility and usability.
• Maintain consistency in data delivery architecture across projects by following established reference architecture.
• Recommend and implement optimized data pipeline architectures for cross-functional teams.
• Design and deploy data flow processes that meet the needs of various systems and products.
• Assemble complex data sets to meet business requirements and support strategic initiatives.
• Identify and implement internal process improvements for data reliability, efficiency, and quality.
• Mentor team members to foster development and ensure adherence to data engineering best practices.
• Collaborate with the data analytics community to integrate new data sources and develop operational procedures with operations teams.

Requirements

• Undergraduate Degree in Computer Science, Logistics, Mathematics, Statistics, or related field.
• 3+ years of experience in data engineering or equivalent role.
• Strong understanding of SQL and database management systems (e.g., MySQL, PostgreSQL, SQL Server).
• Proven experience in data pipeline development and management in Azure environments.
• Proficient with Azure Data Factory, Azure Databricks, or Azure Synapse Analytics.
• Familiar with data warehousing solutions and large datasets.
• Strong understanding of SQL and database management systems (e.g., MySQL, PostgreSQL, SQL Server).
• Proficient in programming languages such as Python, Java, or Scala.
• Knowledge of data modeling, ETL processes, and data warehousing concepts in Azure.
• Experience with data visualization tools (e.g., Power BI).
• Understanding of data governance and quality principles.
• Experience leading data initiatives.
• Able to mentor and support data engineers and guide project direction.
• Knowledge of machine learning concepts and tools for advanced analytics.
• Understanding of containerization and orchestration technologies (e.g., Docker, Kubernetes).