Analyst, Lead Data Engineering

Responsibilities

• Design and implement reliable data pipelines to integrate disparate data sources into a single Data Lakehouse.
• Design and implement data quality pipelines to ensure data correctness and building trusted datasets.
• Design and implement a Data Lakehouse solution which accurately reflects business operations.
• Avoid and address data platform performance issues, including tuning and supporting physical data models such as partitioning and compaction.
• Provide guidance in data visualizations and reporting efforts to ensure solutions align with business objectives.
• Automate and optimize the data lifecycle, find insights from raw data, and apply DevOps principles to data pipelines.
• Work with business leaders to deliver custom software solutions meeting data needs.
• Build and support a data platform for data engineering teams to build, deploy, and manage applications.

Requirements

• 5+ years of experience as a Data Engineer designing and maintaining data pipeline architectures.
• 5+ years of in-depth programming experience in Python and SQL.
• 5+ years of software development lifecycle experience with software engineering, development, testing, version control, refactoring, and deployment.
• Experience with common Python Data Engineering packages including pandas, Numpy, Pyarrow, Pytest, Scikit-Learn, and Boto3.
• Experience in implementing a Data Lakehouse using Apache Iceberg or Delta Lake.
• Experience with data platform architecture responsible for high-level design, strategy, and implementation of data infrastructure, including data modelling, designing scalable architectures, and ensuring data governance, security, and compliance.
• Knowledgeable of modern data platform technologies including Apache Airflow, Kubernetes, and S3 Object Storage.
• Experience with AWS, Snowflake, dbt, and Airbyte is preferred.
• Experience with infrastructure as code, building consistent and repeatable cloud infrastructure.