Data Engineer

About the role

Design, build, and maintain scalable data pipelines for batch and near-real-time processing. Develop robust data integration solutions connecting databases, APIs, cloud platforms, and external data sources. Write production-quality Python code for data ingestion, transformation, automation, and pipeline development. Build and maintain data transformation workflows using dbt. Develop and optimize complex SQL queries, stored procedures, and data transformations.

Responsibilities

• Manage automated file transfers with external partners/vendors via SFTP, including scripting secure uploads/downloads and troubleshooting.
• Build, schedule, monitor, and troubleshoot workflows using Apache Airflow.
• Design data models and structures to support analytics, reporting, and downstream applications.
• Implement data quality controls, validation, error handling, logging, and monitoring across pipelines.
• Collaborate with stakeholders and technical teams to understand data requirements and deliver scalable solutions.
• Troubleshoot pipeline failures, performance issues, and data discrepancies.
• Develop reusable frameworks and components for data ingestion and integration.
• Ensure data pipelines are secure, reliable, scalable, and maintainable.
• Participate in code reviews, testing, documentation, and deployment processes.