Data Engineer - Commodities

Key Responsibilities

• Design and implement end-to-end ETL workflows in Python and SQL to ingest and transform commodities data from multiple vendors and internal sources.
• Build and maintain standardized data models, schemas, and metadata that make commodities datasets easy to understand and discover within the platform.
• Use Airflow (or similar tools) to schedule, monitor, and manage data pipelines, ensuring reliability and timely delivery.
• Implement robust validation, reconciliation, and anomaly-detection checks to ensure data completeness, correctness, and consistency.
• Leverage AI to automate schema inference across structured and semi-structured data sources, manage schema drift, and accelerate development of scalable ingestion pipelines.
• Apply AI-driven data quality, observability, and documentation capabilities to detect anomalies, monitor data health, and generate clear lineage and technical documentation across complex data workflows.
• Leverage Git, GitHub Actions, and automated testing (PyTest) to maintain high-quality code and repeatable deployments.
• Partner with commodities PMs, researchers, and data strategists to understand use cases and continuously refine datasets, definitions, and documentation.

Required Qualifications

• 4 years of experience in data engineering, analytics engineering, or similar roles focused on building and maintaining ETL pipelines.
• Strong skills in Python and SQL, with experience working with large datasets and complex transformations.
• Hands-on experience with Airflow or other workflow schedulers.
• Familiarity with version control (Git), CI/CD pipelines (GitHub Actions or equivalent), and test automation (e.g., PyTest).
• Strong attention to detail, data quality and documentation; ability to reason for edge cases and data integrity.
• Ability to work independently, communicate clearly with both technical and non-technical stakeholders, and manage work across multiple concurrent initiatives.

Preferred Qualifications

• Knowledge of commodities markets and commodities data (e.g., weather, supply/demand, storage, freight, flows).
• Experience with data warehousing technologies (e.g., Snowflake, columnar storage formats, or analytic databases).
• Prior experience in a financial services, trading, or research driven environment.
• Exposure to data catalog / data governance tools and best practices.