Data Engineer

About the role

The Data Engineer designs, builds, and maintains scalable data architecture, pipelines, and platforms that support analytics, reporting, and AI/ML solutions across the organization. This role is responsible for preparing, integrating, governing, and optimizing data from multiple sources to ensure reliable, accessible, and high-quality data assets for business and technical use. The Data Engineer works closely with data scientists, software engineers, analysts, and business stakeholders to deliver cloud-based data solutions that support operational goals, innovation, and emerging AI capabilities.

Responsibilities

• Design, develop, and maintain scalable data pipelines, data models, and lakehouse or data warehouse structures to support analytics, reporting, and AI/ML workloads.
• Build, test, and optimize ETL/ELT workflows that ingest, transform, and deliver data from multiple structured, semi-structured, and unstructured sources.
• Support AI/ML initiatives by preparing and managing datasets for feature engineering, model training, inference, and other AI-enabled applications.
• Implement and maintain data quality controls, validation processes, monitoring, and observability practices to improve reliability, accuracy, and performance of data pipelines.
• Manage data integration processes across internal and external systems to ensure timely, secure, and efficient data availability.
• Support data governance practices, including lineage, cataloging, access management, and documentation, to promote trusted and well-controlled data assets.
• Develop and deploy cloud-based data engineering solutions using modern platforms, distributed processing tools, and containerized environments where applicable.
• Build and maintain batch and near real-time data pipelines to meet business, reporting, and application requirements.
• Collaborate with cross-functional teams to gather requirements, translate business needs into technical solutions, and support enterprise data initiatives.
• Troubleshoot, enhance, and continuously improve data infrastructure, workflows, and related tools to increase scalability, efficiency, and business value.
• Remain current with developments in data engineering, cloud platforms, and AI-enabling technologies and apply practical improvements where beneficial.
• Comply with applicable ABS Health, Safety, Quality, and Environmental Management System requirements and other internal policies and procedures.

Requirements

• Degree in a technical field or equivalent combination of education and experience
• 8+ years of relevant experience in data engineering or a closely related discipline, including significant experience building and maintaining cloud-based data pipelines and architectures.
• Databricks certification (preferred)

Qualifications

• Strong knowledge of data engineering concepts, including data modeling, data integration, ETL/ELT design, and pipeline orchestration.
• Strong programming skills in Python and SQL, with the ability to build, test, and maintain production-grade data workflows.
• Experience working with cloud-based data platforms such as AWS, Azure, or Google Cloud.
• Experience designing, implementing, and supporting data warehouse or lakehouse architectures such as Databricks, Snowflake, Redshift, or similar technologies.
• Knowledge of distributed data processing frameworks and tools, such as Spark or equivalent technologies.