Data Engineer

Responsibilities

• Supporting Data Engineering Infrastructure: Contribute to the design, development, implementation and continuous improvement of our data engineering tools, workflows, processes, and platforms. This includes enhancing the architectural foundations and integrating new data management technologies.
• Writing Well-Structured Code: Develop clean, maintainable, well-documented code that adheres to best practices. Support best coding practices within Habitat's software, machine-learning, and data science teams.
• Enhance data engineering knowledge: Improve expertise within the software team and ensure their ability to support and collaborate on the data infrastructure.
• Data Quality Management: Continuously enhance data quality across multiple dimensions such as accuracy, availability, performance, and accessibility to ensure a clear understanding of data within the company.
• Providing backup/escalation to the tech-on-call team
• Communicating effectively across Software and Data Science teams

Requirements

• 3+ years of Python experience
• 3+ years of working in technical teams, building data pipelines, delivering productionised code, building/maintaining live applications, developing tooling and improving backtesting frameworks
• Experience in applying relational database design
• Proficiency with Orchestration and IaC in AWS (e.g. Terraform, Kubernetes, RabbitMQ, Airflow, Prefect), Git, containerisation (Docker), database management (e.g. Postgres, Alembic)
• Fluent in Python and its wider numerical ecosystem (e.g. Pandas, NumPy, Polars, Pydantic)

Nice to have

• 2+ years of orchestrating machine learning workflows
• Experience with OSS data warehousing tooling and management
• Cloud infrastructure experience
• Experience with monitoring frameworks (e.g. Prometheus)
• Experience archiving data to Parquet on S3 and creating tools for API/Grafana queries
• Centralising diverse datasets for analytics, visualisation and machine learning
• Experience with time-series forecasting and/or optimisation
• Familiarity with data visualisation and dashboards (e.g. Grafana, Superset)
• Familiarity with machine learning and associated techniques (feature engineering, boosting methods, LightGBM)