Data Engineer
Habitat Energy · Austin, TX · 2 wk ago
HybridEngineeringFull-time
Responsibilities
- Supporting Data Engineering Infrastructure: Contribute to the design, development, implementation and continuous improvement of our data engineering tools, workflows, processes, and platforms. This includes enhancing the architectural foundations and integrating new data management technologies.
- Writing Well-Structured Code: Develop clean, maintainable, well-documented code that adheres to best practices. Support best coding practices within Habitat's software, machine-learning, and data science teams.
- Enhance data engineering knowledge: Improve expertise within the software team and ensure their ability to support and collaborate on the data infrastructure.
- Data Quality Management: Continuously enhance data quality across multiple dimensions such as accuracy, availability, performance, and accessibility to ensure a clear understanding of data within the company.
- Providing backup/escalation to the tech-on-call team
- Communicating effectively across Software and Data Science teams
Requirements
- 3+ years of Python experience
- 3+ years of working in technical teams, building data pipelines, delivering productionised code, building/maintaining live applications, developing tooling and improving backtesting frameworks
- Experience in applying relational database design
- Proficiency with Orchestration and IaC in AWS (e.g. Terraform, Kubernetes, RabbitMQ, Airflow, Prefect), Git, containerisation (Docker), database management (e.g. Postgres, Alembic)
- Fluent in Python and its wider numerical ecosystem (e.g. Pandas, NumPy, Polars, Pydantic)
Nice to have
- 2+ years of orchestrating machine learning workflows
- Experience with OSS data warehousing tooling and management
- Cloud infrastructure experience
- Experience with monitoring frameworks (e.g. Prometheus)
- Experience archiving data to Parquet on S3 and creating tools for API/Grafana queries
- Centralising diverse datasets for analytics, visualisation and machine learning
- Experience with time-series forecasting and/or optimisation
- Familiarity with data visualisation and dashboards (e.g. Grafana, Superset)
- Familiarity with machine learning and associated techniques (feature engineering, boosting methods, LightGBM)