Data Engineer
About the job
The position of Data Engineer for the Talon Ark Program is responsible for maintaining and designing new features for the data pipeline. The pipeline end-to-end goes from our Analyst to our Data Scientist on the program. This position is responsible for maintaining the collection of data from various sources, presenting data to the analyst, gathering and curating the inputs from the analyst, and producing the vehicle through which the Data Scientist can use the data created. This position requires a high level of problem-solving skills, and the ability to research using vast amounts of resources to arrive at a solution. The position works closely with the Analyst and Data Scientist on the program to create a viable solution for the customer. What is produced in this position is mainly for the sake of serving the Analyst and Data Scientist. However, creating and maintaining good data is a key factor in making this program successful.
About you
- Python [ Pandas, NumPy, PyTorch, Ultralytics]
- Docker
- Understanding of python data manipulation packages [Pandas, NumPy, PyTorch, Dask]
- Use of APIs (some that are documented and not documented)
- Understanding of relational database design and maintenance
- Understanding of Software Design Patterns and their implementations
- Able to do own research and implement new solutions
Preferred Skills
- Python big data manipulation libraries (ex. PyArrow)
- Python distributed processing libraries (ex. Ray)
- Continuous Integration/Continuous Development (CICD)
- Scales pipeline architecture