Data Engineer - Databricks

MACHINE LEARNING TECHNOLOGIES LLC · New York, NY · 3 wk ago

Information TechnologyContract

About the role

Manage and support batch processes and data pipelines in Azure Databricks and Azure Data Factory.

Monitor, troubleshoot, and resolve data and system issues related to trading applications and market data ingestion.

Develop, automate, and optimize ETL pipelines using Python, Spark, and SQL.

Manage FTP/SFTP file transfers between internal systems and external vendors.

Ensure data quality, completeness, and timeliness for downstream trading and reporting systems.

Collaborate with operations, application support, and infrastructure teams to resolve incidents and enhance data workflows.

Responsibilities

Manage and support batch processes and data pipelines in Azure Databricks and Azure Data Factory.
Monitor, troubleshoot, and resolve data and system issues related to trading applications and market data ingestion.
Develop, automate, and optimize ETL pipelines using Python, Spark, and SQL.
Manage FTP/SFTP file transfers between internal systems and external vendors.
Ensure data quality, completeness, and timeliness for downstream trading and reporting systems.
Collaborate with operations, application support, and infrastructure teams to resolve incidents and enhance data workflows.

Requirements

10+ years of experience in data engineering or production support within financial services or trading environments.
Hands-on experience with Azure Databricks, Azure Data Factory, and Azure Storage, Logic Apps, Fabric.
Strong Python and SQL programming skills.
Experience with Bloomberg data feeds (BPIPE, TSIP,SFTP).

Qualifications

US Citizen, Green Card, OPT-EAD, CPT, H-1B, H4-EAD, L2-EAD, GC-EAD

Skills

Azure Databricks
Azure Data Factory
Azure Storage
Logic Apps
Fabric
Python
SQL
Bloomberg data feeds (BPIPE, TSIP,SFTP)

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A