ETL Data Platform Developer

About the role

Design and implement ETL pipelines using Microsoft Fabric (Dataflows, Pipelines, Lakehouse ,warehouse, sql) and Azure Data Factory.
Build and maintain a metadata driven Lakehouse architecture with threaded datasets to support multiple consumption patterns.
Develop agent specific data lakes and an orchestration layer for an uber agent that can query across agents to answer customer questions.
Enable interactive data consumption via Power BI, Azure OpenAI, and other analytics and AI tools.
Ensure data quality, lineage, and governance across all ingestion and transformation processes.
Collaborate with product teams to understand data needs and deliver scalable solutions.
Optimize performance and cost across storage and compute layers.

Responsibilities

• Design and implement ETL pipelines using Microsoft Fabric (Dataflows, Pipelines, Lakehouse ,warehouse, sql) and Azure Data Factory.
• Build and maintain a metadata driven Lakehouse architecture with threaded datasets to support multiple consumption patterns.
• Develop agent specific data lakes and an orchestration layer for an uber agent that can query across agents to answer customer questions.
• Enable interactive data consumption via Power BI, Azure OpenAI, and other analytics and AI tools.
• Ensure data quality, lineage, and governance across all ingestion and transformation processes.
• Collaborate with product teams to understand data needs and deliver scalable solutions.
• Optimize performance and cost across storage and compute layers.

Requirements

• 7+ years of experience in data engineering with a focus on Microsoft Azure Data stack and Fabric technologies.
• Strong expertise in:
• Microsoft Fabric (Lakehouse, Dataflows Gen2, Pipelines, Notebooks)
• Azure Data Factory, Azure SQL, Azure Data Lake Storage Gen2
• Power BI and/or other visualization tools
• SQL, Python and PySpark/ Scala
• Experience working with structured and semi structured data (JSON, XML, CSV, Parquet).
• Proven ability to build metadata driven architectures and reusable components.
• Familiarity with agent based architectures and conversational AI integration is a plus.
• Strong understanding of data modeling, data governance, and security best practices.

Qualifications

• 7+ years of experience in data engineering with a focus on Microsoft Azure Data stack and Fabric technologies.
• Strong expertise in:
• Microsoft Fabric (Lakehouse, Dataflows Gen2, Pipelines, Notebooks)
• Azure Data Factory, Azure SQL, Azure Data Lake Storage Gen2
• Power BI and/or other visualization tools
• SQL, Python and PySpark/ Scala
• Experience working with structured and semi structured data (JSON, XML, CSV, Parquet).
• Proven ability to build metadata driven architectures and reusable components.
• Familiarity with agent based architectures and conversational AI integration is a plus.
• Strong understanding of data modeling, data governance, and security best practices.

Skills

• Executive written and oral communication
• Microsoft Fabric (Lakehouse, Dataflows Gen2, Pipelines, Notebooks)
• Azure Data Factory, Azure SQL, Azure Data Lake Storage Gen2
• Power BI and/or other visualization tools
• SQL, Python and PySpark/ Scala

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay

Depending on Experience (DOE)

Schedule

N/A

Benefits

N/A

Pay