Lead Principal Data Engineer

Essential Functions

• Own the end-to-end architecture of the Microsoft Fabric environment, including Lakehouse and Warehouse design, the medallion (bronze/silver/gold) pattern, and Direct Lake semantic datasets.
• Design and govern OneLake as the single, organization-wide data lake, managing domains, shortcuts, data mirroring, and Delta Lake/Parquet storage standards.
• Own data ingestion and transformation at scale using Fabric Data Factory pipelines, Dataflows Gen2, and Spark notebooks.
• Define and enforce engineering standards: Git-based source control, CI/CD and deployment pipelines, environment promotion, and code review.
• Establish monitoring, alerting, and observability across pipelines; ensure SLAs for data freshness and availability.
• Optimize Fabric capacity units, and storage layout, for both performance and cost.
• Implement data quality, lineage, and automated testing frameworks across the platform.
• Structure governed, well-modeled data in OneLake to serve as grounding for generative and agentic AI use cases.
• Partner with the AI Engineering Manager to integrate Azure AI services and Microsoft Foundry (formerly Azure AI Foundry) into production data workflows, including retrieval (RAG) pipelines over OneLake.
• Implement security and governance controls, row/column-level security and PII handling, required for AI at scale.
• Direct and set the technical agenda for outsourced engineering partners, shifting them from running the platform to extending it on scoped, time-bound projects.
• Serve as technical hiring manager and interviewer for subsequent engineering hires; mentor junior analytics and data engineers.
• Capture and retain institutional knowledge of our architecture and design patterns in-house.

Technical Competencies (required)

• Microsoft Fabric — deep, hands-on experience with Lakehouse, Warehouse, Data Factory pipelines, Dataflows Gen2, Spark notebooks, and Direct Lake.
• OneLake — demonstrated experience designing and governing OneLake, including domains, shortcuts, mirroring, and Delta Lake/Parquet standards.
• Azure AI — experience integrating Azure AI services / Azure OpenAI into data and retrieval pipelines.
• Microsoft Foundry (Azure AI Foundry) — working knowledge of the model catalog and RAG/agent grounding for GenAI workloads (applicable to this role).
• Azure data platform — Synapse, Azure Data Lake Storage, Azure SQL, Service Bus, and Spark in production.
• Engineering practice — strong CI/CD, Git-based source control, and infrastructure-as-code.

Desired Education And Experience

• 8+ years in data engineering / data architecture, including 2+ years in a lead or principal capacity.
• Expert SQL and Python; strong Spark (PySpark) experience.
• Proven track record building and operating a Microsoft Fabric / OneLake platform in production (Fabric-specific experience strongly preferred and commands a premium).
• Experience managing or directing outsourced/consulting engineering teams.
• Exposure to Microsoft Purview, data governance, and security/compliance frameworks.
• Bachelor’s in Computer Science, Engineering, or related field; Master’s or relevant Microsoft certifications (e.g., Fabric Analytics Engineer, Azure Data Engineer Associate) a plus.