Director Data Engineering

About the role

The Director, Data Engineering is responsible for leading the design, development, and operation of enterprise data engineering platforms and pipelines that support analytics, reporting, data products, and downstream consumption.

Responsibilities

• Own the design, development, and operations of Subway’s enterprise data engineering platform on AWS Databricks, including migration strategy from the existing Amazon Redshift environment.
• Lead the delivery of Medallion Architecture (Bronze / Silver / Gold) pipelines across all QSR domains — Restaurant, Sales, Inventory, Guest, Marketing, and Supply Chain.
• Oversee development of Delta Live Tables, batch and streaming pipelines, and feature engineering workflows across Databricks and Microsoft Fabric.
• Architect and deliver a self-service data platform leveraging Unity Catalog, RBAC/ABAC governance, and Delta/Iceberg interoperability to enable domain teams to independently build and consume trusted data products.
• Define and execute the platform migration roadmap — phasing workloads from Redshift to Databricks/Snowflake with minimal business disruption.
• Partner with the Enterprise Data & AI Architect to implement and evolve a Data Mesh architecture, enabling domain teams to own and publish certified data products.
• Drive adoption of DataOps practices including CI/CD for data pipelines, automated testing, data contract enforcement, and pipeline observability.
• Ensure platform reliability, SLA adherence, and proactive incident management for all production data pipelines and data products.
• Implement infrastructure-as-code and environment management for Databricks workspaces, clusters, and job orchestration.
• Establish on-call processes, runbooks, and escalation paths for Tier-1 data platform incidents, targeting ≤99.5% pipeline uptime.
• Define and enforce data quality standards and SLAs across all engineering pipelines and analytics products — partnering with the Data Governance function and Data Product Owners.
• Implement data observability frameworks (e.g., Monte Carlo, Databricks Lakehouse Monitoring) to proactively detect and resolve data freshness, completeness, and accuracy issues before they impact business decisions.
• Partner with the Data Governance team to ensure pipelines and data products are cataloged, lineage-tracked, and classified in Unity Catalog and/or Microsoft Purview.
• Champion metadata management and data contract enforcement as first-class engineering practices across the platform.
• Collaborate with Data Product Managers to translate business outcomes into engineering priorities and delivery plans.
• Enable Analytics, BI, and Data Science teams with high-quality, well-modeled data assets that accelerate insight generation.
• Communicate platform health, tradeoffs, and delivery status to business and technology stakeholders through regular forums and reporting.
• Serve as the primary data engineering partner to product and domain leaders — translating business needs into engineered, scalable data solutions.
• Lead vendor relationships and participate in technology evaluations, RFPs, and contract negotiations for data platform tooling and services (Databricks, Snowflake, dbt, Fivetran, etc.).
• Manage the data engineering budget — including cloud infrastructure costs (AWS, Azure), tooling licenses, DBU consumption, and contractor spend.
• Develop and present quarterly technology roadmaps to senior leadership, aligning platform investments to enterprise data and AI strategy.
• Collaborate cross-functionally with Application Engineering, InfoSec, and Enterprise Architecture to ensure platform alignment with enterprise standards.
• Lead and develop managers, leads, and senior data engineers across onshore and offshore delivery models.
• Build strong engineering capability through hiring, coaching, and career development — scaling the team to 10–20+ engineers (FTEs and contractors).
• Foster a culture of ownership, reliability, and continuous improvement within the data engineering function.
• Define and track KPIs including pipeline reliability, data freshness, SLA adherence, and cost efficiency.
• Drive operational excellence, automation, and cost optimization across all data platforms and infrastructure.
• Support incident analysis and preventative improvements to continuously raise the reliability bar.

Qualifications

• Strong experience leading data engineering teams and platforms in a technology-forward organization.
• Deep understanding of modern data architectures — cloud data platforms, batch/stream processing, lakehouse, and data mesh.
• Experience balancing delivery speed with governance, reliability, and scalability in complex environments.
• Ability to influence across Product, Analytics, Platform, and Security functions.
• Proven people leadership, delivery management, and vendor management skills.
• Strong communication skills and executive presence; comfortable presenting to C-suite stakeholders.
• Expert in SQL, Python, and PySpark; working knowledge of Scala is a plus.
• Proficient with orchestration tools (Databricks Workflows, Airflow, Azure Data Factory).
• Experience with data quality and observability tools (Great Expectations, Monte Carlo, Soda, or equivalent).
• Experience with dbt, Fivetran/Airbyte, or similar ELT frameworks.
• Familiarity with Infrastructure-as-Code (Terraform, Pulumi, or ARM/Bicep).
• Bachelor’s degree required (Computer Science, Engineering, Data, Information Systems, or related field).
• Advanced degree (Master’s or MBA) preferred.
• 8–12 years of experience in data engineering or adjacent platform roles.
• 3–5 years of experience leading teams or enterprise-scale data capabilities.
• Demonstrated experience delivering production-grade data platforms on Databricks (strongly preferred), Snowflake, and/or Amazon Redshift.
• Experience operating cloud-based, distributed data platforms at enterprise scale.
• Experience in complex, matrixed enterprise environments — QSR, Retail, CPG, or Franchise industries preferred.
• Preferred Qualifications:
• Databricks Certified Data Engineer Professional or equivalent certification.
• Experience leading large-scale data platform migrations (e.g., Redshift to Databricks, on-prem to cloud).
• Exposure to ML/AI platform engineering — feature stores, model serving, and MLOps integration.
• Experience with FinOps practices for cloud cost optimization and chargeback models.
• Familiarity with Microsoft Purview, Collibra, or Alation for enterprise data governance.