Director of Data Engineering & Platforms

Job Description

We are seeking a hybrid or remote Director of Data Engineering & Platforms to lead our data transformation initiatives and establish robust data architecture frameworks. This role reports to the AVP of Cloud Engineering and is responsible for designing, implementing, and maintaining our enterprise data ecosystem across bronze, silver, and gold data layers following Data Mesh and Data Vault methodologies. This role sits at the intersection of data engineering and AI enablement, where the decisions made in the data layer directly shape the quality of what AI systems can deliver.

Key Responsibilities

• Strategic Leadership & Architecture
  - Lead the development and implementation of our data engineering strategy, architecture roadmap, and technical standards.
  - Oversee the design and evolution of our data ecosystem utilizing Data Vault methodologies and Data Mesh principles.
  - Establish governance and quality frameworks across Bronze (raw), Silver (transformed), and Gold (consumption-ready) data layers.
  - Partner with Product Management to align data platform capabilities with business objectives and market demands.
  - Drive the technical roadmap for data integration, transformation, and delivery systems, with explicit milestones for AI readiness.

• Technical Direction & Delivery
  - Provide technical leadership and oversight for the data engineering team, ensuring best practices in data pipelines, transformations, and delivery.
  - Oversee the design and implementation of Snowflake data architecture including warehousing, marts, and access patterns optimized for both BI and AI workloads.
  - Direct the development of robust ETL/ELT processes using Matillion, Python-based pipeline frameworks, and other modern data integration tools.
  - Guide the implementation of data quality monitoring, lineage tracking, and metadata management.
  - Establish standards for data modeling, transformation logic, and performance optimization.

• AI Data Infrastructure
  - Partner with AI/ML and product engineering teams to ensure the data layer supports LLM-powered applications reliably and at scale.
  - Provide architectural direction for retrieval and grounding pipelines, including vector stores, embedding workflows, and hybrid search infrastructure.
  - Define standards for data preparation for AI, covering metadata enrichment, context optimization, and semantic indexing.
  - Guide the team's evaluation and adoption of emerging AI-native data tools, including vector databases and LLM orchestration frameworks.

• AI Governance & Risk
  - Establish governance frameworks for AI data use, including data lineage into models, PII controls upstream of LLM consumption, and output auditability.
  - Define the organization's standards for acceptable AI data quality thresholds and remediation workflows.
  - Partner with Security and Compliance to ensure AI data pipelines meet regulatory and privacy requirements.

• Team Leadership & Development
  - Build, mentor, and lead a high-performing data engineering team with strong core data engineering fundamentals and a growing fluency in modern AI infrastructure.
  - Collaborate cross-functionally with Product Management, Data Analytics, Data Systems, Cloud Engineering, and product teams.
  - Foster a culture of innovation, continuous improvement, and technical excellence where AI is a tool the team uses daily, not a project they hand off.
  - Develop talent through coaching, training, and career development opportunities, with an emphasis on AI-era skills including Python, vector search, and agentic pipeline concepts.
  - Promote adaptive methodologies and DevOps practices within the data engineering discipline.

• BI & Analytics Enablement
  - Oversee the technical implementation of Power BI reporting solutions and analytics platforms.
  - Ensure data pipelines efficiently support BI reporting needs and business intelligence requirements.
  - Partner with Data Analytics teams to optimize data structures for analytical workloads.
  - Guide the design of data models that enable self-service analytics and reporting.
  - Establish patterns for efficient and secure data access across the organization.

• Innovation & Future-State Planning
  - Evaluate emerging technologies and methodologies for potential integration into our data platform, with particular attention to AI/ML tooling and agentic workflow frameworks.
  - Lead proof-of-concepts and pilots for innovative data solutions, including AI-powered pipeline automation and LLM-grounded analytics.
  - Develop the technical foundation to support advanced analytics and machine learning initiatives.
  - Guide the evolution of our data architecture to support real-time and streaming use cases.
  - Stay current with industry trends and incorporate best practices into our data ecosystem.

Job Qualifications

• Bachelor's degree from an accredited institution or equivalent professional experience with demonstrated capability

• 8+ years of progressive experience in data engineering, data architecture, or related technical roles

• 5+ years of leadership experience managing data engineering teams and initiatives

• Extensive experience with modern data platforms, particularly Snowflake and cloud-based data solutions

• Deep understanding of data modeling techniques including Data Vault, dimensional modeling, and Data Mesh concepts

• Hands-on experience with ETL/ELT tools like Matillion and data integration patterns

• Strong knowledge of SQL Server, Cosmos DB, and database technologies

• Experience with Power BI or similar BI platforms and understanding of reporting architectures

• Proven track record implementing data governance, quality, and metadata management solutions

• Experience partnering with product teams and translating business requirements into technical solutions

• Demonstrated interest in AI/ML data infrastructure, whether through independent projects, coursework, or applied experimentation

• Ability to engage credibly with engineers building LLM-powered systems and make sound architectural decisions without being the implementer

Preferred Qualifications

• Bachelor’s or master’s degree in computer science, Information Systems, or a related field

• Proficiency in Python and experience with Spark or other data processing frameworks

• Knowledge of CI/CD practices and DevOps for data pipelines

• Has independently explored or prototyped with vector databases, RAG pipelines, or LLM grounding concepts

• Familiarity with LLM orchestration frameworks such as LangChain or LlamaIndex

• Exposure to agentic workflow concepts and the data contracts they require

• Experience with real-time data integration and streaming architecture

• Background in implementing data security and privacy controls

• Understanding of API design and microservices architectures

• Experience in insurance, financial services, or real estate industries