Lead AI Data Engineer – Platform & Analytics

Key Responsibilities

• Architecture & Design: Architect and maintain a scalable medallion data lake house structure (Bronze, Silver, Gold layers) utilizing Apache Iceberg table formats.
• Multi-Engine Data Consumption: Build and optimize curated, semantic Gold-layer data products designed for seamless, high-performance consumption across an open ecosystem of multiple compute and query engines – including Snowflake, Databricks (Spark), Trino/Starburst, AWS Athena, and Presto.
• Pipeline Orchestration: Design, develop, and manage complex, resilient data workflows and DAGs using Apache Airflow.
• Data Pipeline Development: Build, deploy, and monitor robust end-to-end ETL/ELT pipelines to ingest diverse semiconductor data streams into the data lake.
• Data Modeling: Design and implement high-performance consumption data models, ensuring clean, transformed, and production-ready datasets.
• SQL Optimization: Write and tune complex, highly optimized SQL queries for data transformation, analysis, and performance benchmarking.
• AI-Driven Delivery: Utilize generative AI coding assistants and automation tools to accelerate pipeline development, documentation, and testing.
• Data Governance: Implement data quality checks, schema evolution rules, and governance practices inherent to Iceberg and Snowflake environments.

Preferred Qualifications

• Experience: Dedicated experience in software, data engineering and data management.
• Workflow Orchestration: Strong hands-on experience scheduling and monitoring production-grade pipelines with Apache Airflow.
• Lakehouse Expertise: Proven track record of designing medallion architectures and working extensively with the Apache Iceberg table format.
• Tech Stack Mastery: Advanced proficiency with Snowflake and deep hands-on experience building transformation models in DBT Core.
• Core Skills: Expert-level SQL & Python knowledge and complete mastery of modern ETL/ELT patterns and design principles.
• Ability to write high quality code with keen attention to detail.
• Experience with modern concurrent programming and threading APIs.
• Experience with software development processes and tools such as debuggers, source code control systems (GitHub) and profilers.
• Experience with AI tools (e.g., GitHub Copilot, Snowflake Cortex, LLM APIs, Claude Code, etc.) to speed up code development and problem-solving.
• Problem Solving: Experience delivering multiple enterprise-grade, production-level, end-to-end data pipeline solutions from scratch.