Lead Data Engineer

What You'll Do

• Design, build, and maintain Snowflake-based pipelines that produce and refresh our core consumer and household identity assets on a regular cadence.
• Write complex SQL and Python to transform, deduplicate, and enrich identity data at scale — including direct work with PII fields such as names, addresses, emails, and phone numbers.
• Investigate data anomalies and quality issues at a record level, tracing match decisions back to source signals and surfacing root causes.
• Build and maintain data models that represent consumer and household identity linkage across multiple input sources.
• Mentor and partner with senior engineers and data scientists to enhance our AI-assisted matching engine — contributing to feature design, scoring logic, model evaluation, and threshold tuning.
• Implement and test matching algorithm improvements — both AI-driven and rule-based — and measure their real impact on precision, recall, and overall asset quality.
• Build evaluation tooling: ground-truth comparisons, match quality dashboards, and regression detection across engine versions.
• Help drive the evolution of our matching pipeline toward more intelligent, AI-augmented identity resolution, actively using AI tools as part of your day-to-day engineering workflow.

Collaboration & Delivery

• Work cross-functionally with Data Science, Product, and downstream engineering teams to translate identity requirements into reliable, scalable solutions.
• Participate in code reviews and architectural discussions; apply engineering best practices across the full delivery lifecycle — design, implement, test, and deploy via CI/CD.
• Document data models, pipeline logic, and algorithm decisions clearly for both technical and non-technical audiences.
• Support QA processes and on-call responsibilities for production identity asset pipelines.
• Build automated validation frameworks and quality tracking pipelines that continuously monitor asset health — including data completeness, match consistency, and anomaly detection — and surface results through clear, actionable reporting.

Required

• 4+ years of data engineering or software engineering experience, with a focus on data-intensive systems.
• Strong Python skills — you write clean, well-structured code and are comfortable building data processing logic from scratch.
• Deep Snowflake fluency: data modeling, complex querying, Streams and Tasks, performance tuning, and preferably Snowpark for Python-native workloads.
• Strong SQL fundamentals and comfort working with large, messy, real-world datasets — you know how to interrogate data and know when not to trust it.
• Some experience or genuine curiosity around identity matching, deduplication, record linkage, or data quality at scale.
• Comfort working with PII-class data responsibly, with awareness of data governance and privacy best practices.
• Familiarity with version control (Git), Agile delivery, and CI/CD pipelines.
• Comfort applying AI tools in day-to-day engineering work — including prompt engineering, LLM-assisted data processing, and AI-augmented pipeline logic.

Nice to Have

• Hands-on exposure to matching algorithms — deterministic, probabilistic, or ML/AI-based — and experience evaluating or tuning their performance.
• Experience building agentic workflows and working with MCP servers.
• Some Java experience; comfort with JVM-based tooling is a plus.
• Familiarity with consumer or household identity signals: name, address, email, phone, and cross-source linkage.
• Cloud experience, preferably AWS; Azure or GCP welcome.
• Unix/Bash comfort for scripting and day-to-day environment work.