Lead Data Engineer

Responsibilities

• Lead design, implementation, and optimization of large-scale data processing using Scala, Spark, SQL, and modern data platforms for attribution.
• Design and operate trusted data pipelines handling advertiser, customer, and measurement datasets across AWS, GCP, Azure, and approved ecosystems.
• Collaborate with Product, Data Science, Security, Privacy, and Platform Engineering to deliver privacy-preserving attribution, measurement, forecasting, and analytics.
• Own scalable batch and streaming workflows using orchestration frameworks and cloud-native services.
• Implement data classification, access controls, and privacy-preserving techniques aligned with security and compliance requirements.
• Drive clean-room and trusted data-sharing environments, exposing only aggregated or privacy-protected outputs.
• Build observability, monitoring, alerting, and operational tooling for reliability, performance, and compliance.
• Troubleshoot complex platform, performance, and pipeline issues across distributed systems.
• Influence technical design, architecture, and best practices in partnership with senior engineering leadership.
• Mentor engineers, lead design and code reviews, and provide technical leadership.
• Ensure all sensitive processing occurs within controlled, auditable boundaries with no unintended egress of PII or proprietary signals.
• Define data handling standards and document trust boundaries, data contracts, lineage, and permitted movement between zones.
• Implement encryption, key management, and secure handling using cloud-native security and governance services.
• Support audits, compliance, governance reviews, and secure data-sharing initiatives.

Requirements

• 8+ years in Data Engineering with strong Scala and extensive Apache Spark on AWS and/or GCP.
• Strong Python for pipelines, tooling, automation, and infrastructure modules.
• Advanced SQL across relational, cloud warehouses, and lakehouse platforms handling TB-scale datasets.
• Design, build, and maintenance of batch and streaming pipelines.
• Data warehousing, dimensional modeling, data quality, partitioning, and performance optimization.
• Distributed data processing and lakehouse architectures such as Databricks, Delta Lake, or Apache Spark.
• Operating distributed data platforms at scale with orchestration tools like Airflow, Databricks Workflows, or AWS Step Functions.
• Git-based workflows and automated testing frameworks.
• Cloud-native development on AWS and/or GCP with CI/CD, code reviews, observability, and production support.
• Proven technical leadership, mentorship, and delivery within tight timelines.
• Trusted environment execution: clean rooms or secure data-sharing platforms, handling PII and regulated data, fine-grained access controls, governance policies, and policy enforcement.
• Familiarity with tokenization, pseudonymization, aggregation-before-export, and differential privacy concepts.
• Experience with measurement, attribution, audience analytics, or privacy-preserving reporting solutions.
• Data lineage and governance tooling such as Unity Catalog, AWS Glue Data Catalog, Apache Atlas, or OpenLineage.
• Understanding of trust boundaries, secure data-sharing patterns, and zero-trust data architecture.
• Experience documenting data contracts, flows, lineage, and permitted data movement between zones and domains.
• Experience in environments where only aggregated, anonymized, tokenized, or privacy-protected outputs may leave trusted boundaries.
• Strong written and verbal English communication and experience in Agile/SCRUM.
• Good to have: Databricks, AWS Clean Rooms or PETs, advertising and retail media platforms, collaboration with Security/Privacy/Risk/Compliance, ELK/Grafana/OpenTelemetry, Docker/Kubernetes, and secure design reviews.