Senior Data Engineer

About the role

The Data Engineering team is seeking a Senior Data Engineer to help design, build, and scale the modern data platform that powers analytics, data science, and data products across Versant brands.

Responsibilities

• Design and implement lakehouse architecture using Delta Lake, including medallion pipeline patterns (Bronze/Silver/Gold), schema enforcement, and time travel
• Build and operate batch and real-time ingestion pipelines leveraging Databricks Auto Loader, Structured Streaming, and Change Data Capture patterns
• Implement data governance and security using Unity Catalog, RBAC, and compliance-driven practices for sensitive environments
• Optimize performance and manage costs through FinOps strategies, including cluster sizing, workload tagging, Spark tuning, and Photon acceleration
• Design, implement, and maintain CI/CD pipelines and orchestration workflows using Databricks Workflows, Delta Live Tables, and tools such as Airflow
• Collaborate with Data Science teams on ML workflows, including MLflow, feature store integration, and model lifecycle management
• Ensure data quality, observability, and lineage across media-specific datasets such as streaming logs, ad impressions, and audience metrics
• Provide technical mentorship through code reviews, pairing, and knowledge sharing

Requirements

• Bachelor’s degree in Computer Science, Data Engineering, or equivalent practical experience
• 5+ years of experience building production-grade data pipelines in cloud environments using Spark-based platforms (e.g., Databricks, EMR, Dataproc, open-source Spark)
• Expertise in PySpark, SQL, and Spark-based data processing, with experience operating pipelines at scale in production
• Hands-on experience building batch or streaming production data pipelines using distributed processing frameworks (e.g., Spark, Flink) and query engines such as Presto
• Proficiency with orchestration tools such as Apache Airflow or Dagster, with hands-on experience in CI/CD, monitoring, alerting, and data quality for production systems
• Experience working with modern data architectures, including event-driven and distributed systems
• Solid understanding of infrastructure, networking, and data security fundamentals

Qualifications

• Experience building Lakehouse platforms and medallion pipelines in Databricks
• Familiarity with Unity Catalog, data governance, and compliance frameworks (e.g., PCI)
• Hands-on experience with CI/CD pipelines, orchestration tools, and infrastructure-as-code
• Experience with Lakeflow Spark Declarative Pipelines (SDP), MLflow, feature stores, and MLOps practices
• Background in media and entertainment data (e.g., video metadata, ad tech, audience analytics)
• Experience building data platforms within the media industry, with a strong understanding of audience analytics
• Experience working with large-scale analytical datasets (e.g., event logs, clickstream data, audience metrics)
• Comfortable using AI-assisted development tools (e.g., ChatGPT)
• Databricks or cloud certifications (e.g., Databricks Certified Data Engineer)