Director, Data Platform Engineering

About the role

Lila is seeking an experienced engineering leader to head the product data platform. This role involves owning the entire data platform and infrastructure, from architecture to delivery, reliability, and developer/data scientist experience. The mission is to enable AI/ML, product teams, and scientists to build data-intensive applications confidently and quickly.

Responsibilities

• Build, mentor, and manage a high-performing team of 8-12 data engineering experts.
• Evaluate and adopt modern data infrastructure - including real-time streaming (Kafka, Flink), columnar engines (DuckDB, ClickHouse), lake house, and cloud-native object storage architectures.
• Foster a culture of collaboration, innovation, and continuous improvement.
• Provide technical guidance and mentorship to team members, promoting their professional growth.
• Conduct performance reviews, provide feedback, and identify opportunities for training and development.
• Manage team workload, prioritize projects, and ensure timely delivery of high-quality solutions.
• Define and execute the technical roadmap for our data platform, aligning with Lila’s overall data strategy.
• Drive innovation in data Lakehouse and data serving ecosystem exploring new technologies and approaches to improve usability, performance, scalability, and efficiency.
• Ensure the reliability, availability, and security of our data processing infrastructure.
• Collaborate with other engineering teams to integrate our data processing technologies with other Lila systems and services.
• Partner with data scientists, data engineers, lab scientists, product managers, and other stakeholders to understand their data processing needs and requirements.
• Communicate technical concepts and solutions effectively to both technical and non-technical audiences.
• Advocate for best practices in data processing and engineering.
• Manage expectations and ensure alignment across different teams.
• Represent Lila’s data platform work at external conferences; deliver presentations, and write blog posts highlighting Lila’s leadership in big data processing.
• Drive innovative, agentic, and low-code solutions to deliver data interfaces - exploration, query, analytics, and ML/inference solutions at scale.

Requirements

• 12+ years of software development experience, with a focus on data processing at scale.
• 5+ years of experience leading senior engineers.
• Experience with building on AWS/GCP primitives like S3 + Athena/BigQuery, and query engines.
• Operated data platforms at petabyte scale with sub-second query latency requirements.
• Experience managing data infrastructure supporting 100+ concurrent ML training and inference workloads.
• Familiarity with LLM/AI-native data patterns — vector stores, embedding pipelines, pre/mid/post training.
• Track record of building data platforms in high-growth or early-stage environments where speed-to-value mattered as much as long-term architecture.
• Hands-on coding in Python and modern backend frameworks.
• Experience with infrastructure-as-code and containerized deployments (Kubernetes).

Qualifications

• BS, MS, or Ph.D. in Computer Science or a related field of study.
• Bonus points for thought leadership in the community via presentations in conferences or blog posts.
• Experience building and growing teams focusing on open source technologies.
• Scientific data management and quality experience.
• Built self-service data products/platforms where developer experience was a first-class product concern.

Benefits

The position offers competitive compensation including bonus potential and generous early equity. The final offer will reflect your unique background, expertise, and impact. Expected Base Salary Range: $232,000 USD - $346,000 USD.