Senior Data Engineer

What You'll Do

• Design and build scalable data pipelines for batch and real-time processing across Kai's agentic AI platform
• Own and optimize high-volume data infrastructure handling hundreds of millions of entries with low latency and high reliability
• Build and maintain data models and storage systems optimized for large-scale, high-throughput security data workloads
• Identify bottlenecks in the current architecture and drive optimization — reduce processing time, improve reliability, and make the customer experience better
• Lead the Terraformization of data pipelines to enable cloud-agnostic deployment across Azure, AWS, and GCP
• Integrate and manage cloud data services, ensuring secure service principles, permissions, and cross-service connectivity
• Collaborate closely with Backend Engineering teams on both the ingestion and consumption sides of the data pipeline
• Ensure data quality, consistency, and reliability across all pipelines
• Contribute to code reviews, technical documentation, and best practices
• Bring a point of view — propose solutions, not just problems, and start building before you're asked

Required

• 7+ years of experience in data engineering or data platform engineering
• Must have hands-on experience handling up to 200M+ entries in materialized views in an asynchronous manner
• Strong proficiency in Python and SQL — these are how our systems are written
• Strong data modeling skills — you can design schemas and storage systems that hold up at scale
• Proven experience designing and building large-scale distributed data pipelines in both batch and streaming modes
• Hands-on experience with Flink, Kafka, Spark, or similar stream and batch processing frameworks
• Experience with data pipeline orchestration tools — Airflow, Temporal, or equivalent
• Infrastructure experience — Terraform, Kubernetes, and Docker are expected, not aspirational
• Cloud platform expertise — deep hands-on experience in at least one major cloud platform (Azure, AWS, or GCP); Azure experience strongly preferred
• Strong communication skills — you work cross-functionally and can explain complex systems clearly

PREFERRED

• DataOps experience — ability to own data infrastructure decisions independently, reducing dependency on DevOps for pipeline deployment, permissions, and service integration
• Experience with data systems supporting AI/ML workloads — feature stores, ML pipelines, or dataset versioning
• Experience with DeltaLake, Apache Iceberg, or similar open table formats
• Startup or high-growth experience — you have operated in a fast-paced environment where things change quickly, and ownership is expected