Senior Staff Data Platform Engineer - Kafka - Apache Iceberg - Apache Spark

About the role

As an IC5 Senior Staff Engineer, you will architect and deliver large-scale distributed platform components, lead complex technical initiatives, and define engineering best practices. You will bring strong leadership, hands-on engineering depth, and the ability to design and operate reliable, scalable, and high-performance data systems.

Responsibilities

• Architect, design, and build high-performance distributed systems and platform components.
• Build distributed systems data ingestion solutions with strong emphasis on scalability, quality, and operational excellence.
• Design software that is easy to use, extend, and customize for customer-specific environments.
• Deliver high-quality, clean, modular, and reusable code while enforcing engineering best practices (code reviews, unit testing, automation, design reviews).
• Build foundational libraries, frameworks, and tools focused on modularity, extensibility, configurability, and maintainability.
• Collaborate across engineering teams to refine requirements and deliver end-to-end solutions.
• Provide technical leadership for projects with significant complexity and risk.
• Research, evaluate, and adopt new technologies that enhance platform capabilities.
• Troubleshoot and diagnose complex production issues across distributed systems.

Requirements

• Experience leveraging or critically thinking about how to integrate AI into engineering work—whether using AI-powered coding and operational tooling, automating workflows, or reasoning about how AI changes the way software and infrastructure are built.
• 10+ years of software development experience with a Bachelor’s degree; OR 8+ years with a Master’s degree; OR 6+ years with a PhD; OR equivalent work experience.

Core Distributed Systems Expertise

• Strong fundamentals in distributed systems architecture, design patterns, and algorithms.
• Deep programming expertise in Java, including JVM internals, memory models, and garbage collection.
• Proven experience in JVM performance tuning, profiling, and diagnosing performance bottlenecks.
• Strong understanding of concurrency, networking, sockets, OS internals, and performance optimization.
• Hands-on experience building and operating large-scale distributed systems.
• Experience with relational databases such as Oracle, MySQL, or PostgreSQL.

Streaming & Messaging Systems

• Experience with large-scale deployments of Kafka, or similar streaming platforms.
• Deep knowledge of stream processing, topic design, partitioning, replication, and HA strategies.
• Experience working within DevOps environments for operationalizing distributed platforms.

Data Lake & Ingestion Architectures

• Demonstrated experience architecting and delivering full-stack Data Lake solutions.
• Strong expertise in designing and operating data ingestion pipelines using:
  • Apache Iceberg (tables, catalogs, schema evolution, metadata management)
  • Kafka Connect (source/sink connectors, distributed mode)
  • Apache Kafka (high-scale clusters, topic/partition strategies, HA)
  • Apache Flink (stateful stream processing, exactly-once semantics)
  • Apache Spark (batch & streaming jobs, optimization, partitioning)
• Expertise in data formats such as Parquet, ORC, and Avro, along with compaction and governance strategies.
• Ability to build scalable, fault-tolerant ingestion and transformation workflows.
• Experience integrating Data Lakes with analytics engines, query services, or ML platforms.

Pay

For positions in this location, we offer a base pay of $181,200 – $317,100, plus equity (when applicable), variable/incentive compensation and benefits.

Benefits

• Health plans, including flexible spending accounts
• 401(k) Plan with company match
• Employee Stock Purchase Plan (ESPP)
• Matching donations
• Flexible time away plan
• Family leave programs