Senior Staff Data Platform Engineer - Kafka - Apache Iceberg - Apache Spark

About the Role

As an IC5 Senior Staff Engineer, you will architect and deliver large-scale distributed platform components, lead complex technical initiatives, and define engineering best practices. You will bring strong leadership, hands-on engineering depth, and the ability to design and operate reliable, scalable, and high-performance data systems.

What You Get To Do In This Role

• Architect, design, and build high-performance distributed systems and platform components.
• Build distributed systems data ingestion solutions with strong emphasis on scalability, quality, and operational excellence.
• Design software that is easy to use, extend, and customize for customer-specific environments.
• Deliver high-quality, clean, modular, and reusable code while enforcing engineering best practices (code reviews, unit testing, automation, design reviews).
• Build foundational libraries, frameworks, and tools focused on modularity, extensibility, configurability, and maintainability.
• Collaborate across engineering teams to refine requirements and deliver end-to-end solutions.
• Provide technical leadership for projects with significant complexity and risk.
• Research, evaluate, and adopt new technologies that enhance platform capabilities.
• Troubleshoot and diagnose complex production issues across distributed systems.

Qualifications

To be successful in this role you have:

• Experience leveraging or critically thinking about how to integrate AI into engineering work — whether using AI-powered coding and operational tooling, automating workflows, or reasoning about how AI changes the way software and infrastructure are built.
• 10+ years of software development experience with a Bachelor's degree; OR 8+ years with a Master's degree; OR 6+ years with a PhD OR equivalent work experience.

Core Distributed Systems Expertise

• Strong fundamentals in distributed systems architecture, design patterns, and algorithms.
• Deep programming expertise in Java, including JVM internals, memory models, and garbage collection.
• Proven experience in JVM performance tuning, profiling, and diagnosing performance bottlenecks.
• Strong understanding of concurrency, networking, sockets, OS internals, and performance optimization.
• Hands-on experience building and operating large-scale distributed systems.
• Experience with relational databases such as Oracle, MySQL, or PostgreSQL.

Streaming & Messaging Systems

• Experience with large-scale deployments of Kafka, or similar streaming platforms.
• Deep knowledge of stream processing, topic design, partitioning, replication, and HA strategies.
• Experience working within DevOps environments for operationalizing distributed platforms.
• Demonstrated experience architecting and delivering full-stack Data Lake solutions.
• Strong expertise in designing and operating data ingestion pipelines using:
  • Apache Iceberg (tables, catalogs, schema evolution, metadata management)
  • Kafka Connect (source/sink connectors, distributed mode)
  • Apache Kafka (high-scale clusters, topic/partition strategies, HA)
  • Apache Flink (stateful stream processing, exactly-once semantics)
  • Apache Spark (batch & streaming jobs, optimization, partitioning)
• Expertise in data formats such as Parquet, ORC, and Avro, along with compaction and governance strategies.
• Ability to build scalable, fault-tolerant ingestion and transformation workflows.
• Experience integrating Data Lakes with analytics engines, query services, or ML platforms.

Pay

For positions in this location, we offer a base pay of $181,200 - $317,100, plus equity (when applicable), variable/incentive compensation and benefits. Sales positions generally offer a competitive On Target Earnings (OTE) incentive compensation structure. Please note that the base pay shown is a guideline, and individual total compensation will vary based on factors such as qualifications, skill level, competencies, and work location.

Benefits

• Health plans, including flexible spending accounts
• 401(k) Plan with company match
• ESPP
• Matching donations
• Flexible time away plan
• Family leave programs

Schedule

This is a Flexible (Hybrid) position requiring 2 days per week in a ServiceNow office location. Offices are located in San Francisco, CA; Pleasanton, CA; Santa Clara, CA; and San Diego, CA.

Additional Information

This position will include supporting our US Regulated Markets. This position requires passing a ServiceNow background screening, USFedPASS (US Federal Personnel Authorization Screening Standards) including a credit check, criminal/misdemeanor check, and drug test. Any employment is contingent upon passing the screening. Due to Federal requirements, only US citizens, US naturalized citizens, or US Permanent Residents holding a green card will be considered.

This response is AI-generated, for reference only.