Sr. Software Engineer Data Streaming Systems

Overview

It will build fault-tolerant streaming applications supporting real-time analytics, APIs, AI workflows, and critical data services. This role will design and build high-performance, fault-tolerant streaming applications that power real-time analytics, APIs, AI workflows, and mission-critical data services across the organization. You will architect and implement distributed, event-driven systems using Java, Kafka, Kubernetes, and modern reactive frameworks. As a senior engineer, you will set the technical direction. You will mentor other engineers. You will also ensure that our streaming platforms are reliable, scalable, and operate smoothly. This role requires deep expertise in distributed systems, concurrency, and cloud-native microservices.

Primary Responsibilities

• Design & Build Real-Time Streaming Applications: Develop high-throughput, low-latency streaming applications using Java and Kafka. Design event-driven microservices that process, enrich, and route real-time data at scale. Implement reactive, non-blocking architectures to support high concurrency and resilience. Create reusable streaming frameworks and libraries to improve engineering efficiency and standardization across the platform.
• Architect Scalable Distributed Systems: Design and optimize distributed streaming architectures with Kafka topics, partitioning strategies, consumer groups, schema management, and event lifecycle patterns. Make architecture decisions for the whole platform including scalability, resiliency, high availability, disaster recovery, and deployments across multiple regions. Drive best practices around event modeling, schema evolution, idempotency, replayability, and data consistency. Build and optimize horizontally scalable services deployed within Kubernetes-based cloud environments.
• Production Reliability & Performance: Ensure streaming platforms and services are reliable, easy to monitor, and operationally mature. Optimize systems for throughput, latency, resiliency, resource efficiency, and infrastructure cost management. Set up full observability using metrics, centralized logging, distributed tracing, alerting, and health monitoring. Build automated testing strategies including unit, integration, contract, chaos, and performance testing. Participate in production support, incident response, and root-cause analysis. Work on initiatives to improve reliability continuously.
• Cloud-Native & Kubernetes Engineering: Deploy and manage containerized microservices within Kubernetes environments across GCP, AWS, or similar cloud platforms. Define strategies for autoscaling, deployment, failover, and resource optimization for high-volume production systems. Create and manage CI/CD pipelines including Infrastructure-as-Code and automated deployment workflows. Collaborate with platform engineering teams to improve developer tooling, deployment automation, and runtime reliability.
• Cross-Functional Collaboration: Partner with Data Engineering teams to integrate streaming architectures with batch processing systems, data lakes, and analytical platforms. Collaborate with Software Engineering, Product Management, and API teams to enable real-time services and data-driven applications. Work closely with AI/ML engineering teams to support real-time feature engineering, inference pipelines, and operational AI workloads. Clearly explain tradeoffs, scalability considerations, and operational risks to engineering stakeholders.

Required Technical Skills

• Java & Reactive Programming: Advanced proficiency in Java, including concurrency, multithreading, and JVM performance tuning. Strong experience with reactive frameworks such as Spring WebFlux, Project Reactor, or similar. Deep knowledge of asynchronous, non-blocking system design.
• Kafka & Event-Driven Architecture: Extensive experience with Apache Kafka (producers, consumers, streams, schema registry). Strong knowledge of partitioning strategies, offset management, rebalancing, and failure recovery. Experience designing event schemas and managing schema evolution. Familiarity with Kafka Streams, Flink, or similar stream-processing frameworks.
• Kubernetes & Cloud-Native Systems: Strong hands-on experience deploying and operating applications in Kubernetes. Experience using Docker for containerization. Knowledge of microservices architecture, autoscaling, rolling deployments, and patterns for reliable production systems.

Basic Qualifications

• 5+ years of professional software engineering experience, including 2+ years working on streaming or distributed systems
• Experience designing, building, and running production-grade real-time event-driven systems
• Comfortable working in cloud-based microservices engineering environments
• Ability to take ownership of complex technical work and deliver reliable solutions autonomously
• Willingness to mentor junior engineers and contribute to team engineering standards
• Strong communication and collaboration skills across engineering and product teams
• Bachelor's degree in Computer Science, Engineering, or related technical field; advanced degree preferred

Pay

Hiring Salary Range: $124,000.00 - 186,000.00. The hiring salary range for this position applies to New York, California, Colorado, Washington state, and most other geographies. Starting pay depends on a variety of job-related factors, including geographic location, market demands, experience, training, and education. This position is bonus eligible.

Benefits

• Medical, dental, and vision coverage
• 401(k) plan
• Life insurance coverage
• Disability benefits
• Tuition assistance program
• Paid time off (PTO) or as dictated by Collective Bargaining Agreement
• Generous paid time off
• Opportunities for on-site and virtual engagement events
• Opportunities to make meaningful connections and build community