Senior Kafka Engineer

Responsibilities

• Identify and resolve Kafka messaging issues within a justified timeframe.
• Collaborate with business and IT teams to understand business problems and design, implement, and deliver appropriate solutions using Agile methodology within a larger program.
• Work independently to implement solutions across multiple environments (DEV, QA, UAT, PROD).
• Provide technical direction, guidance, and code reviews for other engineers working on the same project.
• Administer distributed Kafka clusters in DEV, QA, UAT, and PROD environments and troubleshoot performance issues.
• Implement and debug subsystems, microservices, and components.
• Follow an automate-first/automate-everything philosophy.

Key Skills & Expertise

• Deep understanding of Confluent Kafka – Proficient in Kafka concepts, including producers, consumers, topics, partitions, brokers, and replication mechanisms.
• Programming proficiency – Expertise in Java or Scala, with potential Python usage depending on the project.
• System design and architecture – Ability to design robust, scalable Kafka-based data pipelines considering data throughput, fault tolerance, and latency.
• Data management skills – Knowledge of data serialization formats such as JSON, Avro, and Protobuf, and schema evolution management.
• Kafka Streams API (optional) – Familiarity with Kafka Streams for real-time data processing within the Kafka ecosystem.
• Monitoring & troubleshooting – Experience with Kafka cluster health monitoring, identifying performance bottlenecks, and troubleshooting issues.
• Cloud integration – Experience deploying and managing Kafka on AWS, Azure, or GCP.
• Understanding of distributed systems concepts.

Must-Have Qualifications

• 8-12 years of experience in software engineering.
• Deep knowledge of Kafka producers, consumers, topics, partitions, brokers, and replication.
• Strong in Java or Scala, with potential Python usage.
• Experience in designing high-throughput, scalable Kafka pipelines.
• Experience deploying Kafka on AWS, Azure, or GCP.
• Familiarity with Kafka cluster health monitoring and performance tuning.