SR. Kafka Engineer

About the role

Design, develop, and maintain robust Kafka-based applications and data pipelines that support business operations, including the real-time or near-real-time data to AI/ML models.

Collaborate with development, operations, and infrastructure teams to deliver reliable, scalable, and high-performing Kafka solutions.

Ensure the availability, reliability, and performance of Kafka clusters and related systems.

Work closely with architects, data engineers, and stakeholders to define requirements and deliver solutions.

Troubleshoot and resolve issues in Kafka applications, ensuring minimal downtime and optimal performance.

Document code, design decisions, processes, configurations, and best practices for future reference and team knowledge sharing.

Mentor junior developers and share Kafka expertise, fostering a culture of learning and growth.

Stay current with the latest Kafka releases, features, and ecosystem advancements.

Perform statistical analysis to monitor team performance, improve processes, and ensure customer satisfaction.

Define and set SLAs for projects, ensuring high standards of service delivery.

Responsibilities

• Leading the design, development, and implementation of enterprise-scale, fault-tolerant data pipelines using Apache Kafka;
• Providing expert-level management and administration of Kafka clusters throughout the Systems Development Life Cycle (SDLC), including upgrades and patching;
• Overseeing large-scale, cross-functional projects as a senior Product Owner or Agile/Scrum leader, ensuring alignment with organizational goals;
• Demonstrating advanced proficiency in Java programming, with experience in Python as a plus;
• Architecting event-driven and microservices-based solutions leveraging Kafka APIs (Producer, Consumer, Streams, Connect);
• Establishing and enforcing best practices for serialization formats (Avro, Protobuf, JSON) and schema registry/data governance, including Hackolade for data modeling;
• Directing the optimization of producer/consumer performance and large-scale data ingestion strategies;
• Leading the implementation of unit and integration testing frameworks for Kafka applications;
• Managing hybrid integration architecture patterns and ensuring reliability, scalability, and performance of Kafka clusters;
• Overseeing monitoring and troubleshooting activities using tools such as Prometheus and Grafana;
• Providing technical guidance and mentorship to teams on Kafka cluster setup, configuration, and tuning;
• Ensuring compliance with organizational standards and data governance policies.

Qualifications

• At least 52 weeks of specialized experience at the GS-14 level, or equivalent, leading the design, development, and implementation of enterprise-scale, fault-tolerant data pipelines using Apache Kafka;
• Expert-level management and administration of Kafka clusters throughout the Systems Development Life Cycle (SDLC), including upgrades and patching;
• Overseeing large-scale, cross-functional projects as a senior Product Owner or Agile/Scrum leader, ensuring alignment with organizational goals;
• Advanced proficiency in Java programming, with experience in Python as a plus;
• Experience in architecting event-driven and microservices-based solutions leveraging Kafka APIs (Producer, Consumer, Streams, Connect);
• Experience in establishing and enforcing best practices for serialization formats (Avro, Protobuf, JSON) and schema registry/data governance, including Hackolade for data modeling;
• Experience in directing the optimization of producer/consumer performance and large-scale data ingestion strategies;
• Experience in leading the implementation of unit and integration testing frameworks for Kafka applications;
• Experience in managing hybrid integration architecture patterns and ensuring reliability, scalability, and performance of Kafka clusters;
• Experience in overseeing monitoring and troubleshooting activities using tools such as Prometheus and Grafana;
• Experience in providing technical guidance and mentorship to teams on Kafka cluster setup, configuration, and tuning;
• Experience in ensuring compliance with organizational standards and data governance policies.