Kafka Engineer

About the role

We are looking for a Kafka Engineer responsible for designing, developing, implementing, and supporting highly scalable event-streaming and real-time data pipelines using Apache Kafka and related technologies.

Responsibilities

• Design and implement scalable Kafka-based event-streaming solutions.
• Develop and maintain Kafka producers and consumers.
• Create and manage Kafka topics, partitions, consumer groups, and replication.
• Configure and monitor Kafka brokers and clusters.
• Implement high availability, fault tolerance, and disaster-recovery strategies.
• Develop real-time data pipelines and integrations between applications and data platforms.
• Work with Kafka Connect for source and sink integrations.
• Implement and maintain Kafka Streams applications where required.
• Monitor Kafka performance, throughput, latency, lag, and cluster health.
• Troubleshoot producer, consumer, broker, partition, and replication issues.
• Implement security using SSL/TLS, SASL, ACLs, and authentication/authorization.
• Optimize Kafka configurations for performance and scalability.
• Work with development and data engineering teams to define event schemas and integration patterns.
• Implement monitoring and alerting using tools such as Prometheus, Grafana, or equivalent platforms.
• Participate in production support, incident management, and root-cause analysis.
• Automate deployments and operational tasks using CI/CD and scripting.

Requirements

• Strong knowledge of Apache Kafka.
• Kafka Producers and Consumers.
• Kafka Topics, Partitions, Offsets, Consumer Groups, and Replication.
• Kafka Connect.
• Kafka Streams.
• Schema management and serialization formats such as Avro, JSON, and Protobuf.
• Strong programming experience in Java, Python, or Scala.
• REST APIs and microservices.
• Linux/Unix.
• SQL and database fundamentals.
• Git and CI/CD.
• Docker and Kubernetes.
• Cloud platforms such as AWS, Azure, or GCP.