Senior Kafka Platform Engineer

About the role

The Event Streaming team in Enterprise Middleware organization is seeking a Senior Kafka Platform Engineer to lead the evolution of its enterprise event-streaming platform and drive enterprise-scale event streaming, Confluent Cloud adoption, platform efficiency, and AI-enabled SDLC transformation.

Responsibilities

• Design and build AI agents, custom skills, or equivalent frameworks to automate engineering and operational workflows at scale

• Leverage AI-powered tools (e.g., Copilot, AI agents) to accelerate development, automate testing, improve code quality, and enable intelligent SDLC workflows aligned to AI-DLC vision.

• Design agentic workflows and multi-agent orchestration patterns to solve complex distributed systems, integration, and operational challenges

• Act as the senior technical authority for Confluent Platform and Confluent Cloud, providing architecture, design, and implementation guidance to solve complex enterprise integration challenges at scale.

• Lead solution architecture across cross-functional initiatives, ensuring alignment with cloud modernization, resiliency, security, and AI-driven engineering priorities.

• Perform deep-dive analysis of complex production issues, drive root cause identification, and implement sustainable remediation strategies with full documentation.

• Design and build automation frameworks and self-service tools to streamline Kafka operations, reduce manual effort, and improve developer productivity and platform scalability.

• Lead seamless migration of on-prem Kafka workloads to Confluent Cloud using cluster linking, schema portability, and hybrid streaming patterns—ensuring zero business disruption.

• Drive execution across Kafka, infrastructure, security, and application teams to remove dependencies, mitigate risks, and ensure on-time delivery of committed milestones.

• Contribute to enterprise architecture forums, defining and enforcing best practices for event streaming, integration patterns, and platform standardization.

• Establish best practices for testing, reliability, observability, and operational readiness, ensuring high availability and scalable platform performance.

• Identify systemic and architectural risks early, leading mitigation efforts to prevent client impact and ensure platform stability.

• Implement enterprise-grade security controls (mTLS, RBAC, encryption) and drive continuous remediation of vulnerabilities, ensuring compliance with regulatory and audit requirements.

• Interpret technical requirements to build and enhance Kafka configurations, message flows, and integration solutions supporting business-critical applications.

• Partner with distributed teams (including offshore engineering) to ensure consistent standards, efficient delivery, and 24x7 operational support across time zones.

Qualifications

• Bachelor’s degree in computer science, Engineering, or related field

• 10+ years of hands-on software engineering experience with proven technical leadership

• 5+ years of expertise in large-scale distributed systems and mission-critical infrastructure

• Strong hands-on experience with GitHub Copilot and other AI-assisted development tools, including advanced prompt engineering and workflow integration to accelerate engineering

• Proven ability to leverage AI-assisted development (e.g., Copilot, code generation) and build AI-driven agents/workflows to improve team productivity and engineering efficiency

• Deep expertise in Confluent Kafka (capacity planning, administration, internals, performance tuning)

• Strong experience with Kafka clusters, replication, disaster recovery, and data pipelines

• Hands-on experience with Kubernetes, Confluent Operator, and cloud-native modernization (GCP)

• Strong development background on Linux/Unix platforms

• Experience with data modeling and streaming technologies (Avro, Protobuf, JSON Schema)

• Strong automation mindset with hands-on expertise in Python, Ansible, and CI/CD pipelines (Jenkins, DevOps tooling) to drive operational efficiency and reduce manual effort

• Solid foundation in security engineering (encryption, TLS, certificate and key management)

• Solid foundation in observability, reliability, automation, and production readiness with proactive risk mitigation

• Experience leading architecture and execution across cross-functional, large-scale programs

• Strong understanding of networking, load balancing, OS fundamentals, and performance optimization

• Experience in regulated environments (financial services preferred)

• Strong collaboration skills across engineering, architecture, infrastructure, and business teams

• Experience working with globally distributed/offshore teams

• Strong Agile delivery experience (Scrum/Kanban/Scaled Agile)

• Strong leadership and soft skills including strategic thinking, decision-making, mentoring/coaching, and driving engineering excellence

• Excellent communication, influencing, and stakeholder management skills, with ability to align diverse teams toward common goals

• High ownership mindset with ability to prioritize, multitask, and lead through high-pressure production situations

• Strong problem-solving skills with a proactive, innovation-driven approach

• Self-driven, adaptable, and customer-focused with strong relationship management and team-building capabilities