Lead Systems Engineer

About the role

The Kafka Administrator plays a critical role in ensuring the stability, security, and efficiency of our Confluent Kafka infrastructure. This position is part of the Architecture & Enterprise Services department, responsible for maintaining the firm’s IT infrastructure and supporting seamless business operations.

Responsibilities

• Kafka Cluster Management
• Install, configure, and manage Kafka clusters, including Confluent Kafka components.
• Maintain and optimize existing clusters while setting up new environments as needed.
• Performance Monitoring & Optimization
• Continuously monitor Kafka clusters for performance, reliability, and capacity planning.
• Identify and resolve performance bottlenecks, ensuring optimal system efficiency.
• Security & Compliance
• Implement and manage security protocols, including encryption, authentication, and authorization.
• Ensure compliance with industry standards and internal security policies.
• Integration & Data Flow Management
• Work closely with development teams to integrate Kafka with various data sources and sinks.
• Configure and manage Kafka connectors to facilitate smooth data flow.
• Upgrades, Patching & Disaster Recovery
• Plan and execute Kafka upgrades and patches, keeping systems updated with the latest features and security fixes.
• Develop and maintain disaster recovery plans, conducting regular testing to ensure business continuity.
• Administration & Troubleshooting
• Manage Kafka topics, including configurations, retention policies, and cleanup settings.
• Provide technical support for Kafka-related issues, collaborating with cross-functional teams to ensure quick resolutions.
• Documentation & Knowledge Sharing
• Maintain comprehensive documentation for Kafka configurations, processes, and best practices.
• Contribute to knowledge sharing within the team to enhance operational efficiency.

Qualifications

• Minimum of 6 years of related experience
• Bachelor's degree preferred or equivalent experience
• Minimum 5 years of experience in Kafka administration, including Confluent Kafka
• Strong understanding of Kafka security, including encryption, authentication, and authorization
• Experience with Kafka integration, including configuring and managing Kafka connectors
• Hands-on experience with Kafka upgrades, patching, and disaster recovery planning
• Proficiency in troubleshooting Kafka-related issues and optimizing system performance
• Strong documentation skills and ability to communicate effectively across teams