Senior Confluent Kafka Lead / Python Developer (contract)

Role Overview

The Confluent Kafka Lead / Python Developer is responsible for designing, building, and operating enterprise-grade event streaming solutions using Confluent Kafka while developing Python-based producers, consumers, and streaming applications. This role blends deep hands-on development with technical leadership, ensuring scalable, reliable, and secure real-time data flows across distributed systems. The position plays a key role in event-driven architecture (EDA), data platform modernization, and real-time analytics initiatives.

Key Responsibilities

• Kafka & Streaming Platform Leadership
• Lead the design and implementation of enterprise Kafka and Confluent Platform solutions (Kafka, Schema Registry, Connect, ksqlDB).
• Define and enforce topic design, partitioning, retention, and schema evolution standards.
• Act as technical owner for Kafka clusters across dev, test, and production environments.
• Drive best practices for high availability, fault tolerance, and scalability.
• Python Development (Producers & Consumers)
• Design and develop Python-based Kafka producers and consumers using Confluent Kafka Python APIs.
• Build event-driven microservices and streaming applications in Python.
• Implement message serialization and schema validation (Avro, JSON, Protobuf).
• Handle idempotency, retries, back-pressure, and error handling patterns.
• Event-Driven Architecture & Integration
• Design event-driven integration patterns bridging microservices, data stores, APIs, and third-party systems.
• Integrate Kafka with downstream consumers such as databases, data lakes, analytics platforms, and search systems.
• Support real-time pipelines for transactions, telemetry, customer events, and analytics.
• Collaborate with API, data, and application teams to align event contracts.
• Security & Governance
• Implement Kafka security controls: TLS encryption, SASL / OAuth authentication, ACL-based authorization.
• Enforce data governance, schema compatibility rules, and event ownership models.
• Ensure compliance with enterprise security and regulatory standards.
• DevOps, Automation & Observability
• Build and maintain CI/CD pipelines for Kafka-related applications and configurations.
• Use Infrastructure as Code to provision and manage Kafka infrastructure.
• Implement monitoring and alerting using tools such as Confluent Control Center, Prometheus, Grafana, or cloud-native equivalents.
• Troubleshoot production streaming issues related to latency, lag, throughput, or data loss.
• Technical Leadership
• Serve as Kafka subject matter expert and technical lead.
• Mentor developers on event-driven design and streaming best practices.
• Review designs and code for Kafka and Python-based streaming solutions.
• Partner with architects, SREs, and platform teams on roadmap and capacity planning.

Required Qualifications

• Kafka & Streaming Experience
• 6–10+ years of experience in software or data engineering.
• 4+ years of hands-on experience with Apache Kafka and/or Confluent Platform.
• Strong knowledge of: Kafka internals (brokers, partitions, offsets, consumer groups), Schema Registry and schema evolution, Kafka Connect architectures and connectors.
• Python Development
• Strong proficiency in Python for backend and streaming development.
• Experience building production-grade services using Python frameworks and libraries.
• Familiarity with async processing, multithreading, or stream processing patterns is a plus.
• Cloud & DevOps Experience
• Deploying Kafka and applications in cloud or hybrid environments (AWS, Azure, GCP).
• CI/CD pipeline experience (GitHub Actions, Jenkins, GitLab, Azure DevOps).
• Infrastructure-as-Code experience (Terraform, CloudFormation, ARM/Bicep).
• Containerization experience (Docker, Kubernetes) preferred.

Preferred Qualifications

• Experience with ksqlDB, Kafka Streams, or stream processing frameworks (Flink, Spark Streaming).
• Exposure to event sourcing or CQRS patterns.
• Integration of Kafka with data lakes, warehouses, and analytics platforms.
• Confluent or cloud platform certifications.
• Experience supporting high-throughput, low-latency systems.

Soft Skills & Leadership

• Strong communication skills across engineering and stakeholder teams.
• Ability to translate business use cases into event-driven technical solutions.
• Comfortable acting as both hands-on developer and technical lead.
• Experience influencing architecture and standards across teams.

Pay & Benefits

The pay range that the employer in good faith reasonably expects to pay for this position is $59.79/hour - $93.43/hour. Our benefits include medical, dental, vision and retirement benefits.