Sr. Software Engineer Cloud infrastructure

About the role

The Applied Intelligence Data Engineering team is seeking a Senior Software Engineer – Cloud Infrastructure. This is a hybrid role that blends deep software engineering with hands-on ownership of cloud infrastructure.

Responsibilities

• Optimize Data Streaming Applications on Cloud

  • Own end-to-end performance of data streaming applications running on cloud infrastructure — from Kafka topic configuration through consumer processing and downstream delivery.
  • Profile and tune streaming pipelines to maximize throughput and minimize latency, leveraging cloud-native compute, storage, and networking resources.
  • Identify and address bottlenecks at the intersection of application code and cloud resource constraints, including CPU throttling, network saturation, I/O limits, and memory constraints.
  • Design and implement cloud resource utilization strategies. This includes spot/preemptible instances, managed streaming services, and dynamic node pool scaling to balance performance with cost efficiency.
  • Benchmark streaming pipelines end-to-end and translate findings into actionable infrastructure and code improvements.
  • Collaborate with Data and AI/ML engineering teams to ensure streaming pipelines are optimally provisioned for real-time feature engineering, inference, and analytics workloads.

• Design & Build Cloud-Native Applications

  • Develop high-throughput, low-latency streaming applications using Java and Kafka.
  • Design event-driven microservices that process, enrich, and route real-time data at scale.
  • Implement reactive, non-blocking architectures to support high concurrency and resilience.
  • Develop reusable streaming frameworks, libraries, and platform capabilities to improve engineering velocity and standardization.

• Architect & Operate Multi-Cloud Infrastructure

  • Architect, implement, and optimize multi-cloud infrastructure across GCP and OCI to support large-scale data and streaming workloads.
  • Design and implement advanced networking architectures, including VPC peering, VPNs, load balancers, and cross-region failover strategies.
  • Build and maintain Terraform-based infrastructure-as-code frameworks to standardize deployments and enable developer self-service.
  • Define autoscaling, deployment, failover, and resource optimization strategies for high-volume production systems.
  • Contribute to platform-wide architecture decisions related to scalability, resiliency, high availability, and disaster recovery.

• Kubernetes & Container Engineering

  • Deploy and manage containerized microservices within Kubernetes environments (GKE, OKE) across cloud platforms.
  • Implement container orchestration best practices, service-mesh configurations, and rolling-deployment strategies.
  • Partner with platform engineering teams to improve developer tooling, deployment automation, and runtime reliability.

• Production Reliability & Performance

  • Ensure production-grade reliability, observability, and operational maturity across streaming platforms and infrastructure.
  • Implement comprehensive observability using Prometheus, Grafana, centralized logging, distributed tracing, and health monitoring.
  • Optimize systems for throughput, latency, resiliency, resource efficiency, and cloud cost governance.
  • Build automated testing strategies for streaming and infrastructure workflows, including unit, integration, contract, chaos, and performance testing.
  • Lead incident response, root-cause analysis, and postmortems to improve uptime and reduce operational risk.

• Cross-Functional Collaboration

  • Partner with Data Engineering teams to integrate streaming architectures with batch processing, data lakes, and analytical platforms.
  • Collaborate with Software Engineering, Product Management, and API teams to enable real-time services and data-driven applications.
  • Work closely with AI/ML engineering teams to support real-time feature engineering, inference pipelines, and operational AI workload.
  • Clearly communicate technical tradeoffs to engineering stakeholders. Also, discuss scalability considerations and operational risks.

• Technical Management

  • Lead architectural discussions, design reviews, and technical deep dives across distributed systems and cloud infrastructure.
  • Drive engineering standards across code quality, documentation, observability, security, and platform maintainability.
  • Influence long-term technical strategy and modernization initiatives for real-time data infrastructure and cloud platforms.

Required Technical Skills

• Deep expertise in optimizing data streaming applications for throughput, latency, and cost-efficiency across cloud environments.
• Proficient in tuning Kafka producers, consumers, and brokers — including batch sizing, compression, partition strategies, and consumer lag management — within cloud-hosted deployments.
• Experience leveraging cloud-native managed services (e.g., Kafka, GCP Pub/Sub, BigQuery Streaming; OCI Streaming) to complement or extend Kafka-based pipelines.
• Familiarity with cloud-native autoscaling patterns for streaming consumers, including KEDA, HPA, and custom metrics-based scaling in Kubernetes.
• Ability to identify and address performance bottlenecks at the intersection of application code and cloud resource limits.
• Experience benchmarking and profiling streaming pipelines from start to finish. You also need to turn your findings into improvements for infrastructure and code.

Basic Qualifications

• This is a hybrid role. Candidates must have hands-on experience in both software engineering and cloud infrastructure.
• 7+ years of experience in software engineering and cloud infrastructure, with at least 3+ years in each area.
• Demonstrated expertise in optimizing data streaming applications on cloud infrastructure.
• Proven track record building and operating production-grade real-time data platforms.
• Experience mentoring engineers and collaborating across teams.
• Bachelor's degree in Computer Science, Engineering, or a related field; advanced degree preferred.