Senior Systems Engineer, Workers AI

About the role

You'll design and build the core infrastructure that powers AI inference across Cloudflare's global network — real-time voice, frontier open LLMs, and customer-deployed models running on a heterogeneous fleet of GPUs and next-generation accelerators in hundreds of cities worldwide.

Working alongside AI/ML engineers, hardware partners, and Cloudflare product teams, you'll solve hard problems in distributed systems and high-performance computing: sub-second model cold starts, multi-accelerator workload scheduling, efficient KV cache management, and a model deployment platform serving both Cloudflare and customers bringing their own models.

Responsibilities

• Platform Architecture: Develop and maintain core components of the serverless inference platform to ensure high availability and scalability for Cloudflare users.
• Optimization & Performance: Optimize the model scheduling system to significantly increase efficiency and resource utilization across our inference infrastructure, and implement improvements to request routing logic to reduce latency for end-users.
• System Reliability: Drive significant, measurable improvements in the platform's reliability and resilience by identifying and mitigating systemic risks.
• Observability: Expand and refine the observability stack (metrics, logging, tracing) and fine-tune alerts to proactively identify and resolve production issues.
• Tech Leadership: Lead complex, cross-functional technical projects from initial concept and design through final deployment and operationalization.
• Mentorship: Act as a mentor to junior engineers and actively contribute to cultivating a strong, collaborative engineering culture within the team.

Desirable Skills, Knowledge & Experience

• Proven experience in systems engineering, with a primary focus on distributed, high-performance systems.
• Expert proficiency in Rust programming, particularly in an asynchronous environment.
• Solid experience with scaling and performance optimization techniques, including load balancing and caching in a distributed environment.
• Bonus Points: Demonstrable experience with container orchestration platforms, specifically Kubernetes and/or Nomad.
• Familiarity with the unique architectural challenges involved in large-scale inference serving (e.g., LLMs and diffusion models).