Staff Software Engineer, Inference

About the role

The Inference team builds and operates CoreWeave’s Kubernetes-native inference platform, powering low-latency, high-throughput AI workloads at massive scale. The role involves leading cross-team design initiatives, optimizing inference performance, and improving system reliability at scale.

Responsibilities

• Lead cross-team design initiatives impacting multiple services or organizations
• Optimize inference performance (latency, throughput, and GPU utilization)
• Improve system reliability at scale
• Work deeply in distributed systems and Kubernetes-based infrastructure, focusing on areas like scheduling, batching, and memory optimization
• Drive system-wide improvements in architecture, performance, and reliability

Requirements

• 8–12+ years of experience building and operating large-scale distributed systems or cloud platforms
• Proven experience leading cross-team technical initiatives impacting multiple services or organizations
• Strong programming skills in Go, Python, or C++
• Deep expertise in Kubernetes at production scale, including orchestration, scheduling, and service design
• Strong understanding of distributed systems, networking, and performance optimization
• Experience designing and operating low-latency, high-throughput systems with strict P95/P99 latency requirements
• Hands-on experience with inference systems, including batching or micro-batching strategies, caching, and memory optimization
• Experience improving system performance using metrics-driven approaches (e.g., latency, throughput, utilization)
• Familiarity with mixed precision (BF16, FP8) and streaming inference workloads

Qualifications

• Preferred experience with inference frameworks such as vLLM, Triton, TensorRT-LLM, Ray Serve, or TorchServe
• Experience with GPU systems and performance optimization (CUDA, NCCL, RDMA, NUMA, GPU interconnects)
• Experience leading multi-team or org-level technical initiatives
• Exposure to large-scale AI/ML infrastructure or hyperscale cloud environments

Skills

• Expertise in building reliable, low-latency infrastructure
• Knowledge of distributed systems and networking
• Experience with performance optimization techniques
• Understanding of inference systems and frameworks
• Ability to lead and influence engineering direction

Benefits

• Medical, dental, and vision insurance
• Company-paid Life Insurance
• Voluntary supplemental life insurance
• Short and long-term disability insurance
• Flexible Spending Account (FSA)
• Health Savings Account (HSA)
• Tuition Reimbursement
• Ability to participate in Employee Stock Purchase Program (ESPP)
• Mental Wellness Benefits through Spring Health
• Family-Forming support provided by Carrot
• Paid Parental Leave
• Flexible, full-service childcare support with Kinside
• 401(k) with a generous employer match
• Catered lunch each day in our office and data center locations
• A casual work environment
• A work culture focused on innovative disruption

Pay

The base salary range for this role is $188,000 to $275,000. The starting salary will be determined based on job-related knowledge, skills, experience, and market location.

Schedule

CoreWeave offers a flexible schedule to accommodate the diverse needs of its employees.