Software Engineer, Production Inference (Distributed Inference)

About the role

We're hiring a Software Engineer to build and scale the distributed production inference systems that serve Inkling, Inkling-Small, and Tinker in production. You'll own the systems that turn trained models into fast, reliable, cost-efficient services — from request routing and batching to multi-node serving and GPU utilization at scale. This is a systems-heavy, production-first role. You'll work closely with research and infrastructure teams to translate rapidly evolving model architectures into serving systems that meet real-world latency, throughput, and reliability requirements, and you'll be on the front line when production inference systems need to scale, recover, or improve.

Responsibilities

• Design, build, and operate distributed infrastructure for large-scale model serving, including request routing, load balancing, batching, and multi-node coordination
• Optimize inference latency and throughput in production, including work on KV cache management, continuous batching, speculative decoding, and quantization
• Build and maintain high-concurrency serving systems with strong uptime, low tail latency, and deep observability
• Benchmark, tune, and extend inference engines to support new model architectures as they move from research into production
• Partner with research and infrastructure teams to translate emerging model designs into production-ready serving systems
• Build tooling for tracing, debugging, and resolving issues across the serving stack, from orchestration down to GPU kernels
• Participate in on-call rotation to support production inference systems

Skills & Qualifications

• 3+ years of experience building and operating distributed systems in production
• Strong systems programming skills in Python, C++, Rust, or similar languages
• Experience with production infrastructure at scale: reliability, observability, and performance under real-world load
• Solid understanding of networking, concurrency, and distributed systems fundamentals
• Experience with LLM inference engines such as vLLM, SGLang, or TensorRT-LLM
• Familiarity with GPU programming (CUDA) or low-level performance optimization
• Experience with model parallelism, tensor/pipeline parallelism, or other distributed inference techniques
• Track record of operating large-scale production systems with strict latency and uptime requirements
• Experience with Kubernetes or similar orchestration systems for GPU workloads
• Contributions to open-source ML systems or inference infrastructure projects

Benefits

• Generous health, dental, and vision benefits
• Unlimited PTO
• Paid parental leave
• Relocation support as needed

Pay

Depending on background, skills and experience, the expected annual salary range for this position is $350,000 - $500,000 USD (placeholder — verify against current internal bands before publishing).

Location: San Francisco, CA.

Visa sponsorship: We sponsor visas.