Member of Technical Staff — Model Optimization and Inference (Experienced)

About the role

We can train a great model. The next problem is making it fast enough to actually use in a real-time conversation — and that gap is enormous. A model that responds in 3 seconds is a demo. A model that responds in under 500ms is a product.

This posting is aimed at experienced engineers and researchers who’ve operated at a senior to senior-staff level at big tech, a leading AI lab, or a high-traffic inference team. Everyone at Nuance is MTS — we don’t run title ladders — but we’re hiring people who have already done this work at scale.

What You’ll Do

• Own end-to-end inference optimization across our model stack — LLMs, audio models, and diffusion-based components
• Implement and tune KV cache strategies for long-context conversations, including eviction policies, compression, and memory-efficient attention
• Evaluate, deploy, and extend inference serving frameworks (vLLM, SGLang, TensorRT-LLM, etc.) for our specific workloads
• Profile and benchmark end-to-end latency and throughput; identify and systematically eliminate bottlenecks
• Build internal tooling that makes optimization work faster and more rigorous — profiling viewers, end-to-end inference test harnesses, and other infrastructure that helps the team move quickly
• Accelerate diffusion model inference — consistency models, step distillation, caching strategies, and custom kernel optimizations
• Apply and develop quantization techniques (INT8, INT4, GPTQ, AWQ, and beyond) to reduce memory footprint and increase throughput without meaningfully degrading quality
• Work closely with research and infrastructure to ensure new models ship with optimized serving from day one

What We’re Looking For

• Significant hands-on experience with LLM inference optimization — you’ve shipped work on KV caching, memory layout, attention kernels, or batching strategies in a production or high-traffic research context
• Proven proficiency with inference serving frameworks — vLLM, SGLang, TensorRT-LLM, or similar — including going well beyond default configurations and adapting them to non-standard workloads
• Experience optimizing diffusion model inference (latency reduction, step distillation, caching, or kernel-level work)
• Strong Python and PyTorch skills; comfort reading and writing CUDA or Triton kernels is a significant plus
• A systematic approach to profiling and optimization — you measure first, then optimize
• Familiarity with speculative decoding or other inference-time acceleration techniques

Bonus Points

• Hands-on experience with post-training quantization (GPTQ, AWQ, or similar) and a clear sense of quality/performance tradeoffs
• Familiarity with multimodal or streaming inference architectures
• Experience deploying real-time AI systems with hard latency SLAs
• Prior work at an AI lab, inference startup, or on a high-traffic model serving platform
• Contributions to open-source inference frameworks