Software Engineer - Model Products

Responsibilities

• Design, build, and operate the Model APIs surface with focus on advanced inference capabilities: structured outputs (JSON mode, grammar-constrained generation), tool/function calling and multi-modal serving
• Profile and optimize TensorRT-LLM kernels, analyze CUDA kernel performance, implement custom CUDA operators, tune memory allocation patterns for maximum throughput and optimize communication patterns across multi-GPU setups
• Productionize performance improvements across runtimes with deep understanding of their internals: speculative decoding implementations, guided generation for structured outputs, custom scheduling and routing algorithms for high-performance serving
• Build comprehensive benchmarking frameworks that measure real-world performance across different model architectures, batch sizes, sequence lengths, and hardware configurations
• Productionize performance improvements across runtimes (e.g.TensorRT, TensorRT-LLM): speculative decoding, quantization, batching, and KV-cache reuse
• Instrument deep observability (metrics, traces, logs) and build repeatable benchmarks to measure speed, reliability, and quality
• Implement platform fundamentals: API versioning, validation, usage metering, quotas, and authentication
• Collaborate closely with other teams to deliver robust, developer-friendly model serving experiences

Requirements

• 3+ years experience building and operating distributed systems or large-scale APIs
• Proven track record of owning low-latency, reliable backend services (rate-limiting, auth, quotas, metering, migrations)
• Infra instincts with performance sensibilities: profiling, tracing, capacity planning, and SLO management
• Comfortable debugging complex systems, from runtime internals to GPU execution traces
• Strong written communication; able to produce clear design docs and collaborate across functions

Nice to Have

• Experience with LLM runtimes (vLLM, SGLang, TensorRT-LLM) or contributions to open-source inference engines (vLLM, TensorRT-LLM, SGLang, TGI)
• Knowledge of Kubernetes, service meshes, API gateways, or distributed scheduling
• Background in developer-facing infrastructure or open-source APIs

Benefits

• Competitive compensation, including meaningful equity
• 100% coverage of medical, dental, and vision insurance for employee and dependents
• Flexible PTO policy including company-wide Winter Break (our offices are closed from Christmas Eve to New Year's Day!)
• Paid parental leave
• Fertility and family-building stipend through Carrot
• Company-facilitated 401(k)
• Exposure to a variety of ML startups, offering unparalleled learning and networking opportunities