LLM Inference Systems Engineer

Hippocratic AI is building a safety-focused healthcare artificial intelligence platform. The company is seeking an LLM Inference Systems Engineer to build and operate distributed LLM serving, high-performance networking, GPU inference, benchmarking, observability, and Kubernetes production systems. Responsibilities Design, implement, and operate disaggregated LLM serving architectures, including independent prefill and decode pools, KV-cache transfer, routing, batching, and failure recoveryOptimize multi-node inference across modern GPU systems using NVLink, NVSwitch, NVLS, InfiniBand, RoCEv2, GPUDirect RDMA, NCCL, UCX, and related communication pathsBuild and tune serving systems using SGLang, vLLM, NVIDIA Dynamo, TensorRT-LLM, or comparable frameworksProfile end-to-end performance across compute, memory, network, and storage; improve time to first token, inter-token latency, throughput, tail latency, and cost per tokenDevelop repeatable benchmark and capacity-planning workflows across model architectures, GPU types, parallelism strategies, and concurrency levelsProductionize serving on Kubernetes with health checks, autoscaling, safe rollouts, metrics, tracing, and actionable diagnosticsDiagnose complex distributed failures such as collective timeouts, topology mismatches, packet loss, congestion, KV-transfer stalls, GPU OOMs, and uneven loadPartner with model researchers and platform engineers to launch new models, serving features, and hardware generations safely