Member of Technical Staff - Inference

Core Technical Responsibilities

• Build a multi-tenant LLM serving platform that operates across our cloud GPU fleets.
• Design placement and scheduling algorithms for heterogeneous accelerators.
• Implement multi-region/zone failover and traffic shifting for resilience and cost control.
• Build autoscaling, routing, and load balancing to meet throughput/latency SLOs.
• Optimize model distribution and cold-start times across clusters.
• Integrate and contribute to LLM inference frameworks such as vLLM, SGLang, TensorRT-LLM.
• Optimize configurations for tensor/pipeline/expert parallelism, prefix caching, memory management and other axes for maximum performance.
• Profile kernels, memory bandwidth and transport; apply techniques such as quantization and speculative decoding.
• Develop reproducible performance suites (latency, throughput, context length, batch size, precision).
• Embed and optimize distributed inference within our RL stack.
• Establish CI/CD with artifact promotion, performance gates, and reproducible builds.
• Build metrics, logs, tracing; structured incident response and SLO management.
• Document architectures, playbooks, and API contracts; mentor and collaborate cross-functionally.

Technical Requirements

• Building ML Systems at Scale: 3+ years building and running large-scale ML/LLM services with clear latency/availability SLOs.
• Inference Backends: Hands-on with at least one of vLLM, SGLang, TensorRT-LLM.
• Distributed Serving Infra: Familiarity with distributed and disaggregated serving infrastructure such as NVIDIA Dynamo.
• Inference Internals: Deep understanding of prefill vs. decode, KV-cache behavior, batching, sampling, speculative decoding, parallelism strategies.
• Full-Stack Debugging: Comfortable debugging CUDA/NCCL, drivers/kernels, containers, service mesh/networking, and storage, owning incidents end-to-end.
• Infrastructure Skills: Python: Systems tooling and backend services. PyTorch: LLM Inference engine development and integration, deployment readiness. Cloud & Automation: AWS/GCP service experience, cloud deployment patterns. Kubernetes: Running infrastructure at scale with containers on Kubernetes. GPU & Networking: Architecture, CUDA runtime, NCCL, InfiniBand; GPU-aware bin-packing and scheduling across heterogeneous fleets.

Nice to Have

• Kernel-Level Optimization: Familiarity with CUDA/Triton kernel development; Nsight Systems/Compute profiling.
• Systems Performance Languages: Rust, C++.
• Data & Observability: Kafka/PubSub, Redis, gRPC/Protobuf; Prometheus/Grafana, OpenTelemetry; reliability patterns.
• Infra & Config Automation: Terraform/Ansible, infrastructure-as-code, reproducible environments.
• Open Source: Contributions to serving, inference, or RL infrastructure projects.

What we offer

• Cash Compensation Range of $150-300k with significant equity incentives.
• Flexible work arrangement (remote or San Francisco office).
• Full visa sponsorship and relocation support.
• Professional development budget.
• Regular team off-sites and conference attendance.
• Opportunity to shape decentralized AI and RL at Prime Intellect.

Growth Opportunity

• You'll join a team of experienced engineers and researchers working on cutting-edge problems in AI infrastructure.
• We believe in open development and encourage team members to contribute to the broader AI community through research and open-source contributions.
• We value potential over perfection.