Software Engineer - Training/Inference (C++)

About the role

The Member of Technical Staff - Inference position at xAIxAI is dedicated to developing the high-performance inference platform that powers Grok, ensuring lightning-fast and flawless user experiences.

Responsibilities

• Architect and implement scalable distributed infrastructure for model serving (load balancing, auto-scaling, batch scheduling, global KV cache).
• Optimize latency and throughput of model inference under real production workloads.
• Build reliable, high-concurrency serving systems that serve billions of users with 100% uptime, 0% error rate, and excellent tail latency.
• Benchmark, fine-tune, and accelerate inference engines (including low-level GPU kernel work and code generation).
• Create custom tools to trace, replay, and fix issues across the full stack — from orchestration down to GPU kernels.
• Develop robust CI/CD infrastructure for seamless endpoint deployment, image publishing, and inference engine updates.
• Accelerate research on scaling test-time compute, RL rollout, and model-hardware co-design for next-generation systems.

Qualifications

• Deep low-level systems programming (C/C++ or Rust)
• Experience with large-scale, high-concurrent production serving
• Experience with GPU inference engines (vLLM, SGLang, Triton, TensorRT-LLM, etc.)
• Strong background in system optimizations: batching, caching, load balancing, parallelism
• Low-level inference optimizations: GPU kernels, code generation
• Algorithmic inference optimizations: quantization, speculative decoding, distillation, low-precision numerics
• Experience with testing, benchmarking, and reliability of inference services
• Experience designing and implementing CI/CD infrastructure for inference

Pay

$180,000 - $440,000 USD

Schedule

N/A

Benefits

• Base salary
• Equity
• Comprehensive medical, vision, and dental coverage
• Access to a 401(k) retirement plan
• Short & long-term disability insurance
• Life insurance
• Variety of other discounts and perks

Skills

N/A

Benefits

xAI is an equal opportunity employer. For details on data processing, view our Recruitment Privacy Notice.