Distributed LLM Inference Engineer

About the role

This role is critical to Anyscale’s mission of democratizing distributed computing. You will iterate quickly with product teams to ship end-to-end solutions for batch and online inference at large scale, ensuring low cost solutions for large-scale ML inference.

Responsibilities

• Iterate very quickly with product teams to ship the end to end solutions for Batch and Online inference at high scale
• Integrate with Open source software like vLLM, work closely with the community to adopt these techniques in Anyscale solutions, and also contribute improvements to open source
• Follow the latest state-of-the-art in the open source and the research community, implementing and extending best practices

Requirements

• Familiarity with running ML inference at large scale with high throughput and low latency
• Familiarity with deep learning and deep learning frameworks (e.g. PyTorch)
• Solid understanding of distributed systems, ML inference challenges

Qualifications

• ML Systems knowledge
• Experience using Ray
• Work closely with community on LLM engines like vLLM, TensorRT-LLM
• Contributions to deep learning frameworks (PyTorch, TensorFlow)
• Contributions to deep learning compilers (Triton, TVM, MLIR)
• Prior experience working on GPUs / CUDA

Skills

• Deep learning and deep learning frameworks (e.g. PyTorch)
• Distributed systems
• Optimizations for ML inference

Benefits

• Stock Options
• Healthcare plans, with premiums covered by Anyscale at 99% for both employees and dependents
• 401k Retirement Plan
• Education & Wellbeing Stipend
• Paid Parental Leave
• Fertility Benefits
• Paid Time Off
• Commute reimbursement
• 100% of in-office meals covered

Pay

Market-based compensation with data-driven, transparent, and consistent adjustments based on market data.

Schedule

Full-time