Director, Product Management – AI Inference Platform

About the role

The future isn’t just about technology - it’s about people and the limitless possibilities AI can unlock for us all. As AI transforms how we live, work, and connect, Arm is at its very foundation, powering the innovations that are shaping our world.

Responsibilities

• Define and drive the product vision, strategy, and roadmap for large-scale AI inference systems
• Own product direction across compute execution, inference serving, and control plane systems
• Drive inference architecture and orchestration strategy, including distributed serving, routing, batching, and scheduling
• Define capabilities for KV cache management, memory optimization, and token lifecycle efficiency (prefill vs decode)
• Partner with engineering to enable hardware–software co-design, improving performance across accelerators and interconnects
• Shape the development of inference platform capabilities that deliver measurable gains in latency, throughput, and cost efficiency
• Own and evolve control plane services (APIs, policy engines, context/state management, usage/accounting)
• Translate complex technical systems into clear product value and customer impact
• Align multi-functional collaborators across infrastructure, research, and product organizations
• Define and track key performance metrics (P99 latency, TTFT, throughput, cost per inference/token, availability)
• Influence platform direction across constantly evolving AI workloads (LLMs, multimodal, agentic systems)

Required Skills and Experience

• Proven experience in product management within AI/ML infrastructure, cloud platforms, or distributed systems
• Solid understanding of AI inference systems and large-scale serving architectures
• Demonstrated ability to deliver scalable, high-performance platform products
• Strong technical expertise and ability to collaborate directly with engineering teams

Nice To Have Skills and Experience

• Experience with accelerator-based systems and performance optimization
• Familiarity with modern inference frameworks (e.g., TensorRT-LLM, vLLM)
• Experience working on production-scale AI systems

Pay

$260,000-$351,800 per year

Schedule

Hybrid working is offered