Principal Software Engineer – Large-Scale LLM Memory and Storage Systems

The NVIDIA Dynamo Principal Systems Engineer position is dedicated to defining the vision and roadmap for memory management of large-scale Large Language Model (LLM) and storage systems. The role involves designing and evolving a unified memory layer that spans GPU memory, pinned host memory, RDMA-accessible memory, SSD tiers, and remote file/object/cloud storage.

About the role

This position requires a deep understanding of memory hierarchies and experience in designing systems that span multiple tiers for performance and cost efficiency. It also demands hands-on experience with networked I/O and RDMA/NVMe-oF/NVLink technologies, as well as strong skills in profiling and optimizing systems across CPU, GPU, memory, and network.

Responsibilities

• Design and evolve a unified memory layer that supports large-scale LLM inference.
• Architect and implement deep integrations with leading LLM serving engines, focusing on KV-cache offload, reuse, and remote sharing across heterogeneous and disaggregated clusters.
• Co-design interfaces and protocols for disaggregated prefill, peer-to-peer KV-cache sharing, and multi-tier KV-cache storage (GPU, CPU, local disk, and remote memory).
• Partner with GPU architecture, networking, and platform teams to exploit GPUDirect, RDMA, NVLink, and similar technologies for low-latency KV-cache access and sharing across heterogeneous accelerators and memory pools.
• Mentor senior and junior engineers, set technical direction for memory and storage subsystems, and represent the team in internal reviews and external forums (open source, conferences, and customer-facing technical deep dives).

Requirements

• Masters or PhD or equivalent experience with 15+ years of experience building large-scale distributed systems, high-performance storage, or ML systems infrastructure in C/C++ and Python.
• Deep understanding of memory hierarchies and experience designing systems that span multiple tiers for performance and cost efficiency.
• Hands-on experience with networked I/O and RDMA/NVMe-oF/NVLink technologies, and familiarity with concepts like disaggregated and aggregated deployments for AI clusters.
• Strong skills in profiling and optimizing systems across CPU, GPU, memory, and network, using metrics to drive architectural decisions and validate improvements in TTFT and throughput.
• Excellent communication skills and prior experience leading cross-functional efforts with research, product, and customer teams.

Qualifications

• Experience designing unified memory or storage layers that expose a single logical KV or object model across GPU, host, SSD, and cloud tiers, especially in enterprise or hyperscale environments.
• Prior contributions to open-source LLM serving or systems projects focused on KV-cache optimization, compression, streaming, or reuse.
• Publications or patents in areas such as LLM systems, memory-disaggregated architectures, RDMA/NVLink-based data planes, or KV-cache/CDN-like systems for ML.

NVIDIA offers highly competitive salaries and a comprehensive benefits package, including equity and benefits. Applications for this job will be accepted at least until January 13, 2026. NVIDIA is an equal opportunity employer committed to fostering a diverse work environment and does not discriminate on the basis of race, religion, color, national origin, gender, gender expression, sexual orientation, age, marital status, veteran status, disability status, or any other characteristic protected by law.