Sr. Software Development Engineer – AI/ML Networking Disaggregated Inference, Annapurna Labs, Annapurna Labs

About the role

Every token a large language model generates depends on data reaching the right accelerator at the right moment. As AI models outgrow any single chip, the network between accelerators becomes the bottleneck that decides how fast—and how affordably—the world’s largest models can serve real users. Our team builds that network layer. We are looking for an engineer to work at the frontier of disaggregated inference: splitting LLM serving into separate pre‑fill and decode pools and moving the model’s KV cache between them at the absolute limit of what the hardware allows. Success is measured by how close you run to the theoretical peak of the machine.

Responsibilities

• Build and optimize low‑level data‑movement software across accelerators, servers, and heterogeneous memory over AWS’s highest‑performance network fabric.
• Profile performance, identify real bottlenecks, and close the gap between “it works” and “it runs as fast as physics permits.”
• Work across the stack—from kernel and network transport up to inference frameworks—and partner with teams building chips, runtimes, and models.
• Deliver features that run on our largest clusters for our largest customers, serving the largest AI models in production.

Requirements

• 5+ years of professional software development experience (non‑internship).
• 5+ years of programming experience with at least one software language.
• 5+ years leading design or architecture of new and existing systems (design patterns, reliability, scaling).
• 5+ years of full software development life‑cycle experience, including coding standards, code reviews, source control, build processes, testing, and operations.
• Experience as a mentor, tech lead, or engineering team lead.
• Strong C/C++ coding experience and a solid command of Linux, kernels, memory management, and writing fast, performance‑critical code.

Preferred qualifications

• Bachelor’s degree in computer science or equivalent.
• Experience with high‑speed networking or HPC interconnects (RDMA, InfiniBand, libfabric, UCX, NIXL, MPI) is highly valued.
• Embedded‑systems experience is a plus.
• Familiarity with ML communications libraries (NCCL, NIXL, NVSHMEM) is a plus.
• Prior AI/ML experience is welcome but not required.

Team and culture

You’ll join Annapurna Labs, an integral part of AWS that designs the hardware and software building blocks behind every EC2 instance. The team focuses on networking solutions for Machine Learning and High‑Performance Computing workloads on AWS. You’ll work side‑by‑side with infrastructure experts, hardware engineers, RTL engineers, scientists, and architects in a globally distributed, multicultural environment. Mentorship is a core value: you’ll receive senior mentorship and be expected to mentor junior engineers. The pace is fast, with flexibility in working hours and a strong emphasis on work‑life balance. Career growth opportunities are abundant, and continuous learning is encouraged as the AI/ML field evolves.

Benefits

• Sign‑on payments and restricted stock units (RSUs).
• Comprehensive health insurance (medical, dental, vision, prescription).
• Basic Life & AD&D insurance with optional supplemental life plans.
• Employee Assistance Program, mental health support, and medical advice line.
• Flexible Spending Accounts.
• Adoption and surrogacy reimbursement coverage.
• 401(k) matching.
• Paid time off and parental leave.

Pay

Base salary range (Cupertino, CA): $193,300 – $261,500 USD annually. Total compensation includes sign‑on payments and RSUs, with final amounts determined by experience, qualifications, and location.