Software Engineer II - AI/ML, AWS Neuron

About the role

The Annapurna Labs team at Amazon builds AWS Neuron, the software development kit used to accelerate deep learning and GenAI workloads on Amazon’s custom machine learning accelerators, Inferentia and Trainium. The AWS Neuron SDK is the backbone for accelerating deep learning and GenAI workloads on Amazon's Inferentia and Trainium ML accelerators. This comprehensive toolkit includes an ML compiler, runtime, and application framework that seamlessly integrates with popular ML frameworks like PyTorch and JAX enabling unparalleled ML inference and training performance.

The Inference Enablement and Acceleration team runs a wide range of models and supports novel architectures while maximizing their performance for AWS's custom ML accelerators. Working across the stack from PyTorch to the hardware-software boundary, engineers on the team build systematic infrastructure, innovate new methods, and create high-performance kernels for ML functions, ensuring every compute unit is fine-tuned for optimal performance for demanding workloads. The team combines deep hardware knowledge with ML expertise to push the boundaries of AI acceleration.

As part of the broader Neuron organization, the team works across multiple technology layers—from frameworks and kernels to compiler, runtime, and collectives. Responsibilities include optimizing current performance and contributing to future architecture designs, working closely with customers to enable their models and ensure optimal performance. This role offers a unique opportunity to work at the intersection of machine learning, high-performance computing, and distributed architectures, shaping the future of AI acceleration technology.

You will architect and implement business-critical features and mentor a brilliant team of experienced engineers. The team operates in large problem spaces while remaining small and agile, with no established blueprint—focusing on invention and experimentation in a unique learning culture. The team works directly with customers on model enablement, providing optimization expertise to ensure ML workloads achieve peak performance on AWS ML accelerators, and collaborates with open-source ecosystems to deliver seamless integration and performance at scale.

Key responsibilities

• Lead efforts to build distributed inference support for PyTorch in the Neuron SDK.
• Tune models to maximize performance and efficiency on AWS Trainium and Inferentia silicon and servers.
• Design, develop, and optimize ML models and frameworks for deployment on custom ML hardware accelerators.
• Participate in all stages of the ML system development lifecycle, including distributed architecture design, implementation, performance profiling, hardware-specific optimizations, testing, and production deployment.
• Build infrastructure to systematically analyze and onboard multiple models with diverse architectures.
• Design and implement high-performance kernels and features for ML operations, leveraging Neuron architecture and programming models.
• Analyze and optimize system-level performance across multiple generations of Neuron hardware.
• Conduct detailed performance analysis using profiling tools to identify and resolve bottlenecks.
• Implement optimizations such as fusion, sharding, tiling, and scheduling.
• Conduct comprehensive testing, including unit and end-to-end model testing with continuous deployment and releases through pipelines.
• Work directly with customers to enable and optimize their ML models on AWS accelerators.
• Collaborate across teams to develop innovative optimization techniques.

About the team

The Inference Enablement and Acceleration team fosters a builder’s culture where experimentation is encouraged and impact is measurable. Collaboration, technical ownership, and continuous learning are emphasized. The team supports new members through mentorship and knowledge-sharing, with a mix of experience levels and tenures. Senior members provide one-on-one mentoring and thorough code reviews, focusing on career growth and empowering engineers to take on complex tasks.

Basic qualifications

• 3+ years of non-internship professional software development experience.
• Bachelor's degree in computer science or equivalent.
• 3+ years of non-internship design or architecture experience (design patterns, reliability, and scaling) of new and existing systems.
• Fundamentals of machine learning and LLMs, including architecture, training and inference lifecycles, and experience with optimizations for improving model execution.
• Software development experience in C++ and Python.
• Strong understanding of system performance, memory management, and parallel computing principles.
• Proficiency in debugging, profiling, and implementing best software engineering practices in large-scale systems.

Preferred qualifications

• Familiarity with PyTorch, JIT compilation, and AOT tracing.
• Familiarity with CUDA kernels or equivalent ML or low-level kernels.
• Experience with performant kernel development such as CUTLASS, FlashInfer, etc.
• Familiarity with syntax and tile-level semantics similar to Triton.
• Experience with online/offline inference serving with vLLM, SGLang, TensorRT, or similar platforms in production environments.
• Deep understanding of computer architecture, operating systems-level software, and working knowledge of parallel computing.