Staff Software Engineer - Infrastructure Storage

About the role

We are seeking a seasoned Staff Storage Software Engineer with deep experience designing and deploying storage protocol solutions at scale across object, block, and file paradigms. This is a unique opportunity to work at the intersection of large-scale distributed systems and the rapidly evolving field of artificial intelligence infrastructure. This is an opportunity to have a significant impact on the future of AI. You will be building the foundational infrastructure that powers some of the most advanced AI research and products in the world.

Responsibilities

• Technical Leadership:Set technical direction for storage software architecture across the Infrastructure Engineering organization, influencing decisions that span petabyte-scale deployments.
• Author and review design documents for new storage systems, protocols, and integrations; raise the technical bar across the team.
• Mentor and develop senior engineers, providing guidance on systems design, debugging complex distributed systems issues, and navigating technical tradeoffs.
• Serve as a technical anchor for cross-functional initiatives involving storage, networking, compute, and control plane teams.
• Represent the storage software team in architectural reviews, roadmap planning, and customer-facing technical discussions where needed.
• Execution:Design, develop, and maintain high-performance storage systems software with a focus on performance, scalability, reliability, and operational simplicity.
• Implement and optimize storage protocol APIs across file (NFS, SMB, Lustre), block (NVMe-oF, iSCSI, Fibre Channel), and object (S3) access patterns.
• Develop distributed systems for managing and orchestrating storage resources across multiple solutions and redundant arrays.
• Collaborate with hardware and system architects to integrate software with storage solutions including NVMe, GPU-direct storage, and DPU-accelerated data paths.
• Troubleshoot and resolve complex issues in production data center environments, including performance regressions, protocol mismatches, and hardware failures.
• Contribute across the full software development lifecycle — from requirements gathering and system design through deployment, monitoring, and long-term maintenance.
• Build and maintain tooling for storage benchmarking, performance profiling, and capacity planning.
• Collaborate with the observability team to define, build, and track SLOs/SLIs for storage systems.
• Coordinate with Networking, Compute, and Storage Engineering teams to deploy high-performance distributed storage solutions that serve AI/ML workloads.
• Partner with the Fleet Engineering team to ensure seamless deployment, monitoring, and ongoing maintenance of distributed storage infrastructure.
• Partner with the control plane and Kubernetes teams to meet customer and product requirements for usability, reliability, and telemetry.
• Innovate:Stay current with the latest research and developments in AI and HPC storage technologies, and bring relevant advances into Lambda's infrastructure.
• Work with the Lambda product team to identify emerging trends in AI inference and training that will shape next-generation storage requirements.
• Evaluate and prototype new storage solutions, protocols, and hardware integrations - from open-source distributed filesystems to vendor-specific accelerated storage products.
• Optimize storage protocol solutions for AI workloads, including checkpoint I/O for training, high-throughput dataset serving, and latency-sensitive inference pipelines.

Requirements

• 10+ years of experience in storage systems engineering, with at least 5 years in a technical lead or Staff+ IC role.
• Proven track record designing and operating storage infrastructure at scale (multi-petabyte environments preferred) in production data center or cloud settings.
• Experience leading technical projects end-to-end, from architecture through delivery with cross-functional stakeholders.
• Background working in high-performance computing, AI/ML infrastructure, or large-scale cloud storage environments.
• Systems-Level Programming:Strong proficiency in one or more low-level systems programming languages: C, C++, Rust, or Go.
• Demonstrated ability to write high-performance, concurrent, production-grade systems code and conduct thorough code reviews.
• Experience with kernel-level storage drivers, user-space I/O frameworks, or storage daemon development is a strong plus.
• Familiarity with DPDK and SPDK and their role in building high-performance, kernel-bypass storage and networking data paths.
• Experience with GPU-direct storage or similar zero-copy data paths is a plus.
• Physical Infrastructure & Operational Acumen:Comfort working in a physical data center environment — understanding rack-scale infrastructure, storage array hardware, cabling, and failure domains.
• Experience building and operating storage systems with strong reliability expectations: designing for failure, building runbooks, and driving incident response.
• Familiarity with storage observability tooling — metrics pipelines (Prometheus, Grafana), log aggregation, and tracing in distributed storage environments.

Qualifications

• Experience with NVMe, NVMe-oF, and RDMA (RoCE or InfiniBand) and their impact on storage system architecture.
• Familiarity with DPUs (e.g., NVIDIA BlueField) and their role in offloading storage and networking data paths.
• Experience with Ceph at scale (100PB+) in an HPC or AI infrastructure environment.
• Familiarity with emerging storage technologies such as CXL memory pooling, computational storage, or ZNS (Zoned Namespace) SSDs.
• Experience contributing to or maintaining open-source storage projects (e.g., Ceph, DAOS, Lustre, MinIO).

Skills

• Deep hands-on experience with two or more storage protocols: object (S3 or similar), block (iSCSI, Fibre Channel, NVMe-oF), or file (NFS, SMB, Lustre, DAOS).
• Experience implementing or maintaining storage protocol servers or clients in production, not just consuming them.
• Familiarity with storage API performance characteristics such as latency, throughput, IOPS and the ability to diagnose and resolve bottlenecks at the protocol level.
• Experience profiling and tuning storage systems for throughput, latency, and IOPS under real production workloads.
• Familiarity with tools such as fio, blktrace, perf, eBPF/bpftrace, or equivalent for storage performance analysis.
• Understanding of I/O scheduling, caching layers, write amplification, and related performance tradeoffs.
• Working knowledge of NVMe, NVMe-oF, and RDMA and their impact on storage system architecture.
• Familiarity with DPUs (e.g., NVIDIA BlueField) and their role in offloading storage and networking data paths.
• Experience with GPU-direct storage or similar zero-copy data paths is a plus.

Benefits

• Salary Range: $314K - $465K

Pay

• Salary Range: $314K - $465K

Schedule

• Presence in our San Francisco/San Jose/Bellevue office location 4 days per week; Lambda’s designated work from home day is currently Tuesday.

Benefits

• Health, dental, and vision coverage for you and your dependents.
• Wellness and commuter stipends for select roles.
• 401k Plan with 2% company match (USA employees).
• Flexible paid time off plan that we all actually use.

Equal Opportunity Employer

Lambda is an Equal Opportunity employer. Applicants are considered without regard to race, color, religion, creed, national origin, age, sex, gender, marital status, sexual orientation and identity, genetic information, veteran status, citizenship, or any other factors prohibited by local, state, or federal law.