Sr Systems Development Engineer, AWS Hardware Engineering Services, AI UltraServers
Amazon Web Services (AWS) · Cupertino, CA · 2 mo ago
EngineeringFull-time
Fleet Health & Predictive Infrastructure
- Build and own the automation infrastructure for accelerator (AI/ML) fleet health at a large scale of servers, driving toward zero-touch operations that detect, diagnose, triage, and remediate faults without human intervention
- Design and develop test frameworks, test coverage strategies, and diagnostic tooling to validate hardware functionality, detect faults, and ensure qualification coverage across the platform lifecycle
- Design predictive failure detection using telemetry, sensor data, error trending, and log correlation to identify degrading components before customer impact
- Define and track fleet health metrics: failure rates, mean time to detect and resolve issues, first-time fix rate, test dwell time, and predictive accuracy
- Debugging & Troubleshooting
- Debug complex system-level issues across compute, GPU, and networking in production — including Linux boot/runtime failures, PCIe, power, NIC, NVMe, and GPU subsystems on x86 and ARM
- Perform root cause analysis correlating across firmware, kernel, driver, and physical layer; feed findings into manufacturing quality and design improvements
- Design scalable test automation for hardware bring-up, regression, and qualification — reducing manufacturing test cycle times without sacrificing coverage through intelligent test sequencing and parallel execution
- Build data pipelines correlating test results, sensor telemetry, and component-level data to identify systemic yield issues and drive upstream fixes
- Develop and maintain Linux device drivers on ARM and x86; work with OS internals and accelerator/GPU software stacks
- Build and manage tests covering all functional aspects of the system and CI/CD pipelines for rapid deployment to manufacturing lines and production fleet
- Engage with ODMs and design partners on testability, diagnostic coverage, and automation requirements during hardware design and bring-up phases — influencing functional and performance readiness of the platform
- Partner with datacenter operations to close the loop between field failures, manufacturing escapes, and design improvements
- Participate in post-incident reviews, identify contributing causes and drive permanent fixes that eliminate whole classes of risk
- Produce clear, maintainable documentation for systems, runbooks, and automation to enable others to operate and extend your work
- Drive process improvements that increase team agility — reducing development friction, eliminating unnecessary gates, and improving delivery velocity