Senior Site Reliability Engineer, DGX Cloud
Thomas To · Santa Clara, CA · 1 mo ago
EngineeringFull-time
About the role
NVIDIA is driving AI and high‑performance computing forward. The DGX Cloud team delivers a fully managed AI platform on major cloud providers, optimizing AI workloads with high‑performance NVIDIA infrastructure. As a Senior Site Reliability Engineer you will maintain high‑performance DGX Cloud clusters for AI researchers and enterprise clients worldwide, working at the forefront of innovative AI and cloud computing solutions.
Responsibilities
- Build, implement and support operational and reliability aspects of large‑scale Kubernetes clusters with a focus on performance at scale, real‑time monitoring, logging, and alerting.
- Define SLOs/SLIs, monitor error allowances, and streamline reporting.
- Support services before launch through system creation consulting, developing software tools, platforms and frameworks, capacity management, and launch reviews.
- Maintain live services by measuring and supervising availability, latency, and overall system health.
- Operate and optimize GPU workloads across AWS, GCP, Azure, OCI, and private clouds.
- Scale systems sustainably through automation and evolve systems to improve reliability and velocity.
- Lead triage and root‑cause analysis of high‑severity incidents.
- Practice balanced incident response and conduct blameless post‑mortems.
- Participate in on‑call rotation to support production services.
Requirements
- BS in Computer Science or a related technical field, or equivalent experience.
- 8+ years of experience operating production services.
- Expert‑level knowledge of Kubernetes administration, containerization, and microservices architecture.
- Experience with infrastructure automation tools (e.g., Terraform, Ansible, Chef, Puppet).
- Proficiency in at least one high‑level programming language (e.g., Python, Go).
- In‑depth knowledge of Linux operating systems, networking fundamentals (TCP/IP), and cloud security standards.
- Solid grasp of SRE principles such as SLOs, SLIs, error budgets, and incident management.
- Experience building and operating comprehensive observability stacks (monitoring, logging, tracing) using tools like OpenTelemetry, Prometheus, Grafana, ELK Stack, Lightstep, Splunk, etc.
Skills
- Operating GPU‑accelerated clusters with KubeVirt in production.
- Applying generative‑AI techniques to reduce operational toil.
- Experience with workflow orchestration platforms such as Temporal, Cadence, Airflow, Argo Workflows, or Step Functions.
- Operating and troubleshooting production AI inference workloads across the model‑to‑GPU stack, including vLLM, SGLang, PyTorch, TensorRT‑LLM, NVIDIA Dynamo, CUDA, NCCL, and GPU performance analysis.
Pay
Base salary is determined by location, experience, and internal equity. The base salary range is $168,000 – $270,250 USD for Level 4 and $208,000 – $333,500 USD for Level 5. The role also includes equity and benefits.
Applications will be accepted at least until September 19, 2026.