Principal Site Reliability Engineer - Observability and Telemetry Platform

About the role

Site Reliability Engineering (SRE) at NVIDIA is an engineering discipline focused on designing, building, and maintaining large-scale production systems with high efficiency and availability. This role requires expertise in software and systems engineering, including knowledge in various systems, networking, coding, database management, capacity management, continuous delivery, and deployment, as well as open-source cloud technologies like Kubernetes and OpenStack.

Responsibilities

• Design, implement, and support operational and reliability aspects of large-scale observability & telemetry collection platform with a focus on performance at scale, real-time monitoring, logging, and alerting.
• Engage in and improve the whole lifecycle of services—from inception and design through deployment, operation, and refinement.
• Support services before they go live through activities such as system design consulting, developing software tools, platforms, and frameworks, capacity management, and launch reviews.
• Maintain services once they are live by measuring and monitoring availability, latency, and overall system health.
• Scale systems sustainably through mechanisms like automation, and evolve systems by pushing for changes that improve reliability and velocity.
• Practice sustainable incident response and blameless postmortems.
• Participate in an on-call rotation to support production systems.

Requirements

• Bachelor’s degree in Computer Science or a related technical field involving coding (e.g., physics or mathematics), or equivalent experience.
• 15+ years of experience with infrastructure automation, distributed systems design.
• Experience with design, develop tools for running large-scale private or public cloud systems in production.
• 8+ years experience delivering foundational infrastructure and observability platforms.
• Experience in one or more of the following: Python, Go, Perl, or Ruby.
• In-depth knowledge on Linux, Networking, and Containers.

Qualifications

• Interest in crafting, analyzing, and fixing large-scale distributed systems.
• A systematic problem-solving approach, coupled with strong communication skills and a sense of ownership and drive.
• Ability to debug and optimize code and automate routine tasks.
• Experience in using or running large private and public cloud systems based on Kubernetes, OpenStack, and Docker.
• Experience running Grafana, OpenTelemetry, Prometheus, and similar observability-focused tools.

Skills

• Python
• Go
• Perl
• Ruby
• Linux
• Networking
• Containers

Benefits

Base salary range: $248,000 - $396,750 USD. Eligible for equity and benefits.

Pay

Base salary will be determined based on location, experience, and the pay of employees in similar positions.

Schedule

Eligible for on-call rotation to support production systems.