Principal Production Engineer

About the role

We are looking for a Principal Production Engineer to join our team. This role is available as a hybrid opportunity 3 days a week in San Jose, CA or Remote reporting to Production Engineering in the Cloud Infrastructure & Operations department.

Responsibilities

• Design and implement highly available, scalable infrastructure across AWS, GCP, and bare-metal environments
• Drive an "automation-first" culture by writing code (Python/Go) to eliminate manual toil and build self-healing systems
• Implement and maintain sophisticated observability (Prometheus, Grafana, OpenTelemetry), define SLIs/SLOs, and establish error budgets
• Act as a lead Incident Commander (TDO on-call), develop response playbooks, and conduct deep-dive post-incident analyses
• Partner with Engineering and partner teams to conduct operability reviews

Requirements

• 10+ years of experience managing reliability, scalability, and availability for large-scale production services
• Deep expertise in programming (e.g., Python, Go, or C/C++)
• Strong background in networking protocols, Linux/RHEL systems, and distributed architecture
• Experience in high-stakes incident management and participation in a 24/7 on-call rotation
• Proficiency in leveraging ITIL frameworks and incident data to drive service maturity through systematic problem management and technical operability reviews

Qualifications

• Extensive experience with public cloud (AWS, Azure, GCP)
• Infrastructure-as-Code (Ansible, Terraform, Helm, Temporal)
• Experience with chaos engineering and disaster recovery planning at scale
• Expertise in global routing (BGP) and traffic tunneling (GRE, IPSec)
• Deep understanding of L7 proxy architectures (HAProxy), DNS at scale, and OS networking stack internals

Skills

• Data-driven, valuing evidence over assumptions
• Think at scale, building solutions and processes built to last
• Pragmatic builder obsessed with creating, iterating, and shipping
• Act like an owner with a bias for action and integrity

Benefits

• Variety of health plans
• Time off plans for vacation and sick time
• Parental leave options
• Retailment options
• In-office perks, and more!