Lead Principal Site Reliability Engineer

About the role

Serves as a consultant and leads the design and architecture of infrastructure and service, ensuring alignment with reliability and functionality standards. Takes full ownership of forecasting of demands and responding to capacity needs. Owns collaborations with software development teams to develop reliable and scalable infrastructures. Recommends methods for performing data collection to maintain and optimize operations and reliability. Oversees incident response and/or maintenance tasks. Provides strategic, future-oriented health and performance reporting. Contributes to strategies for automation and reviews the development and implementation of automation. Provides expert-level communication about services and anticipates, analyzes, and explains the impact of changes, considering strategic goals. Serves as a role model in providing support for technology and reviews documentation for accuracy. Leads the implementation of innovative tools and provides expertise in site reliability trends.

Responsibilities

• Define and drive the site reliability engineering strategy for large-scale, distributed, and business-critical platforms.
• Establish reliability standards, engineering practices, and operational readiness requirements across multiple teams.
• Serve as a senior technical authority for system reliability, scalability, resilience, performance, and production operations.
• Influence architecture and design decisions to ensure systems are supportable, observable, fault tolerant, and capable of meeting availability objectives.
• Identify systemic reliability risks and lead cross-functional initiatives to address them.
• Provide technical direction and mentorship to site reliability engineers, software engineers, platform engineers, and operations teams.
• Lead technical reviews and promote consistent engineering practices across the organization.
• Define and implement service-level indicators, service-level objectives, error budgets, and operational health metrics.
• Develop comprehensive monitoring, logging, tracing, alerting, and observability strategies.
• Improve the quality and actionability of alerts while reducing unnecessary operational noise.
• Establish dashboards and reporting mechanisms that clearly communicate service health, performance, capacity, and risk.
• Use production data and reliability trends to prioritize engineering investments and continuous-improvement initiatives.
• Design and implement automation that reduces manual intervention, operational toil, and human error.
• Build or enhance tools for deployment, configuration management, infrastructure provisioning, incident response, and service recovery.
• Promote infrastructure-as-code, policy-as-code, automated testing, and repeatable deployment practices.
• Partner with development teams to improve continuous integration and continuous delivery pipelines.
• Develop self-healing and automated remediation capabilities where appropriate.
• Contribute production-quality software and reusable platform components using modern programming and scripting languages.
• Provide technical leadership during complex, high-severity production incidents.
• Coordinate diagnosis, containment, recovery, and stakeholder communication during service disruptions.
• Lead blameless post-incident reviews and ensure that corrective actions address root causes rather than symptoms.
• Identify recurring failure patterns and develop long-term engineering solutions.
• Improve incident-management processes, escalation procedures, runbooks, and recovery playbooks.
• Participate in an on-call rotation or provide senior escalation support for critical services, as required.
• Lead capacity planning, performance analysis, load testing, and demand forecasting for critical platforms.
• Identify performance bottlenecks and recommend architectural or operational improvements.
• Design and validate high-availability, disaster-recovery, backup, and business-continuity capabilities.
• Lead resilience testing, failure-mode analysis, game days, and controlled fault-injection exercises.
• Ensure recovery-time and recovery-point objectives are defined, tested, and achievable.
• Partner with security and compliance teams to embed security into infrastructure, automation, and operational practices.
• Support vulnerability remediation, access-control improvements, audit readiness, and secure configuration management.
• Ensure production environments meet organizational standards for change management, data protection, and operational governance.
• Balance reliability, security, delivery speed, cost, and business priorities when recommending technical solutions.

Qualifications

• Extensive professional experience in site reliability engineering, software engineering, cloud infrastructure, platform engineering, systems engineering, or a related technical discipline.
• Demonstrated experience designing, operating, and improving highly available production systems at significant scale.
• Deep knowledge of distributed systems, cloud architecture, networking, operating systems, storage, databases, and service dependencies.
• Advanced experience with at least one major cloud platform, such as Oracle Cloud Infrastructure, Amazon Web Services, Microsoft Azure, or Google Cloud Platform.
• Strong experience with containerization and orchestration technologies, including Docker and Kubernetes.
• Proven expertise with infrastructure-as-code and configuration-management technologies such as Terraform, Ansible, Chef, Puppet, or equivalent tools.
• Experience implementing observability solutions using metrics, logs, traces, dashboards, and automated alerting.
• Strong programming or scripting skills in one or more languages such as Python, Go, Java, JavaScript, Bash, or similar.
• Experience with continuous integration, continuous delivery, automated testing, and modern release-management practices.
• Demonstrated leadership during critical production incidents and complex technical investigations.
• Able to diagnose difficult system issues across applications, infrastructure, networks, databases, and cloud services.
• Strong written and verbal communication skills, including the ability to explain technical risk and recommendations to engineering leaders and business stakeholders.
• Proven ability to lead cross-functional technical initiatives without relying solely on formal authority.