Senior Site Reliability Engineer

About the role

The Vera C. Rubin Observatory is a new astronomy facility in Chile designed to create a 10-year time-lapse map of the southern sky through the Legacy Survey of Space and Time (LSST). As part of this team, you'll design, operate, and sustain the systems that process Rubin's data in near real time.

Responsibilities

• Ensure, through both architecture and practice, the reliable operation of the near-real-time data processing pipeline and timely delivery of alerts to downstream brokers.
• Design and develop software that reduces operational risk and improves system resilience, scalability, and usability, including addressing failure modes, error handling, and contention in shared resources.
• Improve system performance and resilience by applying architectural and systems-level optimizations to increase throughput and reduce end-to-end latency.
• Operate DevOps-oriented continuous deployment of services using modern distributed systems tooling and development practices (e.g., Kubernetes, Helm, ArgoCD, Kafka, Redis).
• Develop monitoring dashboards and alerts for the prompt processing service and work with teammates to design and implement a sustainable on-call rotation that provides coverage during the start of observing hours in Chile (typically 2-5pm Pacific Time), with limited off-hours responsibility.
• Define KPIs and metrics for observability and accountability of the pipeline.
• Participate in the collective engineering activities of the team, including performing code reviews, acting as a troubleshooting buddy, participating in design discussions, and writing documentation to effectively capture and communicate architectural and implementation choices.
• Collaborate with members of the Data Management team to identify opportunities to improve tools, workflows, and operational practices.
• Share responsibility with the broader team for the overall success of the Data Management system, beyond the Prompt Processing Framework.

Requirements

• Bachelor's degree and eight years of relevant experience, or a combination of education and relevant experience designing and operating distributed systems at-scale in production environments.
• Experience working in an SRE, DevOps, or data-intensive systems role, with responsibility for building, operating, and improving robust services.
• Experience engaging with modern production infrastructure (e.g., containerized services, messaging systems, and databases; see above for our current tech stack), with the ability to learn and apply new tools quickly in a production environment.
• Familiarity with contemporary distributed service architectures, including service-to-service communication patterns, common failure modes, and system behavior under load and scale.
• Fluency in at least one modern programming language (Python preferred) with experience working across the boundary between software engineering and operations.
• Experience working with large-scale datasets or high-throughput data processing systems, and an understanding of the operational challenges that come with data volume and velocity.
• Ability to communicate clearly with engineers and scientists from diverse backgrounds, including explaining technical concepts, participating in design discussions, and documenting systems and decisions.
• Comfort working with a high degree of autonomy, taking ownership of technical decisions and execution, while being supported by an experienced team with clear priorities and goals.

Qualifications

• Effective Decisions: Uses job knowledge and solid judgment to make quality decisions in a timely manner.
• Self-Development: Pursues a variety of venues and opportunities to continue learning and developing.
• Dependability: Can be counted on to deliver results with a sense of personal responsibility for expected outcomes.
• Initiative: Pursues work and interactions proactively with optimism, positive energy, and motivation to move things forward.
• Adaptability: Flexes as needed when change occurs, maintains an open outlook while adjusting and accommodating changes.
• Communication: Ensures effective information flow to various audiences and creates and delivers clear, appropriate written, spoken, presented messages.
• Relationships: Builds relationships to foster trust, team collaboration, and a positive climate to achieve common goals.

Skills

• Experience working in an SRE, DevOps, or data-intensive systems role, with responsibility for building, operating, and improving robust services.
• Experience engaging with modern production infrastructure (e.g., containerized services, messaging systems, and databases; see above for our current tech stack), with the ability to learn and apply new tools quickly in a production environment.
• Familiarity with contemporary distributed service architectures, including service-to-service communication patterns, common failure modes, and system behavior under load and scale.
• Fluency in at least one modern programming language (Python preferred) with experience working across the boundary between software engineering and operations.
• Experience working with large-scale datasets or high-throughput data processing systems, and an understanding of the operational challenges that come with data volume and velocity.
• Ability to communicate clearly with engineers and scientists from diverse backgrounds, including explaining technical concepts, participating in design discussions, and documenting systems and decisions.
• Comfort working with a high degree of autonomy, taking ownership of technical decisions and execution, while being supported by an experienced team with clear priorities and goals.

Benefits

SLAC National Accelerator Laboratory/Stanford University provides a comprehensive benefits package, including health insurance, retirement savings plans, and paid time off.

Pay

The expected pay range for this position is $137,773 to $ 194,585 per annum.

Schedule

Work standards include Interpersonal Skills, Promote Culture of Safety, and Meets the applicable roles and responsibilities as described in the ESH Manual, Chapter 1General Policy and Responsibilities.