Senior Site Reliability Engineer

About the role

The Vera C. Rubin Observatory is a new astronomy facility in Chile designed to create a 10-year time-lapse map of the southern sky through the Legacy Survey of Space and Time (LSST). The Data Management (DM) team is seeking a Software Developer 3 to design, operate, and sustain the systems that process Rubin's data in near real time.

Responsibilities

• Ensure, through both architecture and practice, the reliable operation of the near-real-time data processing pipeline and timely delivery of alerts to downstream brokers.
• Design and develop software that reduces operational risk and improves system resilience, scalability, and usability, including addressing failure modes, error handling, and contention in shared resources.
• Improve system performance and resilience by applying architectural and systems-level optimizations to increase throughput and reduce end-to-end latency.
• Operate DevOps-oriented continuous deployment of services using modern distributed systems tooling and development practices (e.g., Kubernetes, Helm, ArgoCD, Kafka, Redis).
• Develop monitoring dashboards and alerts for the prompt processing service and work with teammates to design and implement a sustainable on-call rotation that provides coverage during the start of observing hours in Chile (typically 2-5pm Pacific Time), with limited off-hours responsibility.
• Define KPIs and metrics for observability and accountability of the pipeline.
• Participate in the collective engineering activities of the team, including performing code reviews, acting as a troubleshooting buddy, participating in design discussions, and writing documentation to effectively capture and communicate architectural and implementation choices.
• Collaborate with members of the Data Management team to identify opportunities to improve tools, workflows, and operational practices.
• Share responsibility with the broader team for the overall success of the Data Management system, beyond the Prompt Processing Framework.

Requirements

• Bachelor's degree and eight years of relevant experience, or a combination of education and relevant experience designing and operating distributed systems at-scale in production environments.
• Experience working in an SRE, DevOps, or data-intensive systems role, with responsibility for building, operating, and improving robust services.
• Experience engaging with modern production infrastructure (e.g., containerized services, messaging systems, and databases; see above for our current tech stack), with the ability to learn and apply new tools quickly in a production environment.
• Familiarity with contemporary distributed service architectures, including service-to-service communication patterns, common failure modes, and system behavior under load and scale.
• Fluency in at least one modern programming language (Python preferred) with experience working across the boundary between software engineering and operations.
• Experience working with large-scale datasets or high-throughput data processing systems, and an understanding of the operational challenges that come with data volume and velocity.
• Ability to communicate clearly with engineers and scientists from diverse backgrounds, including explaining technical concepts, participating in design discussions, and documenting systems and decisions.
• Comfort working with a high degree of autonomy, taking ownership of technical decisions and execution, while being supported by an experienced team with clear priorities and goals.

Qualifications

• Effective Decisions: Uses job knowledge and solid judgment to make quality decisions in a timely manner.
• Self-Development: Pursues a variety of venues and opportunities to continue learning and developing.
• Dependability: Can be counted on to deliver results with a sense of personal responsibility for expected outcomes.
• Initiative: Pursues work and interactions proactively with optimism, positive energy, and motivation to move things forward.
• Adaptability: Flexes as needed when change occurs, maintains an open outlook while adjusting and accommodating changes.
• Communication: Ensures effective information flow to various audiences and creates and delivers clear, appropriate written, spoken, presented messages.
• Relationships: Builds relationships to foster trust, team collaboration, and a positive climate to achieve common goals.

Skills

• Modern distributed service architectures
• Service-to-service communication patterns
• Common failure modes
• System behavior under load and scale
• Large-scale datasets or high-throughput data processing systems
• Technical communication
• Collaboration with diverse teams
• High degree of autonomy

Benefits

SLAC National Accelerator Laboratory/Stanford University provides a comprehensive benefits package, including health insurance, retirement plans, and paid time off.

Pay

The expected pay range for this position is $137,773 to $ 194,585 per annum.