Site Reliability Engineer II

About the role

As a Site Reliability Engineer on the High Performance Data Facility (HPDF) team, you will help build and operate the facility's first systems on its path to operations. You will create the monitoring, alerting, and automation that the full facility will eventually run on, participate in incident response, and help define and report on the service level objectives that measure how well the facility serves its users. You will work as part of a small site reliability engineering team, with day-to-day direction from the team's lead, and alongside staff at both Jefferson Lab and Berkeley Lab. The users you support are research physicists and computational scientists, and helping them succeed is a core measure of this role.

Responsibilities

• Build and operate monitoring, logging, and alerting for HPDF systems using modern observability tooling (for example Prometheus, Grafana, OpenTelemetry, ELK), within guidelines set with the lead site reliability engineer and senior staff.
• Develop automation and tooling in Python, Go, or shell that eliminates manual operations and reduces operational risk, following standard software development practices including version control, code review, and testing.
• Respond to incidents and author the runbooks, postmortems, and operational documentation that convert each incident into a lasting improvement to facility reliability.
• Implement and report on Service Level Objectives (SLOs) and Service Level Indicators (SLIs) defined with the architecture team and scientific stakeholders.
• Support expert scientific users by helping research physicists and computational scientists understand system behavior and by translating their needs into reliability requirements.
• Collaborate with colleagues at both Jefferson Lab and Berkeley Lab, sharing tooling, reviews, and operational practice across the HPDF partnership.
• Conduct testing and performance analysis to validate reliability and resilience decisions and to identify bottlenecks.
• Contribute to evaluations of vendor and open source technologies against reliability, performance, and security requirements.
• Participate in an on-call rotation as the facility moves toward operations.

Requirements

• 3 or more years related experience in Site Reliability Engineering, DevOps, systems engineering, or software engineering with operational responsibility.
• Bachelor's Degree in Computer Science or a related field.

Preferred Qualifications

• 3 or more years experience supporting scientific computing, HPC, or research environments; experience operating systems in support of users outside the engineer's own team.
• Experience with containers and Kubernetes.
• Practical experience applying AI-assisted or autonomous automation to operational tasks, and routine use of AI tools in day-to-day engineering work.
• Experience with configuration management and infrastructure as code tools (for example Ansible, Terraform, Puppet).
• Master's Degree in Computer Science or a related field.

Knowledge, Skills, and Abilities

• Solid Linux systems skills and command-line fluency, with the ability to troubleshoot across the application, operating system, and network layers.
• Scripting and automation ability in Python, Go, or shell, including familiarity with standard software development practices.
• Working knowledge of monitoring and observability tools (for example Prometheus, Grafana, ELK, OpenTelemetry) and strong motivation to deepen that expertise in a new facility environment.
• Clear written and verbal communication, and the ability to work productively with a community of scientific users and with colleagues across both laboratories.
• A self-starter's approach to learning new technologies and to identifying and solving problems within an assigned scope.
• Familiarity with public cloud environments (AWS, Azure, GCP).
• Networking fundamentals (IPv4/IPv6, DNS, firewalls, access control lists) and security-conscious operational habits.
• Exposure to HPC or scientific computing environments.

Pay

The good-faith pay range for this role is $91,800 – $145,050 per year. Actual compensation may vary and may be above the posted range based on factors such as a candidate's skills, experience, education, certifications, and work location.

Benefits

• Medical, Dental, and Vision Care Plans
• Flexible Spending Accounts
• Paid Time-off and Leave Programs (Paid Parental, vacation, holidays, and sick leave)
• 401(k) Plan – 9% Lab Contribution; 100% vested
• Flexible Work Arrangements (Remote & Alternate Work Schedules available)
• Tuition Assistance, Training and Professional Development Programs

About Jefferson Lab

Join a community with a common purpose of solving the most challenging scientific and engineering problems of our time. The Jefferson Lab campus is located in southeastern Virginia amidst a vibrant and growing technology community. A career at Jefferson Lab is more than a job. You will be part of “big science” and work alongside top scientists and engineers from around the world unlocking the secrets of our visible universe. Managed by SURATech, LLC, Thomas Jefferson National Accelerator Facility is entering an exciting period of mission growth and is seeking new team members ready to apply their skills and passion to have an impact.