Scientific Systems Software Developer

About the role

The Application and User Services (AUS) group at SLAC National Accelerator Laboratory is seeking an energetic, forward-thinking software engineer to develop tools and workflows for science projects such as the Vera C. Rubin Observatory US Data Facility.

Responsibilities

• Design, implement, and maintain applications and APIs for S3DF science programs, enabling scientists worldwide to access, process, and analyze large-scale experimental and observational datasets.
• Build and improve data-serving interfaces and processing pipelines that handle petabyte-scale astronomical datasets.
• Containerize applications and deploy them on Kubernetes clusters, following modern cloud-native best practices.
• Contribute to software tools for scientific data management and data processing, with assignments varying according to experiment priorities and lifecycles.
• Integrate applications with Identity and Access Management frameworks (OIDC, SAML2, JWT, LDAP, COManage, etc.) to ensure secure and appropriate data access.
• Write clean, well-tested, well-documented code; contribute to and maintain shared software repositories across S3DF science programs.
• Contribute to the integration and reconciliation of the Rubin data access layer with distributed data management systems (including Rucio), ensuring consistent views of datasets across systems.
• Help develop and implement data lifecycle policies, defining how datasets are created, retained, migrated, and retired across storage tiers in alignment with scientific and operational requirements.
• Work on tooling for metadata integrity and remediation: detecting inconsistencies between catalogs, registries, and physical storage, and building workflows to detect and resolve them.
• Contribute to formalizing levels of data stewardship responsibility by dataset type, clarifying ownership, curation standards, and access controls across the Rubin data portfolio.
• Develop and support workflows for exporting managed data subsets of data for downstream science, collaborators, and community data releases.
• Help rationalize and consolidate the landscape of backing databases, improving consistency and reducing operational complexity.
• Collaborate with Rubin data management teams across institutions to align on standards and tooling.
• Support scientists and users in their day-to-day use of Data Facility services, diagnosing issues, debugging problems, and providing timely, clear resolutions.
• Monitor the health and performance of deployed services and supporting infrastructure; gather metrics and produce reports.
• Identify and resolve bottlenecks in data-intensive workflows, from ingestion and processing through serving and analysis.
• Participate in on-call rotation and incident response for production services.
• Develop and maintain runbooks, operational documentation, and user-facing guides.
• Use the incident management system to track problems to resolution in a timely manner.
• Collaborate with the broader scientific software community (including teams at partner institutions) to align on interfaces, standards, and shared components.
• Evaluate and test emerging technologies and technical developments.
• Provide feedback and concrete recommendations for service improvements to the AUS team and scientific stakeholders.

Qualifications

• Bachelor's degree in physics, computer science, or a related field, and 5 years of relevant experience in software development, systems administration, or scientific/high-performance computing or an equivalent combination of education and experience.
• Proficiency in one or more programming languages; Python strongly preferred, with C/C++ or JavaScript a plus.
• Experience with modern software development practices: version control (Git), CI/CD pipelines, code review, and agile/scrum methodologies.
• Hands-on experience with PostgreSQL (schema design, query optimization, and operational management).
• Experience with Apache Kafka or similar event streaming platforms for high-throughput, real-time data pipelines.
• Practical experience deploying and operating applications on Kubernetes in a production environments at scale.
• Understanding of distributed compute and storage systems, high-performance computing, and networking concepts.
• Experience with system monitoring, benchmarking, and performance analysis.
• Strong organizational and communication skills; ability to work effectively in a collaborative, distributed team environment.
• Ability and genuine willingness to learn, adopt best practices, and grow technical skills on the job.