Senior Kubernetes Engineer - Scientific & Agentic Workflow Platforms

About the role

The Application and User Services (AUS) group within the Scientific Computing Services Division manages the platforms that underpin science at SLAC. We build and operate the systems that let researchers focus on discovery rather than infrastructure.

Key Responsibilities

• Platform Architecture & Engineering
  • Design, build, and operate highly available Kubernetes-based platforms optimized for scientific and agentic workloads
  • Architect scalable solutions for high-throughput data pipelines, real-time streaming, and batch scientific computing
  • Design and implement platform primitives for agentic workflow orchestration – enabling autonomous, multi-step AI-driven pipelines that support experimental science
  • Build and maintain Infrastructure-as-Code using tools such as Helm, Kustomize, and GitOps workflows
  • Evaluate and introduce new technologies and patterns that advance the platform's capabilities for the scientific community
• Agentic & AI Workflow Enablement
  • Lead platform design for agentic scientific workflows – systems where AI agents autonomously orchestrate data acquisition, analysis, and experimental feedback loops
  • Collaborate with researchers and data scientists to define platform requirements for running large language model-driven and reinforcement learning agents at scale
  • Implement infrastructure patterns for agent orchestration frameworks (e.g., multi-agent pipelines, tool-use APIs, memory and state management) within Kubernetes
  • Ensure the platform supports the latency, throughput, and accelerator requirements of agentic workloads
  • Build guardrails, observability, and governance tooling suited to autonomous scientific agents operating on sensitive experimental data
• Scientific Project Support
  • Partner with scientists and researchers – at SLAC and across DOE labs and universities – to design and implement solutions for major scientific programs, including:
  • Vera C. Rubin Observatory / LSST: Petabyte-scale nightly sky surveys requiring real-time alert pipelines and long-running batch analysis for dark matter and dark energy research
  • LCLS (Linac Coherent Light Source): Real-time analysis infrastructure for the world's brightest X-ray laser, capturing femtosecond-scale dynamics of matter
  • Cryo-EM: High-throughput 3D reconstruction pipelines for structural biology at near-atomic resolution
  • Accelerator Operations: Monitoring, control, and data acquisition infrastructure for particle accelerators
  • American Science Cloud: National-scale scientific data infrastructure to democratize access to computing resources across National Laboratories
  • Emerging Initiatives: Co-design of infrastructure for next-generation scientific computing programs not yet fully defined
• User Collaboration & Support
  • Act as a senior technical partner to the scientific user community, translating complex experimental requirements into scalable platform solutions
  • Lead requirements-gathering sessions and technical consultations with research groups
  • Provide hands-on guidance and training to help users adopt platform capabilities effectively
  • Gather user feedback and advocate for user needs in platform planning and roadmap prioritization
  • Develop documentation, runbooks, and reference architectures to empower scientific teams
• Reliability & Operations
  • Define and maintain Service Level Objectives (SLOs)/Service Level Agreements (SLAs) for platform services supporting scientific workflows
  • Implement comprehensive monitoring, logging, and observability (Prometheus, Grafana, OpenTelemetry, Loki)
  • Design and implement CI/CD pipelines for scientific software, data processing workflows, and platform components
  • Lead incident response and post-mortem processes; participate in on-call rotation
  • Drive capacity planning and performance tuning for compute-intensive and data-intensive workloads
  • Optimize GPU and accelerator resource scheduling and utilization across the platform
• Collaboration & Community
  • Build and maintain strong relationships with scientific user communities across SLAC, Stanford, and the broader research ecosystem
  • Collaborate with counterparts at DOE National Laboratories (LBNL, Fermilab, Argonne, etc.) to share architectures and best practices
  • Lead technical workshops, training sessions, and working groups to advance cloud-native and agentic workflow adoption
  • Contribute to and represent SLAC in open-source communities relevant to scientific computing and Kubernetes
  • Mentor junior team members and support a culture of technical excellence within AUS

Required Qualifications

• Minimum 8 years of software or infrastructure engineering experience with demonstrated expertise in distributed systems
• Minimum 4 years of hands-on experience designing, deploying, and operating Kubernetes in production environments
• Strong proficiency in Python and/or Go; comfort reading and contributing to multi-language codebases
• Deep experience with container orchestration, networking, storage, and security in Kubernetes environments
• Hands-on experience with Infrastructure-as-Code and GitOps tooling
• Demonstrated ability to design and operate high-throughput or real-time data processing pipelines at scale
• Solid understanding of observability practices and tooling (Prometheus, Grafana, distributed tracing)
• Familiarity with AI/ML infrastructure: GPU scheduling, model serving, workflow orchestration for ML pipelines
• Awareness of agentic workflow frameworks and patterns (e.g., multi-agent orchestration, LLM tool use, agent state management)
• Understanding of the infrastructure requirements that distinguish agentic workloads from traditional batch or streaming pipelines

Soft Skills

• Exceptional communication skills with the ability to engage credibly with both scientists and engineers
• Demonstrated ability to build trust and translate ambiguous research requirements into concrete technical designs
• Strong customer-service orientation; patience and empathy when supporting users with diverse technical backgrounds
• Self-directed and able to manage multiple high-priority projects in a fast-moving research environment
• Collaborative and collegial team member who actively lifts others