Software Developer 4

Key Responsibilities

• Lead and execute complex software, automation, and operational engineering projects for Rubin production services at USDF, including Alert Production, DRP, data access, databases, and service infrastructure.
• Support and improve Kubernetes-hosted services deployed through Rubin’s Phalanx/GitOps environment, including Helm configuration, secrets management, environment promotion, release coordination, and operational rollback.
• Maintain and troubleshoot PostgreSQL-backed services, including backup and restore, performance, schema-change coordination, monitoring, and incident response.
• Help operate and debug services that depend on Weka storage, S3 gateways, object storage access, shared filesystems, Butler repositories, batch processing, and high-throughput data movement.
• Develop strategies, methods, tools, and procedures that improve reliability, automation, reproducibility, observability, and operational efficiency across USDF production services, consistent with Software Developer 4 expectations for process improvement and innovation.
• Direct or drive all phases of selected technical projects, including requirements analysis, design, implementation, testing, deployment, documentation, and operational handoff.
• Lead testing, debugging, change control, reporting, and documentation for major service and infrastructure changes.
• Build and maintain monitoring, logging, alerting, dashboards, runbooks, and operational documentation using tools such as Prometheus, Grafana, Loki, and related systems.
• Provide subject-matter expertise for complex technical problems involving Kubernetes, databases, storage, networking, deployment automation, and distributed production services.
• Recognize opportunities for high-impact, long-term reliability improvements that span team or organizational boundaries, and recommend or implement actions to resolve them.
• Mentor technical staff and collaborate with developers, operators, database engineers, storage engineers, and scientists to improve service reliability and operability.

Desired Qualifications And Experience

• Bachelor’s degree and ten years of relevant experience, or a combination of education and relevant experience, consistent with the Software Developer 4 profile.
• Demonstrated experience designing, developing, testing, deploying, operating, and maintaining complex applications or infrastructure services.
• Strong experience with Kubernetes, Helm, GitOps, CI/CD, Linux systems, Python, shell scripting, and production service automation.
• Strong understanding of relational databases, especially PostgreSQL, including operational monitoring, performance troubleshooting, backup/restore, and schema-change coordination.
• Experience with large-scale storage or data systems, ideally including Weka, S3/object-storage interfaces, shared filesystems, high-throughput data transfer, or scientific data repositories.
• Experience with observability systems such as Prometheus, Grafana, Loki, alerting systems, logs, metrics, and operational dashboards.
• Strong understanding of software development life cycle, quality-control practices, change management, incident response, and operational documentation.
• Exceptional written and oral communication skills for technical and non-technical audiences, including the ability to work across distributed teams and organizational boundaries.
• Experience with Rubin Observatory software, Phalanx, Butler, Qserv, Kafka, Prompt Processing, DRP workflows, Weka, S3DF/USDF, or large-scale astronomical data processing would be especially valuable.