Contract Site Reliability Engineer — AI Accelerator Infrastructure
d-Matrix · Santa Clara, CA · 2 days ago
HybridEngineeringFull-time
About the role
The role is a 6-month contract with potential conversion to full-time. It is a hands-on, high-ownership role where you will build, operate, and automate real infrastructure.
Responsibilities
- Own reliability and availability of assigned infrastructure domains: colo server fleets, on-premises lab clusters, cloud environments (AWS, Azure, GCP), and customer-facing platform services.
- Perform hands-on infrastructure work: server provisioning, OS configuration, network setup, storage management, and hardware troubleshooting from bare metal up.
- Support and operate high-speed interconnect environments — InfiniBand, RoCE, or high-speed Ethernet — in lab and colo settings.
- Conduct capacity planning and hardware lifecycle management for assigned infrastructure domains.
- Own IaC and configuration management (Terraform, Ansible) for your infrastructure domains — all provisioning and changes through code, not manual steps.
- Build automation to eliminate toil: host lifecycle management, fleet health checks, auto-remediation workflows, and self-service tooling for engineering teams.
- Develop networking automations for cluster interconnects, VLAN management, and lab network configurations.
- Contribute to shared IaC modules and automation libraries used across the SRE team.
- Design and maintain monitoring dashboards, alerting, and SLIs (Prometheus/Grafana, DataDog) for your infrastructure domains—ensuring signal quality and actionable alerts.
- Participate in on-call rotation; triage and resolve incidents from bare metal to application layer, distinguishing infrastructure faults from software or hardware product issues.
- Produce high-quality RCA reports for P0/P1 incidents with root cause analysis and tracked action items.
- Detect performance issues, recommend solutions, and implement fixes that permanently improve system reliability.
- Support and operate platform services used by external customers for hardware and software deployment.
- Ensure QoS and uptime commitments for customer-facing environments; escalate reliability risks proactively.
- Document platform configurations, access procedures, and operational runbooks for customer environments.
- Maintain high-quality runbooks, architecture diagrams, and troubleshooting guides—documentation is part of the job, not an afterthought.
- Partner with the DevOps team to ensure infrastructure reliability supports CI/CD pipeline performance and developer experience.
- Serve as a technical resource for engineering teams—sharing operational knowledge and raising infrastructure risks early.
Requirements
- Bachelor's or Master's in Computer Science, Electrical Engineering, or a related field (or equivalent experience); 5+ years in SRE, infrastructure engineering, or systems administration.
- Strong Linux systems knowledge: networking, storage, systemd, package management, kernel parameters, and performance diagnostics.
- Hands-on experience with colocation or on-premises server infrastructure — physical hardware, rack networking, and bare-metal provisioning.
- IaC experience with Terraform and/or Ansible — writing and maintaining production configurations, not just running existing playbooks.
- Kubernetes operational experience: cluster troubleshooting, workload management, storage, and networking.
- Prometheus + Grafana or DataDog: building dashboards, writing alert rules, and understanding signal quality.
- Python and/or Bash scripting: production-quality automation, not just one-off scripts.
- Incident response experience: structured triage, RCA production, and follow-through on action items.
- Comfort operating in fast-moving startups: you own your systems, document what you build, and iterate without waiting for perfect requirements.
Preferred
- Experience operating customer-facing infrastructure or platform services with external reliability expectations.
- HPC job scheduler experience: Slurm, LSF, or equivalent — operations and troubleshooting.
- Knowledge of high-speed interconnect fabrics: InfiniBand, RoCE, or NVLink — configuration and troubleshooting.
- Go programming for SRE tooling — health-check services, exporters, or auto-remediation agents.
- Experience with large-scale infrastructure automation: host lifecycle management, fleet auto-healing, or AIOps-driven operations.
Benefits
We offer a competitive compensation package, including a market-competitive salary, benefits, and a supportive work environment.
Pay
Competitive compensation package, including a market-competitive salary.
Schedule
6-month contract with potential conversion to full-time.