Site Reliability Engineer, Observability

The Work

This is an engineering-first role with a coaching dimension—not the other way around. You will spend the majority of your time doing hands-on observability and reliability engineering work: building instrumentation, designing alert configurations, authoring Terraform, and troubleshooting production systems. Alongside that, you will coach and consult with stream-aligned product teams, helping them build operational maturity over time.

Observability Engineering

• Design and implement monitoring, alerting, and dashboards in New Relic (APM, Infrastructure, Logs, Synthetics) across Azure and AWS; write NRQL queries for troubleshooting, analysis, and reporting.
• Define and implement SLOs/SLIs and error budgets; coach teams on using them to balance feature velocity with reliability and communicate system health to stakeholders.
• Lead alert noise reduction and signal quality engineering—tune thresholds, eliminate false positives, and ensure every alert is actionable.
• Optimize observability costs through log ingestion management, pipeline rules, and New Relic configuration governance.
• Partner with engineering teams to improve observability maturity: structured logging, metrics instrumentation (RED/USE methods), distributed tracing, and effective dashboard patterns.

Infrastructure & IaC

• Develop and maintain Terraform infrastructure as code for provisioning and managing monitoring resources, alert configurations, and observability infrastructure—this is a primary engineering responsibility, not an occasional task.
• Establish and enforce IaC governance standards for observability infrastructure across teams, providing a repeatable, auditable model for how monitoring resources are managed.
• Author and troubleshoot Azure DevOps pipelines; support teams with deployment visibility, change tracking, and release hygiene as it relates to production reliability.

Incident Management

• Administer and configure Incident.IO: alert routing, notification workflows, Slack and OpsGenie integration, and runbook management—operationalizing what exists today and expanding from there.
• Build out incident management foundations that are largely yours to establish: PIR/postmortem processes, on-call rotation design, escalation policies, incident severity classification, and response playbooks.
• Track and report on MTTR, MTTD, and incident frequency; identify trends and drive continuous improvement in partnership with engineering teams.
• Respond to and debrief on production incidents—providing real-time troubleshooting support and facilitating structured post-incident reviews.

Cross-Functional Enablement

• Enable stream-aligned engineering teams to adopt improved observability and incident management practices through workshops, consultation, and hands-on guidance.
• Collaborate with the Subsystems Platform Team to translate common needs into self-service observability and incident management capabilities.
• Build lasting team competency through documentation, training materials, and knowledge-sharing sessions that outlast any individual engagement.

Qualifications

• Core SRE Experience: 7+ years in Site Reliability Engineering, DevOps, or Platform Engineering with a strong focus on observability and production operations.
• Proven ability to deliver hands-on engineering work while coaching and mentoring teams—comfortable switching between builder and consultant modes.
• Experience working in Agile/Scrum environments and collaborating effectively with cross-functional teams.

Observability & Incident Management Expertise

• Expert-level hands-on experience with New Relic (APM, Infrastructure, Logs, Synthetics, Alerts) and strong NRQL proficiency for troubleshooting and analysis.
• Deep understanding of structured logging, metrics collection (RED/USE methods), distributed tracing, and designing effective dashboards and alerts.
• Expertise defining and implementing SLOs/SLIs and error budgets for reliability management.
• Hands-on experience with incident management platforms (Incident.IO, PagerDuty, OpsGenie, or similar).
• Experience designing incident response workflows, on-call rotations, escalation policies, and facilitating post-incident reviews that drive actionable improvements.
• Demonstrated ability to troubleshoot complex production issues using observability data across distributed systems.

Infrastructure & Tools

• Strong Terraform experience: developing and maintaining IaC for cloud infrastructure and monitoring resources; familiarity with IaC governance patterns.
• Proficiency with PowerShell scripting (required given the 80% Windows environment).
• Strong experience with Azure cloud (App Services, Virtual Machines, Azure SQL, networking, monitoring) and working knowledge of AWS.
• Experience with Azure DevOps for CI/CD pipeline authoring and troubleshooting.
• Experience with Octopus Deploy for deployment management and release orchestration.
• Comfort working across both Windows and Linux server environments.
• Familiarity with Slack for operational workflows, alert routing, and incident communication.