Senior Site Reliability Engineer - Observability

About the Role

We are looking for a Senior SRE to join our Platform Engineering team as the operations owner of our observability platforms. You’ll be responsible for the reliability, scalability, and continued evolution of the tools that give our engineering organization visibility into everything they build and run.

Responsibilities

• Operate as a primary escalation point for production support involving the ELK Stack, Grafana, and New Relic
• Own platform health, capacity planning, and performance tuning for on-premises observability infrastructure – including Elasticsearch cluster management, index lifecycle policies, and retention strategies
• Maintain Service Level Objectives (SLOs) for the observability platforms, ensuring the tools engineers depend on are highly available and performant
• Support engineering teams in onboarding to observability platforms – helping teams instrument their applications, build dashboards, and define meaningful alerts
• Manage patching, upgrades, and configuration management across the observability stack
• Collaborate with security to harden platform configurations and manage software vulnerabilities
• Maintain runbooks and escalation procedures for on-call rotations
• Lead or contribute to platform modernization initiatives – e.g., improving ingestion pipelines, scaling platform capacity, standardizing Grafana dashboard and alerting patterns, or evaluating new capabilities within the existing stack
• Design and build tooling/automation to reduce toil and improve the experience for teams using observability platforms
• Develop and maintain infrastructure-as-code (Terraform, Helm, Ansible, etc.) for platform components
• Build and enforce standards around logging metrics and alerting that help engineering teams adopt observability best practices at scale
• Participate in design reviews and contribute to the overall platform roadmap

Qualifications

• Bachelor’s degree in a technical field or equivalent practical experience
• 5+ years of experience in SRE, DevOps, or platform engineering roles
• Deep hands-on experience with the ELK Stack – Elasticsearch cluster operations, Logstash pipeline development, Kibana, and index lifecycle management
• Solid experience with Grafana, including data source integrations, dashboard design, and alerting
• Strong understanding of observability principles
• Experience operating on-premises infrastructure, including capacity planning, server management, and the operational tradeoffs with managed cloud services
• Proficiency in Python for automation and tooling; familiarity with shell scripting
• Strong Linux systems knowledge and comfort working with configuration management tools (e.g., Ansible, Chef, Puppet, etc.)
• Demonstrated ability to drive incidents to resolution and communicate clearly under pressure
• A bias toward automation and a low tolerance for repetitive manual work

Skills

• Experience with Prometheus
• Experience with New Relic administration or APM instrumentation
• Familiarity with log shipping agents and pipeline tools such as Beats, Fluentd, or Fluent Bit
• Experience with distributed tracing tools like OpenTelemetry
• Exposure to cloud-based observability offerings and experience thinking through hybrid strategies
• Prior experience building or governing observability standards across a large engineering organization

Benefits

Dimensional offers a variety of programs to help take care of you, your family, and your career, including comprehensive benefits, educational initiatives, and special celebrations of our history, culture, and growth.