Senior Observability Engineer

Benefits

A career at CoBank can offer you the opportunity to make a personal impact on the people and communities where we do business. In order to be the best, we hire the best!

• Time-Off Packages, 15 days of vacation, 10 paid sick days and 11 paid holidays
• Competitive Compensation & Incentive
• Hybrid work model: flexible arrangements for most positions
• Benefits Packages, including Medical, Dental and Vision coverage, Disability, AD&D, and Life Insurance
• Roadmap for associate training and development with CoBank University
• Tuition reimbursement for higher education
• Outstanding 401k: up to 6% matching and additional 3% non-elective contribution & Student Loan Match
• Community Impact: United Way Angel Day, Volunteer Day and Associate Directed Contribution
• Associate Resource Groups: creating a culture of respect and inclusion
• Recognize a fellow associate through our GEM awards

Job Description

We are seeking a Senior Observability Engineer to help design, implement, and scale enterprise observability capabilities across both on-premises and cloud environments.

In this role, you will play a key part in defining and advancing CoBank's observability strategy, with an initial focus on instrumentation and telemetry for on-prem systems, evolving into cloud-native observability within AWS.

• Designs, implements, and maintains enterprise observability solutions across on-prem and AWS environments.
• Develops and enhances monitoring, alerting, logging, and tracing capabilities using Splunk and related platforms.
• Implements instrumentation standards using OpenTelemetry and other telemetry frameworks.
• Integrates telemetry from diverse systems (infrastructure, applications, and platforms) into centralized observability solutions.
• Collaborates with application teams to enable APM, distributed tracing, and performance monitoring.
• Builds and manages telemetry pipelines and collectors (e.g., OpenTelemetry Collector or similar tools).
• Maintains system health and proactively identify reliability and performance issues.
• Optimizes telemetry ingest, data quality, and alert effectiveness to reduce noise and cost.
• Troubleshoots issues across infrastructure, applications, and observability platforms.
• Pairs with cross-functional teams to establish and promote observability best practices.
• Stays current with emerging trends and technologies in observability, APM, and SRE practices.

Essential Functions

• Designs, implements, and maintains enterprise observability solutions across on-prem and AWS environments.
• Develops and enhances monitoring, alerting, logging, and tracing capabilities using Splunk and related platforms.
• Implements instrumentation standards using OpenTelemetry and other telemetry frameworks.
• Integrates telemetry from diverse systems (infrastructure, applications, and platforms) into centralized observability solutions.
• Collaborates with application teams to enable APM, distributed tracing, and performance monitoring.
• Builds and manages telemetry pipelines and collectors (e.g., OpenTelemetry Collector, Logstash, Fluentd).
• Maintains system health and proactively identify reliability and performance issues.
• Optimizes telemetry ingest, data quality, and alert effectiveness to reduce noise and cost.
• Troubleshoots issues across infrastructure, applications, and observability platforms.
• Pairs with cross-functional teams to establish and promote observability best practices.
• Stays current with emerging trends and technologies in observability, APM, and SRE practices.