Director of Observability
Kestra Financial · Austin, TX · Yesterday
OTHRFull-time
What you’ll Do
- Observability Strategy & Platform Hands-On Build-Out
- Define and execute the observability strategy for Kestra Holdings, aligned with business objectives, regulatory requirements, and the enterprise technology roadmap.
- Personally lead the initial build-out of the observability platform across metrics, logs, traces, profiles, and alerting — including tool evaluation, POCs, architecture, deployment, and configuration (e.g., Azure Monitor/Log Analytics, Datadog, Grafana, OpenTelemetry, Elastic/Splunk).
- Work with and enforce existing instrumentation standards (OpenTelemetry, structured logging, distributed tracing) across infrastructure and application teams.
- Build the first generation of dashboards, SLO scorecards, and a single pane of glass for Tier-1 service health — rolling up sleeves alongside the Sr. Architect and engineer.
- Operate the firm's end-to-end incident management lifecycle — detection, response, escalation, communication, and blameless post-incident review.
- Stand up on-call schedules, escalation policies, and runbook-driven triage for Sev1–Sev4 incidents via PagerDuty/xMatters or equivalent.
- Serve as primary incident commander for major incidents during the initial build phase, transitioning command responsibilities to senior team members as the practice matures.
- Integrate the incident lifecycle with Jira / Jira Service Management (JSM) for ticketing, change correlation, and remediation tracking; partner with Cybersecurity so incidents run once, not separately by Infra and Cyber.
- Facilitate post-incident reviews (PIRs), track remediation items in Jira, and report trends to leadership.
- Drive adoption of SRE principles across the firm: SLI/SLO definition, error budget policy and enforcement, toil identification and automation, and operational readiness reviews.
- Establish release of reliability gates and embed reliability into the service lifecycle from design through production.
- Partner with Cloud & Platform Engineering, Cybersecurity, and application teams to ensure all services are fully instrumented, measurable, and integrated into the firm's SLO and incident frameworks.
- Ensure observability and incident management practices align with NIST CSF 2.0 maturity targets and support the firm's cybersecurity roadmap.
- Partner with the Cybersecurity team to integrate observability data with Jira/JSM, CMDB, and SIEM for enriched context during incidents.
- Support regulatory and audit requirements appropriate for a SEC-regulated financial services firm (e.g., logging retention, evidentiary integrity, access controls on telemetry data).
- Directly lead and mentor a small, high-leverage team of two to start: a Senior Observability Architect (India-based) and a US-based Observability/Reliability Engineer.
- Operate as a player-coach — splitting time between strategic leadership, hands-on engineering, and direct mentorship of the senior architect.
- Build a multi-year workforce plan and talent pipeline to scale the team as the platform, service catalog, and 24/7 coverage needs grow.
- Foster a culture of blameless learning, operational excellence, and engineering-led reliability across US and India hours of coverage.
- Serve as the primary technical liaison for observability and incident management vendors (e.g., Datadog, PagerDuty/xMatters, Grafana Labs, Elastic/Splunk, Atlassian).
- Represent Observability & Reliability in the Architecture Review Board, IT Change Management Board, and incident command forums.
- Provide regular reporting to SVP and executive leadership on reliability KPIs (MTTD, MTTA, MTTR, SLO compliance, alert signal-to-noise), incident trends, and strategic initiatives.
What You Bring
- 10+ years in observability, SRE, platform engineering, or infrastructure operations roles, with 3+ years in a people leadership capacity (Director or Sr. Manager level).
- Demonstrated experience building an observability or SRE practice from scratch — tool selection, instrumentation rollout, first SLOs, and standing up incident command.
- Strong hands-on technical depths must be willing and able to write code/IaC, configure platforms, build dashboards, and run incidents personally, not just delegate.
- Deep expertise across the observability stack: metrics (Prometheus, Datadog, Azure Monitor), log aggregation (Elastic/OpenSearch, Log Analytics, Splunk), distributed tracing (OpenTelemetry, Jaeger, Datadog APM), and profiling.
- Proven experience defining SLIs/SLOs, error budgets, and toil reduction programs.
- Hands-on experience with incident management platforms (PagerDuty, xMatters) and Jira / Jira Service Management integration for ticketing and workflow.
- Experience leading distributed teams across US and India time zones.
- Experience operating in a regulated industry (financial services, healthcare, or similar) with familiarity with compliance frameworks (NIST CSF, SOC 2, SEC, FINRA).
- Excellent communication skills — able to present reliability posture, incident retrospectives, and risk to executive and board-level audiences.
- Experience with IaC (Terraform, Bicep, ARM), CI/CD pipelines, and embedding observability-as-code into modern DevOps practices.