Staff Observability Platform Engineer

About the role

This role plays a critical part in building and evolving Nscale's observability platform, enabling deep visibility into GPU clusters, AI workloads, and the infrastructure that powers them.

Responsibilities

• Design, build, and evolve observability platforms across metrics, logs, traces, alerting, and telemetry pipelines.
• Lead the implementation of scalable observability solutions that support Nscale's growing GPU and AI infrastructure.
• Partner with SRE, infrastructure, platform, and AI/ML teams to ensure observability is embedded throughout the software and infrastructure lifecycle.
• Drive improvements in monitoring coverage, alert quality, service health visibility, and incident response effectiveness.
• Develop standards, frameworks, and reusable patterns that simplify observability adoption across engineering teams.
• Identify reliability risks and operational blind spots, helping teams proactively address them before they impact customers.
• Contribute to architectural decisions around telemetry collection, storage, retention, cardinality management, and performance optimization.
• Lead technical initiatives and projects that improve platform scalability, reliability, and operational efficiency.
• Mentor engineers and provide technical guidance through design reviews, code reviews, and knowledge sharing.
• Participate in incident investigations and postmortems, translating operational learnings into durable platform improvements.
• Evaluate new observability technologies and practices, balancing innovation with operational simplicity and long-term maintainability.

Requirements

• 6+ years of experience in SRE, platform engineering, infrastructure engineering, observability engineering, or related disciplines.
• Strong experience building and operating observability platforms in cloud-native, distributed environments.
• Deep hands-on experience with several of the following technologies: Prometheus, Thanos, VictoriaMetrics, Grafana, Loki, Tempo, OpenTelemetry, ClickHouse, Elastic, or similar platforms.
• Strong software engineering skills with proficiency in Go, Python, or equivalent languages.
• Experience operating and troubleshooting Kubernetes-based platforms at scale.
• Strong understanding of monitoring, logging, tracing, telemetry pipelines, and modern observability practices.
• Experience designing systems with scalability, reliability, performance, and operational simplicity in mind.
• Proficiency with Infrastructure-as-Code tools such as Terraform, Ansible, or equivalent.
• Able to lead technical initiatives and influence engineering decisions across multiple teams.
• Excellent communication skills with the ability to explain technical tradeoffs and align stakeholders around pragmatic solutions.

Preferred

• Experience operating observability systems in GPU, AI/ML, HPC, or large-scale compute environments.
• Familiarity with Slurm, Kubernetes GPU scheduling, or AI infrastructure platforms.
• Experience with high-volume telemetry pipelines and streaming technologies such as Kafka, Vector, or Fluent Bit.
• Knowledge of observability challenges related to model training, inference workloads, GPU utilization, and distributed AI systems.
• Experience mentoring engineers and helping grow technical capability across teams.