Principal Observability Platform Engineer

About the role

This role is a Principal/Staff Observability Platform Engineer at Nscale, responsible for the technical direction of the observability platform. The engineer sets the architectural roadmap, raises the engineering bar across teams, and ensures the platform scales ahead of the business.

Responsibilities

• Own the technical strategy and architecture for observability across metrics, logs, traces, and alerting at scale.
• Drive platform decisions that have multi-year impact: tooling, data models, ingestion patterns, retention, cardinality management.
• Identify systemic gaps before they become incidents; design platforms that make failure visible and fast to diagnose.
• Partner with SRE, infrastructure, and AI/ML teams to embed observability natively into how Nscale builds and operates.
• Define standards and patterns that other engineers adopt, not by mandate, but because they're clearly better.
• Mentor and technically grow the observability team; raise the ceiling on what the team can build and own.
• Lead incident postmortems and use them to drive durable platform improvements.
• Evaluate and introduce tooling that meaningfully improves signal quality, operational efficiency, or scalability, and retire what doesn't.

Requirements

8+ years in SRE, infrastructure engineering, platform engineering, or observability-focused roles. Operated observability infrastructure at serious scale. Strong engineering fundamentals, proficient in Python, Go, or similar; comfortable owning complex systems end to end. Deep hands-on experience with a significant subset of: Prometheus, Thanos, VictoriaMetrics, Grafana, Loki, Tempo, OpenTelemetry, ClickHouse, Elastic. Experience with Kubernetes at scale; familiarity with GPU infrastructure or HPC environments (Slurm) is a strong plus. Influence without authority. Teams want your opinion because it makes their work better.

Qualifications

Experience with high-volume streaming pipelines for observability data (Kafka, Vector, Fluent Bit, etc.). Background in AI/ML infrastructure observability: GPU utilisation, training job visibility, inference latency. Prior experience defining observability strategy at an organization level.

Skills

Strong bias toward simplicity. Architect systems, write the code, review others' work, and explain the tradeoffs clearly, all in the same week. Infrastructure-as-Code is default, not optional (Terraform, Ansible, or equivalent).

Benefits

Nscale offers a competitive benefits package including medical, dental, vision, flexible paid time off, parental leave, and retirement plan participation. Salary range: $150,000 - $215,000 USD.