Principal Observability Platform Engineer

About the role

This isn't a "maintain and operate" role. It's a "define, build, and lead" role.

Responsibilities

• Own the technical strategy and architecture for observability across metrics, logs, traces, and alerting at scale.
• Drive platform decisions that have multi-year impact: tooling, data models, ingestion patterns, retention, cardinality management.
• Identify systemic gaps before they become incidents; design platforms that make failure visible and fast to diagnose.
• Partner with SRE, infrastructure, and AI/ML teams to embed observability natively into how Nscale builds and operates.
• Define standards and patterns that other engineers adopt, not by mandate, but because they're clearly better.
• Mentor and technically grow the observability team; raise the ceiling on what the team can build and own.
• Lead incident postmortems and use them to drive durable platform improvements.
• Evaluate and introduce tooling that meaningfully improves signal quality, operational efficiency, or scalability, and retire what doesn't.

Requirements

8+ years in SRE, infrastructure engineering, platform engineering, or observability-focused roles.

You've operated observability infrastructure at serious scale. You know what breaks at 10x and you design for it.

You have a strong bias toward simplicity. You've seen over-engineered observability stacks collapse under their own weight and you build accordingly.

Deep hands-on experience with a significant subset of: Prometheus, Thanos, VictoriaMetrics, Grafana, Loki, Tempo, OpenTelemetry, ClickHouse, Elastic.

Strong engineering fundamentals, proficient in Python, Go, or similar; comfortable owning complex systems end to end.

Experience with Kubernetes at scale; familiarity with GPU infrastructure or HPC environments (Slurm) is a strong plus.

You can architect systems, write the code, review others' work, and explain the tradeoffs clearly, all in the same week.

Infrastructure-as-Code is default, not optional (Terraform, Ansible, or equivalent).

You influence without authority. Teams want your opinion because it makes their work better.

Qualifications

Experience with high-volume streaming pipelines for observability data (Kafka, Vector, Fluent Bit, etc.)

Background in AI/ML infrastructure observability: GPU utilisation, training job visibility, inference latency.

Prior experience defining observability strategy at an organisation level.

Skills

Experience with high-volume streaming pipelines for observability data (Kafka, Vector, Fluent Bit, etc.)

Background in AI/ML infrastructure observability: GPU utilisation, training job visibility, inference latency.

Prior experience defining observability strategy at an organisation level.

Benefits

Nscale may offer a competitive benefits package including medical, dental, vision, flexible paid time off, parental leave, and retirement plan participation.

Pay

Salary Range: $150,000 - $215,000 USD

Schedule

Not specified