AI-Factory Observability Principal

Observability Architecture & Strategy

Define and own the end-to-end observability architecture covering metrics, logs, traces, and events across facilities and IT domains.

Establish standards for instrumentation, telemetry pipelines, data retention, cardinality management, and a unified data model that lets power, thermal, network, and compute signals be correlated in one place.

Design for scale: hundreds of thousands of time series per site, high-frequency power and thermal sampling, and GPU-cluster-level granularity.

Facilities & OT Integration (BMS / EPMS)

AI Cluster & Kubernetes Observability

Network Observability

AIOps

Cross-functional Leadership

Required Qualifications

• 8+ years in infrastructure, SRE, observability, or data center engineering, with 3+ years in an architect or principal-level role.
• Demonstrated experience designing and operating observability platforms at scale (metrics, logs, traces).
• Expertise in Datadog, Dynatrace, Grafana, Prometheus and Grafana.
• Hands-on experience integrating BMS and EPMS data, and a working understanding of data center mechanical and electrical systems (cooling topologies, power distribution, redundancy, capacity).
• Strong systems monitoring background — Linux/server fleets, hardware health, baseboard management (IPMI/Redfish).
• Strong network monitoring background, including high-performance / low-latency fabrics relevant to AI workloads. Expertise in SNMP, WMI.
• Production experience with Kubernetes and observability of containerized workloads.
• Experience operating or monitoring GPU / AI-accelerator clusters and understanding of distributed training/inference behavior.
• Practical experience applying AI/ML models to operational data (anomaly detection, forecasting, or AIOps), and comfort using LLMs to derive insights and automate analysis.
• Proficiency in at least one language for data/automation work (Python preferred), and infrastructure-as-code practices.

Preferred Qualifications

• Experience with tooling such as OpenTelemetry, VictoriaMetrics/Thanos, Loki, Tempo, Elastic, Splunk.
• Familiarity with OT/industrial protocols: BACnet, Modbus, OPC-UA, DNP3, IEC 61850, MQTT.
• Familiarity with GPU/accelerator telemetry (NVIDIA DCGM and exporters) and InfiniBand/RDMA monitoring (e.g., UFM).
• Experience with network telemetry: gNMI/OpenConfig streaming, SNMP, NetFlow/sFlow.
• Experience with time-series data at high cardinality, stream processing, and data lake/warehouse patterns for telemetry.
• Background in MLOps, model deployment, or building data/feature pipelines for operational ML.
• Exposure to power and cooling optimization, PUE improvement, or sustainability/energy-efficiency initiatives.
• Relevant certifications (e.g., data center facilities, Kubernetes/CKA, cloud or networking) are a plus.

Pay Range

$173,600.00 - $180,000.00 USD/yr.

We also offer comprehensive benefits options which vary depending on role, location, and employment type. The Talent Acquisition Partner can share more details about compensation or benefits for the role during the interview process.