Principal Engineer

Key Responsibilities

• Scalable Agent Platform Systems
• Architect and build the core AI Agent Platform — agent runtime, orchestration, tool/API invocation, state and memory management, and the retrieval/knowledge services agents reason over
• Design for scale and efficiency: high-throughput multi-tenant serving, concurrency and queueing for agent workloads, model/inference routing, caching, and cost-aware execution
• Build the control plane and systems primitives other teams use to define, deploy, version, and operate agents safely
• Drive latency, throughput, and cost optimization across the agentic request path (planning → retrieval → tool calls → generation)
• Agentic AIOps & Autonomous Operations
• Architect agentic operations workflows where autonomous agents observe platform telemetry, reason about anomalies, perform root-cause analysis, and execute remediation — shifting operations from human-driven to agent-driven
• Design multi-agent operational loops (detection, diagnosis, remediation) that collaborate, escalate, and hand off to on-call humans with clear rationale and audit trails
• Build closed-loop self-healing for the platform: auto-detection and repair of failing agents, degraded tools/connectors, stale knowledge, failed ingestion, and retrieval/index drift
• Define guardrails, confidence thresholds, and human-in-the-loop controls that make autonomous remediation safe at multi-tenant scale
• Apply LLMs to operations directly — incident summarization, runbook generation, on-call copilots, and natural-language querying of platform telemetry
• Observability for Agentic Systems
• Instrument the platform end to end: distributed tracing across planning-retrieval-tool-generation loops, metrics, structured logging, and event correlation so multi-agent behavior is explainable and debuggable
• Define golden signals for both system health and agent quality — task success rate, tool-call accuracy, grounding/hallucination rates, latency, cost-per-task, throughput — as first-class telemetry the operating agents act on
• Establish SLOs/SLIs and error budgets for agent workflows, with alerting that feeds the agentic-ops layer
• Reliability, SRE & Distributed Systems
• Engineer the platform as a resilient, event-driven, cloud-native distributed system (Kubernetes, streaming pipelines, microservices) across regions and tenants
• Drive SRE practices — capacity planning, graceful degradation, failover, chaos/resilience testing, blameless incident response — and progressively automate them through agents
• Build for operability first: every component designed to be observed, diagnosed, and acted on autonomously

Qualifications

• Required
• 10+ years building production software, with deep experience designing and operating large-scale distributed systems and platforms
• Proven experience building scalable AI/agent platforms or high-throughput ML serving systems in production — orchestration, multi-tenancy, latency/cost optimization
• Hands-on experience designing agentic or autonomous workflows — multi-agent reasoning, tool/API invocation, planning loops — applied to real production problems
• Strong AIOps and SRE background: observability tooling (OpenTelemetry, Prometheus, Grafana, distributed tracing), SLOs/error budgets, anomaly detection, incident management, and closed-loop automation with human-in-the-loop safeguards
• Hands-on experience applying LLMs to production workflows (reasoning, decision support, summarization)
• Strong proficiency in Python and a systems language (Go/Java); cloud-native architecture (Kubernetes, event-driven microservices, streaming pipelines).
• Working familiarity with RAG and knowledge systems (retrieval, embeddings, knowledge graphs) sufficient to architect over them — depth here is a plus, not the primary bar
• Preferred
• ML-driven anomaly detection, alert correlation, and predictive operations at scale
• SRE leadership operating AI/ML or data-intensive platforms
• Familiarity with agentic frameworks (LangChain, LangGraph) and vector/graph stores
• AI safety and governance grounding for autonomous enterprise systems.
• Contributions to open-source agentic, observability, or AIOps frameworks

Additional Information

• Please note this is a hybrid role with onsite expectations of 3x/week (Tues - Thurs) from our San Mateo, CA headquarters.
• The annual base salary range for this position is $216,000- $298,000. This role is also eligible for a target bonus.
• Compensation is based on a variety of factors, including but not limited to location, experience, job-related skills, and level.
• Freshworks offers multiple options for dental, medical, vision, disability, and life insurance. Equity + ESPP, flexible PTO, flexible spending, commuter benefits, and wellness benefits are also offered. Freshworks also offers adoption and parental leave benefits.