Staff Machine Learning Engineer, AI Agent Platform

Key Responsibilities

• Platform Engineering Architect scalable multi-tenant backend systems for AI agent workflows — including AI agent configuration, evaluation, synthetic data generation, workflow simulation & evaluation, MCP server registry, A2A communication infrastructure, and guardrail enforcement layers using AKS, FastAPI, etc.
• Build an enterprise AI agent skill ecosystem — a platform for authoring, publishing, discovering, versioning, and governing reusable skill packages that encode domain expertise into portable modules.
• Implement an internal skill marketplace with search/discovery, quality scoring, security vetting pipelines, approval workflows, and progressive disclosure loading.
• Implement production-grade AI agent harnesses — the non-model infrastructure (tool dispatch, context management, error recovery/self-healing, session state, sub-agent coordination) that makes AI agents reliable for long-running tasks.
• Design feedforward guides (linters, type checkers, architecture constraints) and feedback sensors (test execution, LLM-as-judge, semantic analysis) mixing computational and inferential controls.
• Develop observability frameworks (OpenTelemetry, distributed tracing) with LLM-specific telemetry: token usage, latency profiling, hallucination detection, AI agent behavior auditing, and skill execution monitoring.
• Design and implement layered guardrail architectures (input validation, prompt injection defense, PII detection, output verification) with parallelized enforcement for minimal latency impact.
• Design and implement skill-level governance: security vetting for hidden payloads, credential theft, and data exfiltration risks; authoring standards; conflict resolution; version management; and deprecation workflows.

Qualifications

• Bachelor's in CS, Engineering, or related field; advanced degree highly desirable.
• 6+ years designing, implementing, and maintaining multi-tenant AI/ML systems in production.
• 6+ years with cloud platforms (Azure, AWS) and backend systems (Kubernetes, Temporal, OpenSearch, PostgreSQL, Redis, Neo4j).
• Deep understanding of Docker, Prometheus, and OpenTelemetry.
• Proficiency in Python, Java, or Go.
• Extra credit for effectively leveraging AI coding tools (Cursor, Claude Code, GitHub Copilot).
• Deep proficiency in AI/ML and agentic frameworks (TensorFlow, PyTorch, LangGraph, CrewAI, AutoGen).

Leadership Skills

• Demonstrated track record mentoring engineers and leading technical initiatives.
• Excellent communication across diverse seniority levels and professional backgrounds.

Specialized Skills

• Experience with harness engineering concepts and practices such as tool dispatch, error recovery, session state, permissions, sub-agent coordination, planning & reasoning w. feedback loops, etc.
• Experience designing AI agent skill systems — reusable capability packages, skill registries/marketplaces with discovery, versioning, security vetting, and governance controls.
• Hands-on experience with MCP (server development, registries) and A2A (AI agent card discovery, task delegation).
• Experience with LLM observability (LangSmith, Langfuse, Arize Phoenix) and guardrail systems (prompt injection defense, PII scanning, skill-level security auditing).
• Experience with multi-agent orchestration, both open-source (Llama, Qwen, Mistral) and proprietary (GPT, Claude) LLMs, and no-code/low-code AI agent development environments.

Pay

Annual Salary: $115,000.00 - $260,000.00