Staff Machine Learning Engineer, AI Agent Platform

Key Responsibilities

• Platform Engineering Architect scalable multi-tenant backend systems for AI agent workflows — including AI agent configuration, evaluation, synthetic data generation, workflow simulation & evaluation, MCP server registry, A2A communication infrastructure, and guardrail enforcement layers using AKS, FastAPI, etc.
• Build an enterprise AI agent skill ecosystem — a platform for authoring, publishing, discovering, versioning, and governing reusable skill packages that encode domain expertise into portable modules.
• Implement an internal skill marketplace with search/discovery, quality scoring, security vetting pipelines, approval workflows, and progressive disclosure loading.
• Implement production-grade AI agent harnesses — the non-model infrastructure (tool dispatch, context management, error recovery/self-healing, session state, sub-agent coordination) that makes AI agents reliable for long-running tasks.
• Design feedforward guides (linters, type checkers, architecture constraints) and feedback sensors (test execution, LLM-as-judge, semantic analysis) mixing computational and inferential controls.
• Develop observability frameworks (OpenTelemetry, distributed tracing) with LLM-specific telemetry: token usage, latency profiling, hallucination detection, AI agent behavior auditing, and skill execution monitoring.
• Design and implement layered guardrail architectures (input validation, prompt injection defense, PII detection, output verification) with parallelized enforcement for minimal latency impact.
• Design and implement skill-level governance: security vetting for hidden payloads, credential theft, and data exfiltration risks; authoring standards; conflict resolution; version management; and deprecation workflows.

Qualifications

• Bachelor's in CS, Engineering, or related field; advanced degree highly desirable.
• 6+ years designing, implementing, and maintaining multi-tenant AI/ML systems in production.
• 6+ years with cloud platforms (Azure, AWS) and backend systems (Kubernetes, Temporal, OpenSearch, PostgreSQL, Redis, Neo4j).
• Deep understanding of Docker, Prometheus, and OpenTelemetry.
• Deep proficiency in Python, Java, or Go.
• Extra credit for effectively leveraging AI coding tools (Cursor, Claude Code, GitHub Copilot).
• Proficiency in AI/ML and agentic frameworks (TensorFlow, PyTorch, LangGraph, CrewAI, AutoGen).
• Demonstrated track record mentoring engineers and leading technical initiatives.
• Excellent communication across diverse seniority levels and professional backgrounds.

Specialized Skills

• Experience with harness engineering concepts and practices such as tool dispatch, error recovery, session state, permissions, sub-agent coordination, planning & reasoning w. feedback loops, etc.
• Experience designing AI agent skill systems — reusable capability packages, skill registries/marketplaces with discovery, versioning, security vetting, and governance controls.
• Hands-on experience with MCP (server development, registries) and A2A (AI agent card discovery, task delegation).
• Experience with LLM observability (LangSmith, Langfuse, Arize Phoenix) and guardrail systems (prompt injection defense, PII scanning, skill-level security auditing).
• Experience with multi-agent orchestration, both open-source (Llama, Qwen, Mistral) and proprietary (GPT, Claude) LLMs, and no-code/low-code AI agent development environments.

Pay

Annual Salary $115,000.00 - $260,000.00

The above annual salary range is a general guideline. Multiple factors are taken into consideration to arrive at the final hourly rate/ annual salary to be offered to the selected candidate. Factors include, but are not limited to, the scope and responsibilities of the role, the selected candidate’s work experience, education and training, the work location as well as market and business considerations.