Principal Agentic AI Operations Engineer
Claritev · United States · 3 wk ago
RemoteRemoteEngineering$165/hrFull-time
We need to output HTML fragment with sections: headings using for sections like About the role, Responsibilities, Requirements, Qualifications, Skills, Benefits, Pay, Schedule. Only include sections if posting has real specific content.
We need to parse the job description.
Key content:
- About the role: description at start: "We are seeking a Principal Agentic AI Operations Engineer to provide hands-on technical leadership..." That's about the role. So we can have
for sections like About the role, Responsibilities, Requirements, Qualifications, Skills, Benefits, Pay, Schedule. Only include sections if posting has real specific content.
We need to parse the job description.
Key content:
- About the role: description at start: "We are seeking a Principal Agentic AI Operations Engineer to provide hands-on technical leadership..." That's about the role. So we can have About the role
with paragraphs.
- Responsibilities: list under "Job Roles And Responsibilities". Many bullet points. We'll create - for each.
- Requirements: includes Education, Experience, Technical Skills, Other Skills. Could split into Requirements (education, experience) and Skills (technical). But we can follow headings: Requirements, Qualifications, Skills, Benefits, Pay, Schedule.
We have Education, Experience, Technical Skills, Other Skills. Could treat Education+Experience as Requirements, Technical Skills as Skills, Other Skills maybe also Skills. Qualifications maybe Preferred Qualifications section.
We have Preferred Qualifications bullet list.
We have Compensation: salary range $165-185K, plus health insurance, 401k, bonus opportunity. So we can have Pay section with salary range, and Benefits section with health insurance, 401k, bonus.
Schedule not mentioned, so omit.
About Us sections: "Why Claritev?" etc. Could be About the role? Already used. Could add About Us as separate section? Not in allowed headings list but we could treat as About the role? The instruction: use
for section headings (e.g. About the role, Responsibilities, Requirements, Qualifications, Skills, Benefits, Pay, Schedule). So we should stick to those. The "About Us" could be considered part of About the role? Might be better to include as About the role? But we already have About the role description. Could add another About the role? Not ideal. Perhaps we treat "About Us" as part of About the role? Or we could omit because not required. The instruction: include a section ONLY when the posting has real, specific content for it. So we can include About the role (we have). Responsibilities (we have). Requirements (we have education/experience). Qualifications (we have preferred qualifications). Skills (we have technical skills). Benefits (we have health insurance, 401k, bonus). Pay (we have salary range). Schedule (none). So we will output those sections.
We must not repeat job title as heading. No Job ID etc.
We must preserve all substantive information; do not invent or summarize away detail. Must not omit details. So we need to include all bullet points.
We need to be careful not to lose any details.
Let's extract content.
About the role paragraph: "We are seeking a Principal Agentic AI Operations Engineer to provide hands-on technical leadership for the operationalization, evaluation, and reliable production performance of the advanced agentic AI systems powering Claritev's next generation of healthcare products. This role is for an experienced engineer who excels at making agentic AI systems trustworthy at scale — measurable, observable, benchmarked, secure, and cost-efficient in production. Where AI engineering teams build agents, you will own how those agents are evaluated, deployed, monitored, governed, and continuously improved. You will define what “good” looks like for agent behavior in high-stakes healthcare workflows and build the evaluation harnesses, benchmarks, and operational tooling that prove it. You will partner closely with Product, Engineering, AI Science, Security, and business leaders to establish the operational backbone of Claritev's agentic AI platform, set engineering and evaluation standards, make key architectural decisions, and mentor engineers and scientists across the organization."
That's the about role.
Responsibilities bullet list:
- Own the end-to-end operational lifecycle of production agentic AI systems: deployment, versioning, rollout strategies, monitoring, incident response, rollback, and continuous improvement.
- Design and build agent evaluation and benchmarking frameworks, including offline evaluation suites, golden datasets, simulation environments, scenario-based and multi-turn benchmarks, regression testing, and A/B and shadow-mode online evaluation.
- Define and track agent quality metrics across task completion, tool-call accuracy, trajectory efficiency, hallucination and grounding rates, latency, cost per task, escalation/human-handoff rates, and safety violations.
- Build observability and distributed tracing capabilities for agentic workflows — spanning LLM calls, tool invocations, retrieval steps, memory operations, and orchestration graphs — using modern APM, log analytics, and OpenTelemetry-based instrumentation.
- Establish LLMOps/AgentOps best practices, including CI/CD for agents, prompt and model versioning, evaluation gates in deployment pipelines, experiment tracking, drift detection, and automated incident troubleshooting.
- Operate and optimize agentic and RAG workloads across managed generative AI services, container orchestration platforms, vector search, and enterprise data platforms in a cloud environment.
- Implement guardrails, policy enforcement, and human-in-the-loop controls for agent actions, with full auditability of agent decisions and tool executions.
- Manage capacity, throughput, and cost governance for LLM and agent workloads, including model [SC1] [SC2] routing, caching, batching, and GPU/compute utilization.
- Ensure secure and responsible operation of AI, including privacy, PHI/PII protection, secrets management, identity and access controls, explainability, auditability, and compliance with HIPAA and applicable data-governance requirements.
- Establish reusable operational frameworks, evaluation harnesses, dashboards, and runbooks that enable teams to ship and operate agents efficiently and consistently.
- Provide technical leadership across complex, cross-functional initiatives; influence architecture, reliability, and evaluation decisions beyond an individual project.
- Mentor engineers and data scientists and promote a culture of measurement-driven development, operational excellence, and pragmatic innovation.
Requirements:
Education: Bachelor's degree in Computer Science, Engineering, Data Science, a quantitative discipline, or a related field required. Master's degree or PhD preferred.
Experience: 10+ years of hands-on experience in software engineering, ML engineering, platform/SRE engineering, or a related technical discipline. 5+ years of experience operating production-grade ML or AI systems (MLOps, LLMOps, or ML platform roles). 3+ years of experience with generative AI, LLMs, RAG, and/or agentic AI systems in production. Demonstrated experience building evaluation and benchmarking capabilities for LLM or agent systems — or contributing to public agent benchmarks (e.g., SWE-bench, GAIA, AgentBench, tau-bench, HELM-style harnesses) with measurable quality and reliability outcomes. Demonstrated experience leading complex technical initiatives from concept through production deployment and measurable business impact.
Technical Skills: Strong software engineering skills, including expert-level Python proficiency and experience designing scalable services, APIs, and distributed systems. Deep experience with agent and LLM evaluation: eval harness design, golden datasets, LLM-as-judge techniques, rubric-based scoring, statistical significance testing, and regression detection. Experience with evaluation and observability tooling such as LangSmith, Langfuse, Arize Phoenix, Ragas, DeepEval, promptfoo, OpenAI Evals, Braintrust, or equivalent — and building custom harnesses where off-the-shelf tools fall short. Experience with agentic AI frameworks and patterns, such as LangGraph, LangChain, AutoGen, CrewAI, or equivalent, including tool use, planning, orchestration, memory, and guardrails. Experience with cloud environments and large-scale data platforms, including managed generative AI/LLM services, compute and GPU infrastructure, and enterprise databases. Experience with vector databases and embedding-based retrieval, and building/operating RAG pipelines. Experience with MLOps/LLMOps practices: CI/CD, model and prompt versioning, canary and shadow deployments, monitoring, experimentation, and incident troubleshooting. Experience with infrastructure-as-code (e.g., Terraform), containerization, and Kubernetes-based deployment. Working knowledge of deep-learning frameworks such as PyTorch or TensorFlow.
Other Skills: Strong problem-solving, critical-thinking, communication, and organizational skills. Ability to communicate agent quality, reliability, and cost trade-offs clearly to technical and non-technical stakeholders. Ability to operate effectively in a fast-moving, cross-functional environment.
Preferred Qualifications: Experience with Oracle Cloud Infrastructure (OCI), including its generative AI, data science, database, and observability capabilities. Experience in healthcare, health technology, insurance, claims, payment integrity, or other regulated industries. Experience operating AI systems that process sensitive data, including PHI or PII, in HIPAA-regulated environments. Experience with red-teaming, adversarial testing, or safety evaluation of LLM/agent systems. Experience with process automation and integration across enterprise workflows and systems. SRE background: SLOs/SLIs, error budgets, and reliability engineering applied to AI systems.
Compensation: The salary range for this position is $165-185K. Specific offers take into account a candidate’s education, experience and skills, as well as the candidate’s work location and internal equity. This position is also eligible for health insurance, 401k and bonus opportunity.
Benefits: health insurance, 401k, bonus opportunity. Also maybe other benefits from "My Total Value" but that's more generic; we could include but it's not explicit in posting? It is under "About Us" but includes medical, dental, vision coverage, life and disability, 401k match, employee stock purchase plan, flexible spending accounts, health savings accounts, paid time off, company holidays, paid parental leave, mental health resources, tuition reimbursement, financial planning resources, professional development opportunities. That is substantive. Should we include? The instruction: Preserve ALL substantive information; do NOT invent or summarize away detail. So we should include those benefits. However the Benefits section should only be included if posting has real specific content for it. It does. So we should include all those benefits.
Thus Benefits section will list those items.
Pay section: salary range $165-185K.
Schedule: none.
We must not include boilerplate like EEO etc. The "Our Commitment to Inclusion" is EEO boilerplate; we should remove.
Also remove "About Us" sections like "Why Claritev?" etc? That is not boilerplate but could be considered fluff. However it's substantive about company mission. The instruction says remove cookie/navigation/legal/EEO/equal-opportunity/scam-warning boilerplate, duplicated lines, and application instructions. "Why Claritev?" is not boilerplate; it's about company. Should we keep? It's not in the allowed headings list but we could incorporate into About the role? Might be okay to add as a separate section like "About the company"? Not allowed headings. Safer to omit because not required headings. However we must preserve ALL substantive information; omitting that would lose info. But maybe we can incorporate into About the role? The About the role already describes role; we could add a paragraph about company mission under About the role? Might be okay to extend About the role with that info. But the instruction: Use for section headings (e.g. About the role, Responsibilities, Requirements, Qualifications, Skills, Benefits, Pay, Schedule). So we can have About the role heading and include both role description and company info under it. That seems acceptable.
Thus About the role paragraph will include the initial role description plus the "Why Claritev?" and "What Guides Us" etc. Also "About The Team My Total Value" etc. We'll include all that.
We must not repeat job title as heading.
Now craft HTML.
We'll output:
About the role
... combined paragraphs ...
Responsibilities
- ...
...
Requirements
Education: ...
Experience: ...
Alternatively could combine into one paragraph but better separate.
Qualifications
Preferred Qualifications items
Skills
for section headings (e.g. About the role, Responsibilities, Requirements, Qualifications, Skills, Benefits, Pay, Schedule). So we should stick to those. The "About Us" could be considered part of About the role? Might be better to include as About the role? But we already have About the role description. Could add another About the role? Not ideal. Perhaps we treat "About Us" as part of About the role? Or we could omit because not required. The instruction: include a section ONLY when the posting has real, specific content for it. So we can include About the role (we have). Responsibilities (we have). Requirements (we have education/experience). Qualifications (we have preferred qualifications). Skills (we have technical skills). Benefits (we have health insurance, 401k, bonus). Pay (we have salary range). Schedule (none). So we will output those sections.
We must not repeat job title as heading. No Job ID etc.
We must preserve all substantive information; do not invent or summarize away detail. Must not omit details. So we need to include all bullet points.
We need to be careful not to lose any details.
Let's extract content.
About the role paragraph: "We are seeking a Principal Agentic AI Operations Engineer to provide hands-on technical leadership for the operationalization, evaluation, and reliable production performance of the advanced agentic AI systems powering Claritev's next generation of healthcare products. This role is for an experienced engineer who excels at making agentic AI systems trustworthy at scale — measurable, observable, benchmarked, secure, and cost-efficient in production. Where AI engineering teams build agents, you will own how those agents are evaluated, deployed, monitored, governed, and continuously improved. You will define what “good” looks like for agent behavior in high-stakes healthcare workflows and build the evaluation harnesses, benchmarks, and operational tooling that prove it. You will partner closely with Product, Engineering, AI Science, Security, and business leaders to establish the operational backbone of Claritev's agentic AI platform, set engineering and evaluation standards, make key architectural decisions, and mentor engineers and scientists across the organization."
That's the about role.
Responsibilities bullet list:
- Own the end-to-end operational lifecycle of production agentic AI systems: deployment, versioning, rollout strategies, monitoring, incident response, rollback, and continuous improvement.
- Design and build agent evaluation and benchmarking frameworks, including offline evaluation suites, golden datasets, simulation environments, scenario-based and multi-turn benchmarks, regression testing, and A/B and shadow-mode online evaluation.
- Define and track agent quality metrics across task completion, tool-call accuracy, trajectory efficiency, hallucination and grounding rates, latency, cost per task, escalation/human-handoff rates, and safety violations.
- Build observability and distributed tracing capabilities for agentic workflows — spanning LLM calls, tool invocations, retrieval steps, memory operations, and orchestration graphs — using modern APM, log analytics, and OpenTelemetry-based instrumentation.
- Establish LLMOps/AgentOps best practices, including CI/CD for agents, prompt and model versioning, evaluation gates in deployment pipelines, experiment tracking, drift detection, and automated incident troubleshooting.
- Operate and optimize agentic and RAG workloads across managed generative AI services, container orchestration platforms, vector search, and enterprise data platforms in a cloud environment.
- Implement guardrails, policy enforcement, and human-in-the-loop controls for agent actions, with full auditability of agent decisions and tool executions.
- Manage capacity, throughput, and cost governance for LLM and agent workloads, including model [SC1] [SC2] routing, caching, batching, and GPU/compute utilization.
- Ensure secure and responsible operation of AI, including privacy, PHI/PII protection, secrets management, identity and access controls, explainability, auditability, and compliance with HIPAA and applicable data-governance requirements.
- Establish reusable operational frameworks, evaluation harnesses, dashboards, and runbooks that enable teams to ship and operate agents efficiently and consistently.
- Provide technical leadership across complex, cross-functional initiatives; influence architecture, reliability, and evaluation decisions beyond an individual project.
- Mentor engineers and data scientists and promote a culture of measurement-driven development, operational excellence, and pragmatic innovation.
Requirements:
Education: Bachelor's degree in Computer Science, Engineering, Data Science, a quantitative discipline, or a related field required. Master's degree or PhD preferred.
Experience: 10+ years of hands-on experience in software engineering, ML engineering, platform/SRE engineering, or a related technical discipline. 5+ years of experience operating production-grade ML or AI systems (MLOps, LLMOps, or ML platform roles). 3+ years of experience with generative AI, LLMs, RAG, and/or agentic AI systems in production. Demonstrated experience building evaluation and benchmarking capabilities for LLM or agent systems — or contributing to public agent benchmarks (e.g., SWE-bench, GAIA, AgentBench, tau-bench, HELM-style harnesses) with measurable quality and reliability outcomes. Demonstrated experience leading complex technical initiatives from concept through production deployment and measurable business impact.
Technical Skills: Strong software engineering skills, including expert-level Python proficiency and experience designing scalable services, APIs, and distributed systems. Deep experience with agent and LLM evaluation: eval harness design, golden datasets, LLM-as-judge techniques, rubric-based scoring, statistical significance testing, and regression detection. Experience with evaluation and observability tooling such as LangSmith, Langfuse, Arize Phoenix, Ragas, DeepEval, promptfoo, OpenAI Evals, Braintrust, or equivalent — and building custom harnesses where off-the-shelf tools fall short. Experience with agentic AI frameworks and patterns, such as LangGraph, LangChain, AutoGen, CrewAI, or equivalent, including tool use, planning, orchestration, memory, and guardrails. Experience with cloud environments and large-scale data platforms, including managed generative AI/LLM services, compute and GPU infrastructure, and enterprise databases. Experience with vector databases and embedding-based retrieval, and building/operating RAG pipelines. Experience with MLOps/LLMOps practices: CI/CD, model and prompt versioning, canary and shadow deployments, monitoring, experimentation, and incident troubleshooting. Experience with infrastructure-as-code (e.g., Terraform), containerization, and Kubernetes-based deployment. Working knowledge of deep-learning frameworks such as PyTorch or TensorFlow.
Other Skills: Strong problem-solving, critical-thinking, communication, and organizational skills. Ability to communicate agent quality, reliability, and cost trade-offs clearly to technical and non-technical stakeholders. Ability to operate effectively in a fast-moving, cross-functional environment.
Preferred Qualifications: Experience with Oracle Cloud Infrastructure (OCI), including its generative AI, data science, database, and observability capabilities. Experience in healthcare, health technology, insurance, claims, payment integrity, or other regulated industries. Experience operating AI systems that process sensitive data, including PHI or PII, in HIPAA-regulated environments. Experience with red-teaming, adversarial testing, or safety evaluation of LLM/agent systems. Experience with process automation and integration across enterprise workflows and systems. SRE background: SLOs/SLIs, error budgets, and reliability engineering applied to AI systems.
Compensation: The salary range for this position is $165-185K. Specific offers take into account a candidate’s education, experience and skills, as well as the candidate’s work location and internal equity. This position is also eligible for health insurance, 401k and bonus opportunity.
Benefits: health insurance, 401k, bonus opportunity. Also maybe other benefits from "My Total Value" but that's more generic; we could include but it's not explicit in posting? It is under "About Us" but includes medical, dental, vision coverage, life and disability, 401k match, employee stock purchase plan, flexible spending accounts, health savings accounts, paid time off, company holidays, paid parental leave, mental health resources, tuition reimbursement, financial planning resources, professional development opportunities. That is substantive. Should we include? The instruction: Preserve ALL substantive information; do NOT invent or summarize away detail. So we should include those benefits. However the Benefits section should only be included if posting has real specific content for it. It does. So we should include all those benefits.
Thus Benefits section will list those items.
Pay section: salary range $165-185K.
Schedule: none.
We must not include boilerplate like EEO etc. The "Our Commitment to Inclusion" is EEO boilerplate; we should remove.
Also remove "About Us" sections like "Why Claritev?" etc? That is not boilerplate but could be considered fluff. However it's substantive about company mission. The instruction says remove cookie/navigation/legal/EEO/equal-opportunity/scam-warning boilerplate, duplicated lines, and application instructions. "Why Claritev?" is not boilerplate; it's about company. Should we keep? It's not in the allowed headings list but we could incorporate into About the role? Might be okay to add as a separate section like "About the company"? Not allowed headings. Safer to omit because not required headings. However we must preserve ALL substantive information; omitting that would lose info. But maybe we can incorporate into About the role? The About the role already describes role; we could add a paragraph about company mission under About the role? Might be okay to extend About the role with that info. But the instruction: Use for section headings (e.g. About the role, Responsibilities, Requirements, Qualifications, Skills, Benefits, Pay, Schedule). So we can have About the role heading and include both role description and company info under it. That seems acceptable.
Thus About the role paragraph will include the initial role description plus the "Why Claritev?" and "What Guides Us" etc. Also "About The Team My Total Value" etc. We'll include all that.
We must not repeat job title as heading.
Now craft HTML.
We'll output:
About the role
About the role
... combined paragraphs ...
Responsibilities
- ... ...
Requirements
Education: ...
Experience: ...
Alternatively could combine into one paragraph but better separate.Qualifications
- Preferred Qualifications items