Quality Lead, Agentic AI Workflow Evaluation

We need to output HTML fragment with headings

for sections: About the role, Responsibilities, Requirements, Qualifications, Skills, Benefits, Pay, Schedule. Only include sections that have real specific content. Must not repeat job title. Must not include Job ID etc. Must preserve all substantive info, no boilerplate. Remove cookie/navigation/legal/EEO etc. Also remove application instructions and scam warning? The instruction says remove cookie/navigation/legal/EEO/equal-opportunity/scam-warning boilerplate, duplicated lines, and application instructions. So we should remove the scam warning paragraphs. Also remove the line about Innodata description? That's part of About the role? The posting includes a paragraph about Innodata mission etc. That could be considered "About the role" or maybe "About the company". Since we have only allowed headings, we can put that under About the role. The "Scope Of The Role" etc is responsibilities. We'll need to parse. We need sections: About the role (maybe company overview), Responsibilities (what you'll own), Requirements (maybe Bachelor's degree etc), Qualifications (maybe direct experience etc), Skills (maybe hands-on familiarity etc), Benefits (none mentioned), Pay (salary range given), Schedule (none). So we will include About the role, Responsibilities, Requirements, Qualifications, Skills, Pay. No Benefits or Schedule. We must not create sections with filler like "commensurate with experience". Pay section we have actual range $75-85 p/hour. Good. We need to use

• for any list of duties/requirements/benefits. So for responsibilities we can list bullet points. For requirements maybe bullet list. For qualifications maybe bullet list. For skills maybe bullet list. We must preserve all substantive information, not invent or summarize away detail. So we need to include all details from the posting, but we can condense into bullet points but must not lose info. Must not omit any substantive info. Let's extract. Posting text: "Innodata (Nasdaq: INOD) is a global data engineering company. We believe that data and Artificial Intelligence (AI) are inextricably linked. Our mission is to enable the responsible advancement of artificial intelligence by providing the data, evaluation frameworks, and human expertise required to build AI systems that can be trusted at scale. We provide a range of transferable solutions, platforms, and services for Generative AI / AI builders and adopters. In every relationship, we honor our 36+ year legacy delivering the highest quality data and outstanding outcomes for our customers." That's about company. "Scope Of The Role We are standing up a dedicated onsite team to evaluate complex, real-world agentic AI workflows for a frontier AI customer. Reviewers work through ambiguous, multi-step scenarios inside isolated test environments, assessing whether AI agents complete tasks safely, respect user intent and consent, and hold up under close scrutiny. The Quality Lead is the person accountable for whether that output is any good. This is a senior individual contributor role. You will not manage the reviewers — that sits with the Engagement Manager — but you set the standard they are held to. You own the audit sample, run calibration, keep the rubric usable as real cases stress it, and train reviewers into the work. You are also the deputy: when the Engagement Manager is out, the engagement runs on you. The quality approach here is not fully defined. We expect you to build it in partnership with the customer's quality leads, or at minimum to take what they have, run it honestly, and come back with specific recommendations for where it falls short." Then "What You’ll Own" list: - Own the quality system for the engagement: audit design, sampling strategy, scoring standards, and how quality gets measured and reported - Build that system with the customer's quality leads where none exists, and where one does, operate it and recommend concrete improvements based on what the data shows - Re-score a sample of reviewer output as a second pass; identify error patterns rather than isolated mistakes - Run calibration sessions: surface disagreement, work it to resolution, and document the reasoning so the outcome holds for future cases - Maintain rubric health — flag criteria that are ambiguous, overlapping, or silent on cases the team keeps hitting, and drive revisions through the customer - Train and onboard new reviewers, including nesting plans, ramp criteria, and the judgment call on when someone is production-ready - Give the Engagement Manager the evidence behind performance conversations: who is drifting, on what, and whether coaching is working - Report quality trends to the Engagement Manager and, alongside them, to the customer - Deputize for the Engagement Manager on delivery operations during absences - Maintain information security, privacy, and facility access practices required by the customer's onsite environment Then "You’ll Thrive In This Role If You Have": - Bachelor's degree or equivalent practical experience - 4+ years in quality assurance, quality management, or senior review work within annotation, evaluation, trust and safety, or a similarly judgment-intensive domain - Direct experience owning a quality function: you designed the audit, not just executed someone else’s - Significant experience with AI/ML evaluation work: annotation, red-teaming, RLHF, model or agent evaluation, or trust and safety review - Hands-on familiarity with agentic systems: tool use, multi-step task execution, sandboxed environments, and common failure modes - Demonstrated ability to run calibration with peers — including holding a position under disagreement and changing it when the argument is better - Strong written communication; able to document a scoring standard clearly enough that a reviewer can apply it and an auditor can check it - Comfortable in spreadsheets and in a dashboarding tool, with enough Python or SQL to pull and slice your own data (you will not be asked to build interfaces) - Experience training or onboarding reviewers into rubric-based work Then "The expected hourly salary range for this position is $75-85 p/hour, based on experience, skills, and qualifications." Then scam warning etc to remove. Thus we need sections: About the role: include the company description paragraph. Responsibilities: the "What You’ll Own" bullet list. Requirements: maybe the Bachelor's degree or equivalent practical experience? Actually that could be under Requirements. But we also have Qualifications separate. The instruction says use headings for Requirements, Qualifications, Skills. We need to decide what goes where. Typically Requirements are basic qualifications (education, years experience). Qualifications are more specific experiences. Skills are abilities. We'll map: Requirements: - Bachelor's degree or equivalent practical experience - 4+ years in quality assurance, quality management, or senior review work within annotation, evaluation, trust and safety, or a similarly judgment-intensive domain Qualifications: - Direct experience owning a quality function: you designed the audit, not just executed someone else’s - Significant experience with AI/ML evaluation work: annotation, red-teaming, RLHF, model or agent evaluation, or trust and safety review - Hands-on familiarity with agentic systems: tool use, multi-step task execution, sandboxed environments, and common failure modes - Demonstrated ability to run calibration with peers — including holding a position under disagreement and changing it when the argument is better - Strong written communication; able to document a scoring standard clearly enough that a reviewer can apply it and an auditor can check it - Comfortable in spreadsheets and in a dashboarding tool, with enough Python or SQL to pull and slice your own data (you will not be asked to build interfaces) - Experience training or onboarding reviewers into rubric-based work Skills: maybe overlapping with qualifications; but we could treat the above as Qualifications and maybe Skills could be something like "Ability to run calibration sessions", "Strong written communication", etc. However we already used those. To avoid duplication, we could put the more ability-focused items under Skills, and the experience-focused under Qualifications. Let's decide: Requirements: education + years experience. Qualifications: direct experience owning quality function, significant experience with AI/ML evaluation, hands-on familiarity with agentic systems, experience training/onboarding reviewers. Skills: demonstrated ability to run calibration with peers, strong written communication, comfortable in spreadsheets/dashboarding tool with Python/SQL. But we need to ensure we don't lose any info. Let's list all items from "You’ll Thrive In This Role If You Have": 1. Bachelor's degree or equivalent practical experience 2. 4+ years in quality assurance, quality management, or senior review work within annotation, evaluation, trust and safety, or a similarly judgment-intensive domain 3. Direct experience owning a quality function: you designed the audit, not just executed someone else’s 4. Significant experience with AI/ML evaluation work: annotation, red-teaming, RLHF, model or agent evaluation, or trust and safety review 5. Hands-on familiarity with agentic systems: tool use, multi-step task execution, sandboxed environments, and common failure modes 6. Demonstrated ability to run calibration with peers — including holding a position under disagreement and changing it when the argument is better 7. Strong written communication; able to document a scoring standard clearly enough that a reviewer can apply it and an auditor can check it 8. Comfortable in spreadsheets and in a dashboarding tool, with enough Python or SQL to pull and slice your own data (you will not be asked to build interfaces) 9. Experience training or onboarding reviewers into rubric-based work We need to split into Requirements, Qualifications, Skills. Let's assign: Requirements: items 1 and 2 (education + years experience). Qualifications: items 3,4,5,9 (direct experience owning quality function, significant AI/ML eval experience, hands-on familiarity with agentic systems, experience training/onboarding reviewers). Skills: items 6,7,8 (ability to run calibration, strong written communication, comfortable with spreadsheets/dashboarding tool Python/SQL). That seems logical. Now Responsibilities: list all bullet points from "What You’ll Own". Ensure we keep all details. About the role: include the company description paragraph. Pay: include the salary range. We must not include Benefits or Schedule. Now produce HTML fragment:

  About the role

  Innodata (Nasdaq: INOD) is a global data engineering company. We believe that data and Artificial Intelligence (AI) are inextricably linked. Our mission is to enable the responsible advancement of artificial intelligence by providing the data, evaluation frameworks, and human expertise required to build AI systems that can be trusted at scale. We provide a range of transferable solutions, platforms, and services for Generative AI / AI builders and adopters. In every relationship, we honor our 36+ year legacy delivering the highest quality data and outstanding outcomes for our customers.

  Responsibilities

  • Own the quality system for the engagement: audit design, sampling strategy, scoring standards, and how quality gets measured and reported
  • Build that system with the customer's quality leads where none exists, and where one does, operate it and recommend concrete improvements based on what the data shows
  • Re-score a sample of reviewer output as a second pass; identify error patterns rather than isolated mistakes
  • Run calibration sessions: surface disagreement, work it to resolution, and document the reasoning so the outcome holds for future cases
  • Maintain rubric health — flag criteria that are ambiguous, overlapping, or silent on cases the team keeps hitting, and drive revisions through the customer
  • Train and onboard new reviewers, including nesting plans, ramp criteria, and the judgment call on when someone is production-ready
  • Give the Engagement Manager the evidence behind performance conversations: who is drifting, on what, and whether coaching is working
  • Report quality trends to the Engagement Manager and, alongside them, to the customer
  • Deputize for the Engagement Manager on delivery operations during absences
  • Maintain information security, privacy, and facility access practices required by the customer's onsite environment

  Requirements

  • Bachelor's degree or equivalent practical experience
  • 4+ years in quality assurance, quality management