AI Implementation Quality Analyst

About the role

The AI Quality & Evaluation Analyst is responsible for assessing the quality, correctness, completeness, and safety of AI-generated responses across defined use cases. This role combines hands-on human review with structured, rubric-based evaluation incorporating automation to ensure AI systems meet documented standards before customer implementations go live.

Responsibilities

• Partner directly with clients during implementation to understand use cases, success criteria, and risk tolerance
• Translate client requirements into evaluation frameworks, prompt strategies, and test coverage
• Act as the quality liaison between client, product, and engineering to ensure alignment pre- and post-launch
• Review and score AI responses using standard rubrics
• Validate against sources/ground truth; flag hallucinations, omissions, and other risks
• Document findings and calibrate with peers to keep scoring consistent
• Build and maintain prompt banks and golden sets (expected results)
• Expand coverage for edge cases, high-risk scenarios, and real user language
• Track regression/drift and feed dashboards and quality reports
• Triage internal and client feedback; synthesize themes across deployments
• Identify systemic risks and escalate high-impact findings with clear, client-relevant context
• Partner with product/engineering to validate and verify fixes, ensuring alignment with client expectations

Qualifications

• A strong instinct for what “good” looks like and the discipline to define and measure it
• An ability to dig into ambiguous outputs, spot subtle issues, and back your judgment with clear reasoning and evidence
• A comfort operating at the intersection of clients and technical teams—translating real-world needs into structured evaluation criteria and turning findings into actionable improvements
• A desire to build systems that scale (rubrics, test sets, coverage) and take pride in making complex behavior consistent, reliable, and trustworthy

Skills

• Strong analytical judgment and attention to detail
• Excellent written communication skills, including the ability to explain reasoning clearly
• Experience reviewing, auditing, or evaluating structured outputs such as content, decisions, or recommendations
• Familiarity with large language models and common failure modes such as hallucinations, overgeneralization, or unsafe responses
• Ability to work in spreadsheets, evaluation tools, or annotation platforms
• Experience with AI evaluation, data annotation, QA, trust and safety, or policy review
• Exposure to human-in-the-loop workflows, or benchmarking processes
• Domain expertise in a regulated or high-risk field such as government, education, healthcare, or legal services
• Experience contributing to test suites, evaluation dashboards, or quality reporting
• Experience in client-facing roles such as implementation, consulting, or solution delivery in a SaaS or regulated environment
• Comfort working cross-functionally with product and engineering teams

Benefits

Pay Range: USD $80,000.00 - USD $105,700.00 /Yr.

About Us: Don’t have all the skills/experience mentioned above? At Granicus, we are trying to build diverse, inclusive teams. We do not have degree requirements for most of our roles. If you don’t meet every requirement above but are excited to learn more, we encourage you to apply. We might just be able to find another role that could be a perfect fit!

Security And Privacy Requirements: Responsible for Granicus information security by appropriately preserving the Confidentiality, Integrity, and Availability (CIA) of Granicus information assets in accordance with the company's information security program. Responsible for ensuring the data privacy of our employees and customers, their data, as well as taking all required privacy training in a timely manner, in accordance with company policies.

The Team: We are a remote-first company with a globally distributed workforce across the United States, Canada, United Kingdom, India, Armenia, Australia, and New Zealand. The Culture: At Granicus, we are building a transparent, inclusive, and safe space for everyone who wants to be a part of our journey. A few culture highlights include – Employee Resource Groups to encourage diverse voices, Coffee with Mark sessions – Our employees get to interact with our CEO on very important and sometimes difficult issues ranging from mental health to work-life balance and current affairs, Microsoft Teams communities focused on wellness, art, furbabies, family, parenting, and more. We bring in special guests from time to time to discuss issues that impact our employee population.

The Impact: We are proud to serve dynamic organizations around the globe that use our digital solutions to make the world a better place — quite literally. We have so many powerful success stories that illustrate how our solutions are impacting the world. See more of our impact here.

The Benefits: At Granicus, we offer a comprehensive and flexible benefits package designed to support your well-being, growth, and work-life balance—starting from day one. Here’s what you can expect as a U.S.-based team member:

• Flexibility & Balance: Flexible Time Off – Take the time you need to rest, recharge, and live your life. Company-Wide Wellbeing Days – Paid days off to unplug and focus on your mental health. Work From Home Reimbursement – Support a productive home office environment.
• Health & Wellness: Multiple Health Plan Options – Including a 100% employer-paid plan. Employer HSA Contributions – When enrolled in a High-Deductible Health Plan. Fitness Reimbursement Program – Stay active, your way. On-Demand Mental Health Support – Access to Headspace and other wellness tools.
• Family & Future: Paid Parental Leave – For both birthing and non-birthing parents. Traditional & Roth 401(k) – With a generous company match. Life & AD&D Insurance – 100% employer-paid coverage for peace of mind.
• Growth & Recognition: Online Learning Platforms – Fuel your professional development. Competitive Salary & Bonuses – Your contributions are valued and rewarded.

Equal Opportunity Employer: Granicus is committed to providing equal employment opportunities. All qualified applicants and employees will be considered for employment and advancement without regard to race, color, religion, creed, national origin, ancestry, sex, gender, gender identity, gender expression, physical or mental disability, age, genetic information, sexual or affectional orientation, marital status, status with regard to public assistance, familial status, military or veteran status or any other status protected by applicable law.