Quality Analyst II

Key Responsibilities

• Evaluate AI Quality Across the Advisor Stack
• Review pre-call briefs for accuracy, completeness and signal quality - customer context, adoption signals, retention flags
• Review in-call guidance - next-best-action prompts, suggestions and recommendation flows - for relevance and accuracy in live conversation
• Score post-call outputs: quality grading, AI-generated summaries, CRM updates, sentiment signals
• Audit transcription-driven follow-ups - customer updates, recap notes, follow-up tasks - for accuracy, tone and customer impact
• Classify issues by type and severity - hallucination, missing context, wrong action, tone mismatch, KB gap - using standardized rubrics
• Participate in calibration sessions to keep scoring consistent across reviewers
• Iterate Inside ORA - Light Creation Work
• Close the loop. When you spot an issue you can fix without engineering involvement, you fix it inside ORA.
• Refine and iterate on prompts within existing skills and sub-agents
• Update knowledge base entries to close information gaps surfaced in reviews
• Tweak existing skills and orchestrations to improve output quality
• Test changes, validate improvements and roll out updates inside ORA
• Document what you changed and why - every iteration is a data point
• Surface Trends & Drive Continuous Improvement
• Connect individual evaluations to systemic insights
• Tag conversations by product area, severity and issue type with consistent classification
• Maintain a clear, trusted log of issue types, trends and conversation behaviors
• Surface recurring patterns and systemic gaps to leadership and the Lead Analyst

Qualifications

Commensurate with experience, but typically requires:

• Bachelor’s degree in Computer Science, Artificial Intelligence, Data Science, or related field
• Minimum of 2 years of experience in AI evaluation, testing, or similar role
• Strong understanding of natural language processing, machine learning, and AI ethics
• Experience with AI platforms and tools, including familiarity with ORA
• Excellent communication and interpersonal skills, with the ability to collaborate effectively with cross-functional teams
• Ability to analyze complex data and identify patterns
• Proficiency in Microsoft Office Suite, particularly Excel for data analysis