Quality Analyst II

About The Company HighLevel is an innovative AI-powered business operating system designed to empower agencies, entrepreneurs, and small to medium-sized businesses (SMBs) with the infrastructure needed to build, automate, and scale their operations. Supporting SMBs across more than 150 countries, HighLevel is committed to driving community-driven growth rooted in delivering tangible customer outcomes. The platform has generated over $7 billion in ecosystem value, showcasing the significant impact of shared infrastructure at scale. By integrating conversations, automation, and intelligence into a unified system, HighLevel enables businesses to operate more efficiently, reduce complexity, and accelerate their growth trajectories. Behind the scenes, HighLevel handles over 4 billion API hits and 2.5 billion message events daily, supported by a robust architecture comprising 250 terabytes of distributed data, over 250 microservices, and more than 1 million domain names. The company's infrastructure is built for high performance, resilience, and scalability, ensuring reliable service delivery to its global customer base. With a diverse team of over 2,000 remote-first professionals across more than 10 countries, HighLevel fosters a culture of initiative, clarity, and ownership. The company values innovation, celebrates ideas, and prioritizes its people, creating an environment where ambitious individuals can develop systems that support millions of businesses worldwide. HighLevel's ongoing impact is reflected in enabling over 1.5 billion messages, 200 million leads, and 20 million conversations each month for its extensive customer base. About The Role We are seeking a dedicated and detail-oriented Quality Analyst II to join our Customer Success - Programs & AI Innovation team. This evaluator-first specialist will focus on ensuring the quality of AI-powered workflows that support our advisors throughout every customer interaction. The role involves evaluating AI outputs across our internal AI platform, ORA, including reviewing pre-call briefs, in-call guidance, post-call grading, and transcription-driven customer follow-ups. The successful candidate will classify failure modes, document patterns, and provide structured insights that inform continuous improvements. In addition to evaluation, the role includes light hands-on work within ORA, such as refining prompts, updating knowledge base entries, and tweaking existing skills. This ensures that identified issues are addressed promptly, closing the loop from issue detection to implementation of fixes. The position offers a clear growth path: starting with evaluation responsibilities, high performers will have opportunities to expand into more creation-focused tasks, collaborating closely with the Lead Analyst and engineering teams to enhance AI workflows and platform capabilities. Qualifications Insatiable curiosity and a self-learning mindsetMinimum of 2+ years experience in Quality Assurance, Customer Support, AI evaluation, or related rolesWorking understanding of large language models (LLMs), prompt design, and common failure modesKnowledge of database concepts and data flow between systemsComfort reading code and making minor edits for debugging and iteration purposesStrong written communication skills with meticulous attention to detailAbility to follow structured guidelines and scoring rubrics consistentlyComfort working with evolving AI tools and workflowsBachelor’s degree in a related field or equivalent practical experience Preferred Qualifications Experience with AI development platforms such as Claude, Cursor, or similar toolsFamiliarity with retrieval-augmented generation (RAG), vector databases, or knowledge-base architectureProficiency in SQL or similar database querying languagesExperience with APIs, webhooks, or data integrationsBackground in customer success, support, sales enablement, or call center environmentsExperience with HighLevel or comparable SaaS platformsPrior experience evaluating AI-generated conversations or automated support outputs Responsibilities Evaluate AI output across the full call lifecycle, including pre-call briefs, in-call guidance, post-call summaries, and transcription accuracyClassify issues by type and severity, using standardized rubrics, and participate in calibration sessions to ensure scoring consistencyIdentify and document failure modes such as hallucinations, missing context, tone mismatches, and knowledge base gapsPerform light creation work inside ORA, including refining prompts, updating knowledge base entries, and tweaking skills to improve output qualityTest and validate changes, then roll out updates within the platform, ensuring continuous improvementSurface systemic trends and patterns to leadership, recommending prioritized areas for improvement and building reporting tools for visibilityPartner with RevOps Engineering and the Lead Analyst to implement platform changes, bringing structured findings, test cases, and quality criteriaStay current with AI advancements, prompt engineering techniques, and evaluation methodologies; run experiments and share insights with the team Benefits Competitive salary and performance-based incentivesRemote work flexibility and global team collaborationHealth, dental, and vision insurance coveragePaid time off and holidaysOpportunities for professional growth and skill developmentParticipation in innovative AI and SaaS projectsInclusive and diverse work environment that values initiative and ownership Equal Opportunity We are proud to be an Equal Employment Opportunity employer. We are committed to equal employment opportunity regardless of race, color, ancestry, religion, sex, national origin, sexual orientation, age, citizenship, marital status, disability, gender identity, veteran status, or any other basis protected by federal, state, or local law.