Quality Analyst II

About the Company

HighLevel is an innovative AI-powered business operating system designed to empower agencies, entrepreneurs, and small to medium-sized businesses (SMBs) with the infrastructure needed to build, automate, and scale their operations. Supporting SMBs across more than 150 countries, HighLevel is committed to driving community-driven growth rooted in delivering tangible customer outcomes. The platform has generated over $7 billion in ecosystem value, showcasing the significant impact of shared infrastructure at scale.

By integrating conversations, automation, and intelligence into a unified system, HighLevel enables businesses to operate more efficiently, reduce complexity, and accelerate their growth trajectories. Behind the scenes, HighLevel handles over 4 billion API hits and 2.5 billion message events daily, supported by a robust architecture comprising 250 terabytes of distributed data, over 250 microservices, and more than 1 million domain names. The company's infrastructure is built for high performance, resilience, and scalability, ensuring reliable service delivery to its global customer base.

With a diverse team of over 2,000 remote-first professionals across more than 10 countries, HighLevel fosters a culture of initiative, clarity, and ownership. The company values innovation, celebrates ideas, and prioritizes its people, creating an environment where ambitious individuals can develop systems that support millions of businesses worldwide. HighLevel's ongoing impact is reflected in enabling over 1.5 billion messages, 200 million leads, and 20 million conversations each month for its extensive customer base.

About the Role

We are seeking a dedicated and detail-oriented Quality Analyst II to join our Customer Success - Programs & AI Innovation team. This evaluator-first specialist will focus on ensuring the quality of AI-powered workflows that support our advisors throughout every customer interaction. The role involves evaluating AI outputs across our internal AI platform, ORA, including reviewing pre-call briefs, in-call guidance, post-call grading, and transcription-driven customer follow-ups.

The successful candidate will classify failure modes, document patterns, and provide structured insights that inform continuous improvements. In addition to evaluation, the role includes light hands-on work within ORA, such as refining prompts, updating knowledge base entries, and tweaking existing skills. This ensures that identified issues are addressed promptly, closing the loop from issue detection to implementation of fixes.

The position offers a clear growth path: starting with evaluation responsibilities, high performers will have opportunities to expand into more creation-focused tasks, collaborating closely with the Lead Analyst and engineering teams to enhance AI workflows and platform capabilities.

Responsibilities

• Evaluate AI output across the full call lifecycle, including pre-call briefs, in-call guidance, post-call summaries, and transcription accuracy
• Classify issues by type and severity, using standardized rubrics, and participate in calibration sessions to ensure scoring consistency
• Identify and document failure modes such as hallucinations, missing context, tone mismatches, and knowledge base gaps
• Perform light creation work inside ORA, including refining prompts, updating knowledge base entries, and tweaking skills to improve output quality
• Test and validate changes, then roll out updates within the platform, ensuring continuous improvement
• Surface systemic trends and patterns to leadership, recommending prioritized areas for improvement and building reporting tools for visibility
• Partner with RevOps Engineering and the Lead Analyst to implement platform changes, bringing structured findings, test cases, and quality criteria
• Stay current with AI advancements, prompt engineering techniques, and evaluation methodologies; run experiments and share insights with the team

Qualifications

• Insatiable curiosity and a self-learning mindset
• Minimum of 2+ years experience in Quality Assurance, Customer Support, AI evaluation, or related roles
• Working understanding of large language models (LLMs), prompt design, and common failure modes
• Knowledge of database concepts and data flow between systems
• Comfort reading code and making minor edits for debugging and iteration purposes
• Strong written communication skills with meticulous attention to detail
• Ability to follow structured guidelines and scoring rubrics consistently
• Comfort working with evolving AI tools and workflows
• Bachelor’s degree in a related field or equivalent practical experience

Preferred Qualifications

• Experience with AI development platforms such as Claude, Cursor, or similar tools
• Familiarity with retrieval-augmented generation (RAG), vector databases, or knowledge-base architecture
• Proficiency in SQL or similar database querying languages
• Experience with APIs, webhooks, or data integrations
• Background in customer success, support, sales enablement, or call center environments
• Experience with HighLevel or comparable SaaS platforms
• Prior experience evaluating AI-generated conversations or automated support outputs

Benefits

• Competitive salary and performance-based incentives
• Remote work flexibility and global team collaboration
• Health, dental, and vision insurance coverage
• Paid time off and holidays
• Opportunities for professional growth and skill development
• Participation in innovative AI and SaaS projects
• Inclusive and diverse work environment that values initiative and ownership