AI Solutions QA Analyst

Job Description

McDermott Will & Schulte LLP is building one of the most ambitious AI programs in the legal industry — and quality is not an afterthought. As we deploy AI-powered solutions across legal practices and operational teams, we need someone who can own the full quality lifecycle: from unit tests and edge cases through system testing, performance, and CI/CD pipeline integration.

The AI Solutions QA Analyst partners directly with our Analysts and engineering team to ensure every solution we ship is tested, reliable, and production-ready — leveraging AI itself to do it faster and more thoroughly than traditional QA ever could.

Why Mws

Shape quality standards for an entirely new category of legal AI solutions — from the ground up
Work with frontier AI tools to build and run tests faster than traditional QA methods allow
High-visibility role reporting to Director of QA with direct impact on every solution the team ships
Remote-first, technically rigorous team that takes production quality seriously

What You'll Do

• Own the full QA lifecycle for AI-powered solutions — from unit and integration testing through system, performance, and scalability testing
• Partner with Analysts to generate and evaluate test cases using AI-assisted tools, with particular focus on boundary conditions, edge cases, and AI output validation
• Work with the engineering team to define and integrate automated test suites into the Azure DevOps CI/CD pipeline for each solution
• Define system-level test strategies in collaboration with the Director of QA, covering functional, regression, and non-functional testing requirements
• Coordinate with the dedicated security team to translate security testing guidance into solution-specific security test cases
• Evaluate AI solution reliability across varied inputs — ensuring consistent, predictable behavior under real-world conditions
• Maintain test documentation, defect tracking, and quality metrics that give leadership clear visibility into solution readiness
• Champion a quality culture across the delivery team — making testing a natural part of the development workflow, not a gate at the end

What You Bring

• 4+ years of formal QA experience, including writing and executing test scripts across the full testing lifecycle
• Strong experience with unit, integration, system, and regression testing — both manual and automated
• Hands-on experience integrating test suites into CI/CD pipelines; Azure DevOps experience strongly preferred
• Demonstrated ability to design boundary and edge case tests for complex, data-driven applications
• AI fluency — you use AI tools actively and can direct leading AI platforms to generate, evaluate, and refine test cases
• Understanding of performance and scalability testing concepts; experience with load testing tools a plus
• Solid grasp of defect tracking, test documentation, and quality metrics reporting
• Strong analytical thinking — you approach untested AI behavior the way an engineer approaches an unknown system
• BS in Computer Science, Information Systems, Engineering, or equivalent experience

Preferred Experience

• Testing AI-powered applications — prompt-driven systems, agentic workflows, or LLM-integrated tools
• Familiarity with AI-assisted test generation workflows and code-generating platforms
• Exposure to security testing principles in collaboration with a dedicated security team
• Experience in legal, financial services, or other regulated industry environments

What You Get

• Competitive total rewards package with the opportunity to earn performance-based bonuses
• Discretionary bonus for full-time positions
• A comprehensive benefits package

Physical Demands And Work Environment

• Sit, use hands, reach with hands and arms, stoop, talk and hear
• Occasionally lift up to twenty (20) pounds
• Typical indoor office environment

Disclaimer

• The above statements are intended to describe the general nature and level of the work being performed by people within this classification. They are not intended to be an exhaustive list of all responsibilities, duties and skills required of employees assigned to this job.