Senior Quality Engineer, Data & AI Platform

Description Summary

The Senior Quality Engineer - Data & AI sits at the intersection of data engineering, machine learning product delivery, and quality assurance. This role is responsible for validating AI-powered and data-driven features across M3’s hospitality accounting platform, with primary ownership of quality for the Data & AI team’s active products.

Essential Duties

• Design and execute test strategies for AI/ML-enabled features; define acceptance thresholds (precision, recall, F1) in partnership with Engineering and Product.
• Validate AI-generated outputs - mappings, anomaly flags, LLM summaries - against domain-grounded acceptance criteria; build regression suites to detect concept drift, label drift, and feature degradation.
• Evaluate LLM outputs for accuracy, hallucination, and format compliance; build prompt regression test suites to catch behavior changes when model versions or system prompts are updated.
• Implement automated validation checkpoints in Databricks pipelines; apply statistical testing methods to pipeline output validation.
• Define and instrument production monitoring checks for model performance: alert thresholds, confidence degradation, and data drift.
• Perform functional, regression, integration, and exploratory testing across releases, enhancements, and defect fixes using manual and automated approaches.
• Design, develop, maintain, and execute automated test scripts using frameworks such as Katalon and Playwright; support CI/CD quality gates.
• Develop and execute test cases for ETL processes, data warehouse workflows, and API validation.
• Review application logs and monitoring tools to identify and track defects; coordinate builds, deployments, and software migrations for planned releases and hotfixes.
• Maintain and utilize testing tools including Azure DevOps, SQL Server, Postman/Swagger/SoapUI, and automation frameworks.
• Participate in Agile ceremonies; collaborate with Product, Engineering, Data, and external contractor teams to ensure comprehensive coverage and aligned acceptance criteria.
• Communicate quality risks early and in writing; develop and maintain a QE runbook for AI/ML products.
• Adhere to secure testing practices and compliance requirements associated with the assigned Technical Security Level.

Education/Training/Experience

• Minimum of 5 years of experience in software testing or quality engineering, with at least 2 years focused on data-intensive or AI/ML-enabled products.
• Bachelor’s Degree in Computer Science, IT, MIS, or equivalent combination of education and experience.
• 3+ years of hands-on experience with Microsoft SQL Server for data validation and testing; experience using Azure DevOps for test management and release coordination.
• 2+ years of API testing experience (Postman, Swagger, or SoapUI); experience writing and executing test cases for ETL processes and data warehouse environments.
• Hands-on experience with Databricks, Delta Lake, or equivalent Lakehouse platforms and Python for data validation strongly preferred.
• Experience validating ML model outputs and familiarity with LLM evaluation, prompt regression testing, or generative AI quality workflows preferred.
• Familiarity with dbt, Great Expectations, MLflow, or equivalent pipeline-level assertion and experiment tracking tools preferred.
• Working experience with automation frameworks such as Katalon or Playwright; familiarity with USALI and hospitality GL structure a plus.
• Understanding of Agile/Scrum methodologies; strong written and verbal communication skills.
• CSQA or ISTQB certification strongly preferred; certifications in AI/ML quality or data engineering are welcomed.