AI Test Engineer II

Role and Responsibilities

• Design and execute test strategies for AI/ML models, including functional, regression, and scenario-based testing
• Validate accuracy, consistency, and reliability of AI-generated outputs, including summaries, classifications, and recommendations
• Develop test cases for edge cases, model drift, and unexpected behavior scenarios
• Evaluate model performance against defined success criteria and business requirements
• Agent & Workflow Testing
• Test agent-driven workflows, including multi-step AI processes, automation pipelines, and orchestration logic
• Validate interactions between AI agents, APIs, and downstream systems
• Identify failure points in end-to-end AI workflows and recommend improvements
• Data Quality & Integrity
• Validate training, validation, and test datasets for completeness, accuracy, and bias risks
• Perform data validation testing across pipelines to ensure integrity of inputs and outputs
• Partner with Data Engineering to detect anomalies, inconsistencies, and gaps
• Automation & Tooling
• Build and maintain automated test frameworks for AI-enabled systems
• Develop scripts and tools to support repeatable testing of AI outputs at scale
• Integrate testing into CI/CD pipelines to enable continuous validation
• AI Risk & Guardrails
• Help define and validate guardrails for AI systems, including accuracy thresholds, explainability, and fallback logic
• Identify risks related to hallucinations, bias, and model drift
• Partner with Engineering and AI teams to improve reliability and safety
• Collaboration & Continuous Improvement
• Work cross-functionally with Engineering, QA, AI, Product, and Operations teams
• Participate in sprint planning, backlog refinement, and release validation
• Contribute to testing standards, best practices, and documentation
• Support root cause analysis for AI-related defects and production issues

Qualifications

• Bachelor’s degree in Computer Science, Data Science, Engineering, or a related field, or equivalent experience
• Experience in QA, testing, or a related engineering role
• Experience testing APIs, data pipelines, or distributed systems
• Strong understanding of software testing principles and test automation
• Experience working in an Agile development environment
• Familiarity with AI/ML concepts, including model evaluation, training versus inference, and prompt behavior
• Experience with test automation tools
• Strong SQL and data validation skills
• Experience working with cloud-based systems, preferably AWS or Databricks
• Ability to analyze system behavior and identify root causes

Preferred Skills

• Experience testing AI/ML models or data-driven systems
• Experience validating AI-generated outputs or prompt-based systems
• Familiarity with LLMs, agent workflows, or AI-assisted coding tools
• Healthcare or regulated data environment experience, including HIPAA and compliance
• Experience working with CI/CD and DevOps pipelines