AI Test Engineer

About the role

Schedule: Monday to Friday. Hours: Standard business. Setting: Hybrid (at least 3 days a week onsite; Remote work is an option for the right candidate).

Responsibilities

• Design, develop, and maintain AI testing and evaluation frameworks for generative AI, predictive AI, LLM, RAG, and AI-enabled developer experience solutions
• Create and execute automated test suites to validate AI model behavior, output quality, accuracy, reliability, consistency, and performance
• Develop automated evaluation pipelines integrated with CI/CD workflows to support continuous quality validation
• Perform functional, regression, integration, performance, scalability, and scenario-based testing of AI-enabled applications and services
• Analyze AI-generated outputs, investigate defects, and document testing findings, metrics, and recommendations for stakeholders

Qualifications

• 3+ years of experience in software testing, quality engineering, test automation, AI/ML testing, or a related discipline
• Bachelor's degree in Computer Science, Information Systems, Software Engineering, Data Science, or a related technical field
• Strong programming and scripting skills in Python
• Experience designing and implementing automated test strategies for complex software applications
• Hands-on experience building automated test frameworks and testing pipelines
• Experience testing APIs, integrations, microservices, or distributed applications
• Experience working with CI/CD environments and modern software delivery practices
• Working knowledge of machine learning, generative AI, large language models, or AI-powered applications
• Experience with data validation, test-result analysis, defect investigation, and root-cause analysis
• Knowledge of modern software testing methodologies, quality engineering practices, and software development lifecycles
• Knowledge of statistical analysis and evaluation techniques used to validate system performance
• Experience with Git or comparable source control systems and collaborative development practices

Skills

• Experience testing generative AI applications, LLM-based solutions, AI agents, or Retrieval-Augmented Generation systems
• Experience with AI evaluation frameworks such as DeepEval, Ragas, LangSmith, or comparable tools
• Experience with LLM observability, monitoring, tracing, and evaluation practices
• Experience with Azure AI, GitHub Copilot, OpenAI, Hugging Face, LangChain, or comparable AI platforms and frameworks
• Familiarity with prompt engineering, benchmark dataset creation, prompt libraries, and AI output validation techniques
• Understanding of Responsible AI principles, AI governance, model risk management, and enterprise AI controls
• Experience with performance, scalability, reliability, and resilience testing
• Experience supporting Developer Experience, developer productivity, software engineering enablement, or internal developer platforms
• Experience working within enterprise, financial services, regulated, or highly controlled technology environments
• Experience developing quality dashboards, automated reporting, or AI evaluation analytics
• Familiarity with model comparison, A/B testing, experimentation, and statistical evaluation methodologies
• Strong curiosity about emerging AI technologies and a passion for building scalable quality engineering capabilities