Gen AI / ML Application Testing Engineer (BA + QA)

Key Responsibilities

• Design and execute test cases for Gen AI / ML features and user workflows
• Use GenAI-specific tools to validate prompt consistency, hallucination detection, etc.
• Collaborate with product managers to convert requirements into test cases and test data
• Perform exploratory testing, regression, and prompt-based scenario testing
• Write automation scripts to simulate user behavior and backend interactions
• Track and manage issues using QA platforms and agile tools
• Document test plans, test reports, and AI evaluation metrics

Required Skills

• Hands-on testing experience with Gen AI / ML products
• Experience with LLM testing tools like: Promptfoo (prompt testing & evaluation), LangSmith (LangChain tracing & evals), TruLens (feedback tracking for LLMs), Rebuff (security and behavior testing)
• Solid understanding of LLM behavior, hallucinations, prompt design
• Scripting: Python, Shell, or JavaScript
• REST APIs, JSON, YAML
• Familiarity with PyTest, Postman, Selenium, or similar tools

Nice-to-Have Skills

• Experience testing RAG, chatbot, or LLM agent systems
• Familiarity with LangChain, LlamaIndex, or Haystack
• Business analysis experience in AI projects
• Knowledge of AI/ML model evaluation metrics

Education

Bachelor's or Master's in CS, Data Science, AI/ML, or related field

Preferred skills

• Hands-on testing experience with Gen AI / ML products
• Experience with LLM testing tools like: Promptfoo (prompt testing & evaluation), LangSmith (LangChain tracing & evals), TruLens (feedback tracking for LLMs), Rebuff (security and behavior testing)