AI Quality & Automation Engineer (LLM Focused)

About the role

Join the AI Engineering team to test and optimize Generative AI, conversational AI, and RAG applications. Focus on AI quality, benchmarking, safety, observability, and automation to ensure production-ready systems. Healthcare/insurance domain knowledge preferred.

Responsibilities

• Test Generative AI and conversational systems
• Perform LLM evaluation and benchmark testing
• Validate RAG (retrieval quality, relevance, grounding)
• Build automation frameworks (Playwright/Selenium)
• Conduct API, regression, and E2E testing
• Perform AI safety, red teaming, and bias validation
• Support observability, monitoring, and HITL workflows
• Collaborate with engineering, QA, and DevOps teams
• Support CI/CD quality pipelines

Required Skills

• QA & automation testing
• Playwright/Selenium, API testing (Postman, REST)
• GenAI & RAG validation, LLM evaluation
• Prompt engineering, conversational AI testing
• HITL testing, AI safety & observability
• SQL, Git, Python, CI/CD concepts

Preferred Skills

• OpenAI, Azure OpenAI, ChatGPT, Claude
• LangChain, CrewAI, LangGraph, MCP
• Vector DBs (Pinecone, ChromaDB, Weaviate)
• RAGAS, DeepEval
• Python/JS, Docker, Kubernetes, GitHub Actions

Location: Remote