AI Quality Assurance Analyst Associate(Fellowship Program)

Responsibilities

• AI Quality Assurance & Test Automation
• Supports designing, developing, and executing automated test strategies for AI-enabled applications.
• Supports testing of AI workflows, data pipelines, and system integrations.
• Assists with prompt testing, hallucination detection, and evaluation of AI model performance using defined metrics.
• Supports basic bias and safety validation activities and participates in human-in-the-loop review processes to ensure accurate and appropriate AI outputs.
• Identifies and documents defects, inconsistencies, and risks during testing cycles.
• Supports validating AI model inputs, outputs, and data flows to ensure accuracy and reliability.
• Supports testing of AI-driven features, including conversational agents and workflow automation.
• Supports functions such as data analysis related to AI system performance, including reviewing evaluation outputs, identifying trends.
• Assists with reporting model quality metrics.

Qualifications

• Required Education: Bachelor’s degree or Master's degree in Computer Science, Information Technology, Software Engineering, Data Science, or a closely related field awarded within the last 12 months.
• Required Experience: None.
• Required Registration, Certification, Licensure: None.
• Preferred Qualifications: Experience with prompt engineering, proficiency in Python or a similar programming language, experience with database queries (e.g., SQL), familiarity with testing AI workflows or data-driven applications, exposure to Microsoft Copilot Studio, familiarity with Azure AI services, and understanding of AI data flows, familiarity with CI/CD tools and automated testing frameworks, knowledge of machine learning and large language models, AI system evaluation techniques, prompt engineering concepts and conversational AI design, responsible AI principles, basic QA methodologies, software testing concepts, and automation approaches, software development lifecycle and modern DevOps practices, skill in evaluating AI generated outputs using quantitative and qualitative methods, writing effective prompts for conversational AI systems, using evaluating datasets and defining expected AI behavior, writing and executing test cases (manual and automated), analyzing data and identifying defects or inconsistencies, communicating technical findings to technical and non-technical stakeholders, managing multiple tasks in a structured, team-based environment, ability to analyze AI system behavior and identify non-deterministic issues, evaluate ambiguous outputs and apply judgment-based validation, collaborating with data scientists and AI engineers on model improvements, working with business users on requirement clarification, user acceptance tests, and troubleshooting issues, planning, organizing, and coordinating work assignments to effectively meet frequent and/or multiple deadlines; handling multiple tasks simultaneously; and managing conflicting priorities and demands, working in a team test environment, analyzing problems and devising effective solutions, effective verbal and written communication of complex technical information, establishing and maintaining harmonious working relationships with co-workers, agency staff, and external contacts, working effectively in a professional team environment.