Product Analyst III (AI Context Designer)
Bayside Solutions · Carrollton, TX · Yesterday
On-siteInformation Technology$38.07–$48.07/hrContract
Duties and Responsibilities
- Act as a "Context Designer" to write, refine, and configure natural language prompts and conversational flows based on defined business logic.
- Collaborate daily with sales, servicing, and fulfillment teams to capture, define, and negotiate business requirements for AI agent interactions.
- Review and curate operational PDFs and documentation to provide a verified "Source of Truth" for the RAG Vector Database.
- Execute A/B tests on various prompt configurations and analyze the resulting data to improve accuracy, tone, and resolution rates.
- Assist the Product Owner in defining and documenting the "Golden Dataset" of test scenarios to support automated AI evaluations.
- Work alongside platform engineers who manage the underlying technical architecture within the AWS tech stack to ensure successful API handshakes and platform functionality.
Requirements and Qualifications
- 3-5 years of experience as a Product Analyst, Business Analyst, or similar role (targeting Analyst III level).
- A non-engineer with a strong "technical bent" and the ability to quickly grasp Generative AI concepts (e.g., RAG, LLM orchestration, prompt tuning).
Desired Skills and Experience
- Prompt Engineering
- AI Context Design
- Conversational AI
- Voice AI
- Natural Language Prompt Development
- Conversational Flow Configuration
- Generative AI
- Retrieval-Augmented Generation (RAG)
- Large Language Models (LLMs)
- LLM Orchestration
- Prompt Tuning
- Vector Databases
- Knowledge Base Curation
- Business Requirements Gathering
- Business Analysis
- Product Analysis
- Use Case Development
- A/B Testing
- Data Analysis
- Golden Dataset Development
- AI Evaluation
- Test Scenario Documentation
- API Integrations
- AWS
- Cross-Functional Collaboration
- Sales
- Mortgage Servicing
- Fulfillment