Sr. AI Data Analyst-Agentic Systems & GenAI
GM Financial · Irving, TX · 2 days ago
HybridInformation TechnologyFull-time
Responsibilities
- Validate that AI use cases align with real-world outcomes.
- Monitor agent behavior, performance, and reliability.
- Establish data-driven feedback loops for continuous improvement.
Qualifications
- Validate readiness of Agentic AI use cases for production deployment.
- Track deployment success metrics and post-production performance.
- Identify gaps between expected vs. actual outcomes.
- Define metrics for: Accuracy and response quality, Task completion success rates, Hallucination and failure cases, Latency and throughput.
- Analyze: Agent workflows and decisions, Prompt-response chains, Tool usage and orchestration behavior.
- Develop observability dashboards using telemetry and logs.
- Detect and escalate production issues and anomalies.
- Deliver executive-level reporting on AI system effectiveness.
- Provide actionable insights to improve system design and outcomes.
- Work closely with: Lead Architects for feasibility alignment AI/ML engineers for model/system improvements Product teams for use case refinement.
- Translate technical findings into clear business insights.