Sr. AI Data Analyst-Agentic Systems & GenAI

Responsibilities

• Drive the delivery and operational validation of Agentic AI solutions through structured data analysis and reporting.
• Define and implement data-driven validation frameworks to evaluate AI system performance, accuracy, reliability, and business impact.
• Analyze production data from AI systems (agents, workflows, prompts, responses) to identify trends, issues, and optimization opportunities.
• Develop dashboards, reports, and metrics to track the health and effectiveness of Agentic AI deployments.
• Partner with architecture and engineering teams to validate feasibility outcomes and ensure solutions align with real-world system behavior.
• Monitor AI systems in production, identifying anomalies, failure patterns, hallucinations, and performance degradation.
• Support deployment efforts by validating readiness criteria, including performance thresholds, guardrails, and compliance requirements.
• Enable continuous improvement loops by feeding insights back into model tuning, prompt design, and system architecture.
• Collaborate with business stakeholders to measure and report on AI-driven business outcomes and ROI.
• Ensure transparency and traceability of AI decisions through structured logging, trace analysis, and reporting.
• Contribute to the development of AI observability frameworks, including metrics, KPIs, and alerting strategies.

Qualifications

• Validate readiness of Agentic AI use cases for production deployment.
• Track deployment success metrics and post-production performance.
• Identify gaps between expected vs. actual outcomes.
• Define metrics for: Accuracy and response quality, Task completion success rates, Hallucination and failure cases, Latency and throughput.
• Build evaluation datasets and validation pipelines.
• Analyze: Agent workflows and decisions, Prompt-response chains, Tool usage and orchestration behavior.
• Develop observability dashboards using telemetry and logs.
• Detect and escalate production issues and anomalies.
• Data Analysis & Reporting:
• Perform root cause analysis on failures and performance issues.
• Deliver executive-level reporting on AI system effectiveness.
• Provide actionable insights to improve system design and outcomes.
• Work closely with: Lead Architects for feasibility alignment, AI/ML engineers for model/system improvements, Product teams for use case refinement.
• Translate technical findings into clear business insights.
• Advanced SQL, Python (Pandas, NumPy), or similar tools.
• Data visualization platforms (Power BI, Tableau).
• Strong experience in data validation, anomaly detection, and statistical analysis.
• Familiarity with: LLM workflows and prompt engineering, RAG pipelines and evaluation strategies, Agent orchestration and tool integration.
• Understanding of AI failure modes (hallucinations, drift, inconsistency).
• Experience with: Logging, tracing, and telemetry systems, AI evaluation tools and frameworks, Monitoring production systems (Azure Monitor, Application Insights).
• Strong working knowledge of Azure ecosystem, including: Azure OpenAI / AI services, Azure Databricks, Data platforms (Azure SQL, Cosmos DB), Monitoring tools (Log Analytics, App Insights).
• Strong analytical and problem-solving skills in complex AI-driven systems.
• Able to connect system behavior with business outcomes.
• Expertise in translating data into actionable insights.
• High attention to detail in validation, quality, and accuracy.
• Strong communication skills across technical and non-technical stakeholders.
• Able to thrive in fast-evolving AI environments.
• Able to wrangle large datasets, structured and non-structured data, including data mining and manipulation.

What We Offer

• Generous benefits package available on day one to include: 401K matching, bonding leave for new parents (12 weeks, 100% paid), tuition assistance, training, GM employee auto discount, community service pay and nine company holidays.
• Our Culture: An environment that welcomes innovative ideas, fosters integrity, and creates a sense of community and belonging.
• Compensation: Competitive pay and bonus eligibility.
• Work Life Balance: Hybrid work environment, 2-days a week in office.