Business Intelligence Analyst Senior (AI Evaluation & Monitoring)

Job Summary

The Senior BI Analyst serves as a partner to the business or clinical area supported (AI Evaluation & Monitoring) and will use analytic expertise including knowledge of data structures, programming, and statistics with critical thinking and an understanding of their assigned subject area/business to help the organization make data-driven decisions.

Senior BI Analysts consistently act as the analytics expert in meetings and discussions with their stakeholders and can operate independently. Must be articulate in both oral and written communications, including presentations with the ability to translate analyses and opportunity insights to non-technical stakeholders.

Job Duties

• Independently consults with AI stakeholders to gather evaluation and data requirements and define scope on projects of all sizes.
• Execute the evaluation work that keeps Geisinger's AI portfolio safe, performant, and accountable — running validation studies, instrumenting production monitoring, analyzing results, and turning findings into clear written work product that governance and program teams can act on.
• Collaborates with internal partners and organizations in the design and development of reporting solutions and analytic studies to support business needs.
• Plans projects and request completion tasks.
• Provides time and effort estimates based on requirements gathering.
• Strong experience independently managing multiple projects is also required.
• Demonstrates working knowledge of relevant source systems needed to support primary stakeholders and proficiency with related databases/data marts within first year in role.
• Must stay abreast of current industry trends and become familiar with the various business intelligence software systems used at Geisinger.
• Recommends analysis approach and additional analyses or reporting needed based on business questions, requirements, and initial analysis findings.
• Develops, validates, and executes complex queries.
• Develops complex reports and performs complex data analysis that answers business questions for internal customers.
• Interprets results and makes recommendations to business areas based on findings.
• Presents analysis and reports to internal customers and stakeholders.
• Routinely uses statistical methods during analyses and considers how best to visualize data that accurately reflects the intended message including the limitations.
• Reports and helps troubleshoot data quality issues when performing data profiling, testing, or validation.
• Collaborative and eager to share knowledge with team members.
• Responsible for participating in strategic departmental initiatives.
• Acts as a mentor to other analysts.
• May participate in industry advisory and/or user groups.
• Strong attention to detail and the ability to be precise and clear when describing results of data analyses.
• Shares important and relevant information with the team.

Preferred Skills

• Healthcare, clinical, or regulated-industry data experience strongly preferred
• Applied statistics: hypothesis testing, confidence intervals, power and sample size, regression, and non-parametric methods
• SQL with large, messy clinical data warehouses, including window functions, CTEs, debugging query plans, and performance tuning
• Python for analysis: pandas, numpy, scikit-learn, statsmodels, common evaluation libraries
• Production monitoring design and execution: drift detection, performance decay tracking, adoption and outcome metrics
• LLM and generative AI evaluation execution: golden sets, judge-based scoring, hallucination and grounding checks

Education

• High School Diploma or Equivalent (GED)

Experience

• Minimum of 8 years - Relevant experience

Certification(s) and License(s)

• OUR PURPOSE & VALUES: Everything we do is about caring for our patients, our members, our students, our Geisinger family and our communities.
• KINDNESS: We strive to treat everyone as we would hope to be treated ourselves.
• EXCELLENCE: We treasure colleagues who humbly strive for excellence.
• LEARNING: We share our knowledge with the best and brightest to better prepare the caregivers for tomorrow.
• INNOVATION: We constantly seek new and better ways to care for our patients, our members, our community, and the nation.
• SAFETY: We provide a safe environment for our patients and members and the Geisinger family

Benefits

• We offer healthcare benefits for full time and part time positions from day one, including vision, dental and domestic partners.