Strategic Operations Manager, AI

About the role

Reports to the Head of Learning.ai (UGHE, CTLT). Based at the University of Global Health Equity (UGHE) in Rwanda. The Strategic Operations Manager owns the measurement framework and reporting for the Learning.ai portfolio, including the OneSpaceAfrica (OSA) platform and the eLearning program. The role defines and tracks metrics that show whether OSA is working — covering engagement, efficacy, efficiency, and AI performance, including quality, reliability, safety, bias, fairness, and human review effectiveness. It builds the outcomes and evaluation framework used to report results to partners (including COMS-A) and donors, turns OSA performance data into findings for academic publication, and contributes to the design of needs assessments across the COMS-A network.

Responsibilities

• Own the Learning.ai measurement framework: define core metrics for OSA and the broader portfolio (engagement, efficacy, efficiency, AI performance), set targets the team is held to, including AI quality, reliability, safety, bias, fairness, and human review effectiveness measures.
• Track progress against targets with the Head of Learning.ai and the Manager, Product & Engineering, holding the roadmap accountable to evidence without owning the roadmap itself.
• Define what AI and product outcomes need to be measured; work with engineering so the platform captures the data evaluation requires. Product decisions, instrumentation, and roadmap remain with the Product and Engineering team.
• Lead evaluation design for each funding phase: primary and secondary outcomes, baseline and comparator approach drawing on pilot usage and infrastructure data, implementation fidelity measures, adoption and usage thresholds, AI performance and safety measures, and success thresholds that would justify further investment.
• Contribute faculty time and cost metrics (cost per site, cost per student) to funding proposals.
• Contribute to needs-assessment design with the Partner Success Coordinator on interview and survey instruments so findings feed the evaluation framework.
• Hire and manage Phase 0 needs-assessment consultant(s) and the Senior Advisor, Metrics, Analytics, and AI Evaluation; hold them to their terms of reference. Day-to-day direction of needs-assessment fieldwork sits with the Partner Success Coordinator.
• Design the data strategy for performance management: determine what OSA captures directly and what comes from surveys, interviews, and other faculty/student input, first at UGHE then across the COMS-A network.
• Co-own the data management plan with the Infrastructure and DevOps Engineer, covering de-identification, data deposit, and ethics approval required before platform data is used for research.
• Lead reporting to partners, donors, and internal leadership; set guardrails for how OSA results are reported externally.
• Prepare OSA performance findings for academic publication, working with the team on analysis and write-up.
• Work with external advisors, CTLT, COMS-A, and PIH's monitoring and evaluation team to ground Learning.ai's measurement approach in established practice.

Deliverables

• Approved Learning.ai measurement framework with defined metrics, targets, and AI quality, reliability, safety, and fairness measures.
• Evaluation design for each funding phase, including baseline, comparator, and success thresholds.
• Data strategy and data management plan in place, with ethics approval secured before platform data is used for research.
• Timely and accurate reporting to partners, donors, and internal leadership.
• OSA performance findings prepared for academic publication.

Qualifications

• Bachelor's degree required in a quantitative or research-oriented field (e.g., Statistics, Economics, Public Health, Social Science, Data Science, or Business); master's degree preferred.
• Experience designing or managing program evaluation, M&E, or impact measurement; experience in education, global health, or international development preferred.
• Experience measuring or evaluating digital or AI-enabled systems, including the ability to work with technical and domain experts to define meaningful measures of quality, reliability, safety, and fairness.
• Experience with research design and ethics review, including baselines, comparators, and governance of personal data.
• Project management experience, including scoping work and managing external consultants.

Skills

• Strong quantitative skills: defining metrics, building reporting, turning data into recommendations for decision-makers.
• Comfortable working with product and engineering teams to define what data needs to be captured and how.
• Excellent oral and written communication skills in English (required); French and/or Kinyarwanda proficiency preferred.
• Ability to work effectively in a team in a complex, fast-paced environment.
• Exceptional interpersonal skills, including the ability to interact professionally with culturally diverse staff, partners, and clients.
• Flexibility and agility.
• Passion for social justice and global health.