Lead Assessment Developer

What You’ll Be Doing

• You'll manage the full assessment development lifecycle at Outsmart — from item authoring through field testing, standard setting, scoring, and reporting.
• You'll work closely with psychometricians, faculty, product managers, and engineers to build rigorous, scalable assessments.
• As an AI-forward organization, you'll also help shape how AI tools are responsibly woven into our development and QA workflows.

Who You Are

• Master's degree (Ph.D. preferred) in educational measurement, assessment, or a related field
• 3+ years managing operational, high-stakes assessment lifecycles end-to-end
• Proven experience leading standard-setting studies and managing diverse SME panels
• Experience managing external vendors for field tests and pilot studies
• Working knowledge of validity, reliability, and fairness standards (e.g., AERA/APA/NCME)
• Familiarity with cognitive complexity frameworks (DOK, Bloom's)
• Strong communicator who can translate psychometric concepts for non-technical audiences
• Bonus: knowledge of CTT, IRT, CDM, DIF, AI-assisted item generation, or accreditation reviews

What You’ll Own

• End-to-end assessment development operations for assigned assessments
• SME network — recruitment, training, calibration, and quality
• Standard-setting studies — method selection, panel facilitation, and final reporting
• Vendor and external partner relationships, SLAs, and security protocols
• Field testing and pilot operations
• Scoring operations for constructed-response and performance-based assessments
• Project timelines, risk identification, and stakeholder communication
• Core assessment documentation — specs, blueprints, training materials, technical reports

What Helps You Succeed

• You thrive in ambiguity and are energized by building from scratch
• You balance psychometric rigor with startup pragmatism — you know when to ship and iterate
• You communicate with unusual clarity across technical and non-technical audiences
• You take ownership without being asked and surface risks early
• You're excited about AI's role in assessment and know where validation is non-negotiable
• You care deeply about fairness and the learners behind every score

Pay

Salary Range: $120K-$180K

Perks

• Competitive salary, equity, and benefits
• Healthcare for you and your dependents (medical, dental, and vision)
• Parental leave for any full-time employee (birthing, non-birthing, and adoptive parents)
• 15 PTO days per year, a winter break, and 12+ paid company holidays per year
• Remote work days
• Lunch provided in-office daily

About the Role

Outsmart is a mission-driven, early-stage startup reimagining higher education for the age of AI by creating an entirely new collegiate experience that expands access and opportunity. Backed by $38M from top venture firms including Khosla Ventures, Forerunner, DST Global Partners, and Lightspeed, the company is led by former Duolingo executives and a founding team that also includes Emmy Award winners. With a strong culture centered around high-impact, detail-oriented team members, Outsmart is already seeing significant traction, generating 1–5 million weekly organic views on social media.