Sr. Psychometrician, Data Center Certifications

Key job responsibilities

• Lead the psychometric design and validation of certification assessments across role-based and specialty-based programs, including written, practical, and performance-based formats
• Design and audit scalable Job Task Analyses (JTAs) in partnership with subject matter experts and operations stakeholders to establish defensible content blueprints
• Apply classical test theory (CTT) and item response theory (IRT) to evaluate item and test quality; conduct standard-setting studies (e.g., Angoff, Bookmark) to establish and validate cut scores
• Develop and govern item development, review, and validation workflows across global content contributors, ensuring consistency and rigor
• Analyze assessment data to identify bias, differential item functioning (DIF), and reliability issues; recommend and implement remediation strategies
• Maintain independence between training and certification development to protect assessment integrity
• Build and maintain psychometric reporting frameworks that surface actionable insights on pass rates, item performance, and certification quality
• Partner with third-party assessment vendors on test development, delivery, and quality assurance; hold vendors accountable to psychometric standards
• Communicate complex psychometric concepts clearly to non-technical stakeholders, including senior leadership

About the team

The DC Certifications team is redefining how work readiness is measured across AWS data centers. Rather than tracking training completion, we deliver outcome-based certifications that validate whether a Builder can confidently do the job. Our program assesses competencies tied to specific data center tasks, using practical and simulated environments to maintain high, consistent standards. We automate assessment delivery to keep the process simple, while ensuring independence and rigor through qualified assessors. Behind the scenes, we provide the tooling and data solutions that help sites plan and manage their workforce's capabilities.

Basic Qualifications

• Knowledge of data analysis languages (e.g., R, Stata, Python, etc.) or other statistical software
• Master's degree in Psychometrics, Educational Measurement, Industrial-Organizational Psychology, or a related field
• 5+ years of experience in psychometrics, assessment development, or credentialing programs
• Demonstrated experience with JTA facilitation, item development, and standard-setting methodologies
• Experience working with large-scale, high-stakes certification or licensure programs

Preferred Qualifications

• Ph.D. in econometrics, statistics, industrial engineering, operations research, optimization, data mining, analytics, or equivalent quantitative field
• Experience in technical or workforce credentialing programs (e.g., trades, engineering, operations)
• Familiarity with performance-based and simulation-based assessment design
• Experience managing vendor relationships in an assessment or credentialing context
• Ability to operate effectively in a fast-paced, globally distributed environment