Research Scientist III - AMZ9674099

Multiple positions available.

About the role

Perform all psychometric aspects of exam development, operations, and research including automated test assembly, item and test analyses, optimal item bank design, job task analysis, standard setting, quality assurance, and project planning for AWS certification exams.

Responsibilities

• Conduct psychometric analysis in operational work, including:
  • Performing item analysis using psychometric methods
  • Building optimal test forms and pools via optimization techniques
  • Analyzing and monitoring item bank health
  • Setting pass standards via standard setting studies
  • Supporting Job Task Analysis (JTA) to define and refresh test blueprints
• Develop and apply statistical and psychometric modeling to evaluate and ensure AWS certification exams’ validity, reliability, applicability, efficiency, and accuracy.
• Identify and lead research projects to improve operational processes and quality using advanced techniques such as Machine Learning (ML), statistical modeling, Natural Language Processing (NLP), and Generative Artificial Intelligence (GenAI).
• Leverage emerging AI and ML technologies to enhance exam content development, content management, evaluation, and other psychometric tasks.
• Develop and maintain automation tools using R or Python for psychometric workflow pipelines and other tasks to improve operational efficiencies.
• Present, interpret, and communicate analysis results to stakeholders through written and oral reports.
• Follow accreditation standards set by ISO/IEC:2012 17024 and the National Council for Certifying Agencies (NCCA) for valid psychometric practices.
• Contribute to the development and execution of strategic goals for the AWS certification program.
• Consult with leadership, internal staff, external consultants, and industry leaders regarding advancement of current offerings, notably within the AI/ML space.
• Engage with the professional community through conferences and publications.

Requirements

• Ph.D. or foreign equivalent degree in Statistics, Psychometrics, Educational Measurement, Quantitative Psychology, Data Science, Industrial-Organizational (I/O) Psychology, or a related field.
• Three years of research or work experience in the job offered or a related occupation.
• Three years of experience in the following:
  • Working with large-scale education, licensure, or certification assessment programs.
  • Performing psychometric analyses for large-scale programs, including:
    • Item analysis
    • Equating and scaling
    • Item response theory and classical test theory
    • Form and pool assembly
    • Item bank health analysis
    • Standard setting
    • Job task analysis
  • Operational and research experience with at least one advanced test design, such as linear-on-the-fly testing (LOFT) or computerized adaptive testing (CAT).
  • Technical knowledge and practical experience in at least one of the following: Machine Learning (ML), Natural Language Processing (NLP), or Generative Artificial Intelligence (GenAI).
  • Script-based programming skills in R or Python.

Benefits

Amazon offers a total compensation package, including medical, financial, and other benefits. Equity, sign-on payments, and other forms of compensation may also be provided. For more information, visit Amazon Employee Benefits.

Pay

Salary range: $164,697/year to $215,300/year.

Schedule

40 hours per week, 8:00 AM – 5:00 PM.