Novel AI Lead Methodologist

About the role

Novel Testing is a team within Trust and Safety specializing in complex testing, defining protocols and methodologies for assessing risk where best practices do not currently exist. We pioneer and scale testing programs, streamlining the launch of trustworthy, novel AI products. Work spans from designing first-of-their-kind evaluations for Google’s most ambitious product bets—including autonomous agents, personalization, and the latest hardware—to developing new methodologies for assessing novel foundational model capabilities as they emerge. Advancing in AI evaluation is central to this mission.

Responsibilities

• Drive the methodological frontier of model evaluation.
• Partner with DeepMind and Data Science, developing novel, data-driven methodologies for structured and unstructured testing of emerging AI products.
• Move beyond standard benchmarks, designing sophisticated experimental frameworks, uncovering latent model behaviors and capabilities.
• Define testing and safety standards, working with cross-functional colleagues to ensure they are met.
• Perform analyses and drive insights to develop model-level and product-level safety mitigations.
• Lead and influence cross-functional teams to implement safety initiatives.
• Advise executive leadership on complex safety issues.
• Represent Google's AI safety efforts in external forums and collaborations, contributing to industry-wide best practices.
• Mentor analysts, fostering a culture of excellence, acting as a subject matter expert on adversarial techniques.
• Work with sensitive content or situations and may be exposed to graphic, controversial or upsetting topics or content.

Requirements

• Bachelor's degree or equivalent practical experience.
• 10 years of experience in AI testing or research, data analytics, data science, or a related field.
• Preferred qualifications:
  • Master's degree or PhD in relevant field.
  • 5 years of experience in data analysis for AI Testing with experience in SQL or Python.
  • Experience building or partnering with engineering teams to build prototypes for AI testing.
  • Experience in designing and conducting experiments or quantitative research, preferably in a technology or AI context.
  • Experience in AI systems, machine learning, and their potential risks.
  • Strong technical competency with a data-driven investigative approach to solve complex tests, including demonstrable proficiency in data manipulation, analysis, and automation using languages like Python and SQL.

Qualifications

• Strong technical competency with a data-driven investigative approach to solve complex tests, including demonstrable proficiency in data manipulation, analysis, and automation using languages like Python and SQL.

Skills

• Strong technical competency with a data-driven investigative approach to solve complex tests, including demonstrable proficiency in data manipulation, analysis, and automation using languages like Python and SQL.

Benefits

• Health, dental, vision, life, disability insurance.
• Retailment Benefits: 401(k) with company match.
• Paid Time Off: 20 days of vacation per year, accruing at a rate of 6.15 hours per pay period for the first five years of employment.
• Sick Time: 40 hours/year (increased to 69 hours/year for Seattle) including 5 discretionary sick days per instance.
• Maternity Leave (Short-Term Disability + Baby Bonding): 28-30 weeks.
• Baby Bonding Leave: 18 weeks.
• Holidays: 13 paid days per year.

Pay

US: $171000 - $248000 (USD) + 20% bonus target + equity + benefits

Schedule

Not specified