Member of Technical Staff (Data Scientist/Engineer, Online Metrics)
Perplexity · San Francisco, CA · 3 wk ago
HybridEngineering$200k–$320k/yrFull-time
Responsibilities
- Discover and validate online signals from user interactions that serve as reliable proxies for true answer quality
- Design and implement novel online metrics to be tracked both in A/B testing and on product health dashboards, ensuring alignment with ground-truth evaluations
- Analyze experimental results to validate these metrics, ensuring they accurately predict user satisfaction and drive product decisions
- Build and maintain the data pipelines that calculate these metrics at scale, delivering actionable quality signals to Search, Product, and model training teams
- Communicate findings and bring clarity through close collaboration with Product and Search teams
Qualifications
- MS in a technical field or equivalent experience
- 4+ years of experience working as a Data Scientist, Analytics Engineer, or related role
- Experience working on search, recommendation, or LLM-based products, with an emphasis on designing online metrics and analyzing A/B experiments
- Strong proficiency in Python and SQL (expected to write production-grade code)
- Deep knowledge of statistical analysis
- Experience with Business Intelligence (BI) tools for visualization and reporting
- Comfortable with agentic coding workflows and using AI-assisted development tools to iterate faster
Preferred Qualifications
- Proficiency with Apache Spark and Databricks
- Experience with the development or validation of LLM-as-a-judge systems
- Prior work supporting customer-facing products at scale
Compensation Range
$200K - $320K