Product Analyst

About the role

The role involves PMs running multiple workstreams and analysts taking individual complex items within those streams. Analysts are responsible for scope the problem, design the experiment, define how success is measured, run the analysis, and provide recommendations. They don't write training or infrastructure code but rather focus on experimental design and analytical rigor.

Responsibilities

• Guardrail training data: curate and design the data that trains an SLM guardrail (e.g., a prompt injection detector), and ensure labeling consistency as datasets grow.
• Orchestration experiments: test different configurations of guardrails and controls against each other to identify the best performing setup.
• Synthetic data quality: assess the fidelity and diversity of synthetic training or evaluation data.
• New evaluations: research and scope novel ways to evaluate guardrails and AI applications, then specify how they should work.
• Rapid tooling: build small custom tools with AI to validate a hypothesis or unblock the team.
• Benchmarking: measure performance using standard classification metrics (FNR, FPR, precision, recall) and report on what works and what doesn't.

Requirements

• A degree with strong quantitative or analytical content, particularly from less common backgrounds like physics, information or data theory, business, etc., combined with solid data-analysis experience.
• A clear, demonstrable approach to problem-solving.
• A good instinct for data, machine learning concepts, and what makes an experiment or metric trustworthy.
• Proficiency in Python and pandas for data manipulation.
• An interest in AI security or safety and adversarial thinking.
• An attention to detail and clear writing skills.

Nice to have

• ML coursework.
• Reading or working proficiency in Japanese, Chinese, or a European language.
• Coursework or projects in statistics or natural language processing.
• SQL skills.
• A habit of building small tools or scripts (including with AI assistants) to answer one’s own questions.
• Exposure to large language models, security tooling, experimental design, or annotation work.