Volunteer AI Data Annotator (Benchmarks & Communication Rubrics) — AI Apps for Human Communication

About the role

Crossing Party Lines (CPL) provides opportunities, training, and support for civil, respectful discourse between Americans with dissimilar ideologies. Its primary offering is the CPL meetup—regularly scheduled, facilitated meetings where attendees discuss current hot topics while facilitators teach and model skills for active listening, perspective‑taking, neutrality, de‑escalation, and connection across political divides. In 2016 two independently founded CPL groups joined forces to bring this work nationwide.

Crossing Party Lines is seeking a thoughtful, detail‑focused AI Data Annotator to join our Apps and Agents team. Our conversational agents are built to handle emotionally sensitive, politically nuanced interactions, and human ground‑truth data is essential to train, evaluate, and align models that reduce political polarization.

The Apps and Agents team integrates generative AI with CPL’s proprietary frameworks developed through nearly a decade of real‑world dialogue facilitation. As we prepare a Y Combinator application and expand our digital role‑play and coaching tools, high‑quality human data is essential to ensure our agents remain objective, emotionally safe, and genuinely depolarizing.

Responsibilities

• Review multi‑turn conversational transcripts between users and AI agents across diverse, high‑friction social and political topics.
• Annotate and score AI outputs using structured rubrics covering active listening, perspective‑taking, neutrality, de‑escalation, tone calibration, and defensiveness triggers.
• Identify and tag failure modes such as subtle ideological bias, sycophancy, tone‑policing, preachy phrasing, and hallucinations.
• Write and curate high‑quality "golden responses" (ideal model outputs) to serve as few‑shot exemplars and ground‑truth references for evaluation pipelines.
• Provide qualitative feedback to AI engineers, prompt designers, and subject‑matter experts on recurring model behaviors and annotation edge cases.
• Participate in inter‑annotator calibration sessions to maintain high data consistency and label reliability across the team.

Requirements

• Strong reading comprehension, analytical judgment, and close attention to language nuance and subtext.
• An objective, open‑minded approach capable of evaluating dialogue across a wide spectrum of political and social viewpoints without personal bias.
• Interest in linguistics, psychology, communication studies, political science, philosophy, or AI safety.
• Enjoy repetitive, analytical tasks that require high precision, consistency, and adherence to labeling guidelines.
• Prior experience with data annotation platforms (Label Studio, Prodigy, or custom spreadsheets) is a plus, but not required.

Benefits

• Practical experience in human‑in‑the‑loop (HITL) AI development and LLM benchmarking methodologies.
• A direct role in setting the safety and alignment standards for civic technology applications.
• Collaboration with AI engineers, data specialists, and conflict‑resolution experts.
• Professional references recognizing your attention to detail, analytical rigor, and contributions to production AI evaluation datasets.