Clinical Annotator

About the role

A highly skilled and detail‑oriented Human Subject Matter Expert (SME) specializing in clinical pre‑annotation validation. The successful candidate will play a critical role in the incremental annotation process, focusing on human validation of pre‑annotated data, refining annotation guidelines, and contributing to the development of high‑quality golden datasets. Experience with the John Snow Labs (JSL) ecosystem and advanced NLP concepts is highly desirable.

Responsibilities

• Perform rigorous human validation on pre‑annotated data generated by commercial clinical NLP models (e.g., Amazon Comprehend Medical) or internal LLM/encoder‑based tools.
• Contribute to the iterative refinement of annotation guidelines and examples to improve inter‑annotator agreement and overall data quality.
• Identify and resolve disagreements between pre‑annotations and human validations, ensuring accuracy and consistency of the annotated datasets.
• Collaborate closely with data science and NLP teams to provide feedback on model performance and support continuous improvement of pre‑annotation models.
• Assist in the creation and maintenance of golden datasets for training and evaluating NLP models.
• Participate in discussions regarding data sensitivity and ensure adherence to all relevant data privacy and security protocols, especially for patient data.
• Work on diverse clinical annotation projects (e.g., alcohol and smoking use cases, oncology – ECOG, Karnofsky scores, biomarkers, SDO, and mental health).

Qualifications

Must Have (Required)

• Bachelor's or Master's degree in a relevant field (e.g., Life Sciences, Linguistics, Computer Science, or a related healthcare discipline).
• Proven experience in data annotation, specifically with clinical or biomedical text.
• Strong foundational understanding of NLP concepts and terminology.
• Exceptional attention to detail and ability to maintain high levels of accuracy in data validation.
• Excellent communication and collaboration skills.
• Ability to quickly adapt to various annotation tools (e.g., Inception, Label Studio, Prodigy).

Good To Have (Preferred)

• Prior experience working within the John Snow Labs (JSL) ecosystem for modeling and annotation, including familiarity with their Health AI Lab and GenAI tool.
• Experience with advanced annotation types such as named entity recognition (NER), relationship extraction, and assertion status annotation.
• Direct clinical background or extensive practical experience working with sensitive patient data in a healthcare context.
• Familiarity with incremental batch training processes in machine learning.