Applied Research Engineer

About the role

Sully.ai is building the most impactful healthcare company on earth, aiming to make access to a great doctor a basic human right. We develop AI teammates that augment clinicians across various roles in healthcare.

Responsibilities

• Design and scale automated evaluation pipelines using LLMs and human review for clinical tasks.
• Develop and audit existing evaluation approaches for clinical and agentic tasks.
• Define initial benchmarks and build early automated pipelines for evaluation.
• Partner with engineering to implement CI gates for accuracy, factuality, and safety.
• Deliver a repeatable evaluation framework with automated pipelines in production.
• Demonstrate measurable improvements in robustness, hallucination reduction, or safety.
• Publish or present internal research findings that directly shape product reliability.

Requirements

• Proven experience designing agentic processes and LLM evaluation/benchmarking frameworks.
• Strong Python and ML background (PyTorch/TensorFlow, Hugging Face, LangChain/LlamaIndex).
• Demonstrated ability to design rigorous experiments and translate findings into production.
• Track record of published research or deep applied work in LLMs and agent evaluation.
• Strong communication and technical writing skills to articulate complex findings clearly.

Qualifications

• A-players capable of 4 months = 1 year output.
• Experience with clinical-grade benchmarks and evaluation frameworks.
• Ability to work independently and in a fast-paced environment.
• Passionate about improving healthcare outcomes and patient care.

Skills

• Python and ML background (PyTorch/TensorFlow, Hugging Face, LangChain/LlamaIndex).
• Experience with LLMs and agent evaluation.
• Strong communication and technical writing skills.
• Ability to design rigorous experiments and translate findings into production.

Benefits

• Competitive compensation range of $180K - $220K.
• Remote work options available in Mountain View, CA, US; San Francisco, CA, US; Berkeley, CA, US; Stanford, CA, US; Walnut Creek, CA, US; Pleasanton, CA, US; Danville, CA, US; CA, US; San Jose, CA, US; Santa Rosa, CA, US; Oakland, CA, US; Menlo Park, CA, US; San Mateo, CA, US; San Ramon, CA, US; Dublin, CA, US; Richmond, CA, US; Concord, CA, US; Fremont, CA, US; Milpitas, CA, US; Livermore, CA, US; Sunnyvale, CA, US; Palo Alto, CA, US.

Pay

• $180K - $220K

Schedule

• Remote (Mountain View, CA, US; San Francisco, CA, US; Berkeley, CA, US; Stanford, CA, US; Walnut Creek, CA, US; Pleasanton, CA, US; Danville, CA, US; CA, US; San Jose, CA, US; Santa Rosa, CA, US; Oakland, CA, US; Menlo Park, CA, US; San Mateo, CA, US; San Ramon, CA, US; Dublin, CA, US; Richmond, CA, US; Concord, CA, US; Fremont, CA, US; Milpitas, CA, US; Livermore, CA, US; Sunnyvale, CA, US; Palo Alto, CA, US)