QA Engineer, AI Products

The Opportunity

Since 2005, MDCalc has been an essential part of the clinician’s workflow to help achieve better patient outcomes. Actively used by more than 65% of physicians worldwide, MDCalc is the most broadly used medical reference – at the point-of-care – for clinical decision tools and content, and one of only four references used by >50% of US HCPs. These evidence-based tools and content are used by millions of medical professionals globally and support 50+ specialties and cover 200+ patient conditions. To continue to further accelerate and steward this growth, we are expanding the AI product team with a QA Engineer.

Role

To continue to further accelerate and steward this growth, we are expanding the AI product team with a QA Engineer. This role will be critical to MDCalc’s expanded success in continuing to support our millions of clinical users worldwide in taking care of hundreds of millions of patients.

Responsibilities

• Design and execute test strategies for LLM-powered features, including prompt regression testing, output evaluation, and hallucination detection
• Build and maintain automated evaluation pipelines (eval sets, golden datasets, LLM-as-judge frameworks) to catch quality regressions in non-deterministic outputs
• Perform black-box and exploratory testing of MDCalc's AI features across web and mobile, with particular attention to clinical accuracy, safety, and edge cases
• Define quality metrics for AI outputs (accuracy, faithfulness, relevance, safety, latency, cost) and establish thresholds for release readiness
• Collaborate cross-functionally with engineers, product managers, ML/AI engineers, and clinical reviewers to define what "good" looks like for AI responses
• Investigate and triage AI failure modes, distinguishing model issues, prompt issues, retrieval issues, and integration bugs
• Participate in team discussions, offering feedback on testability, risks, prompt design, and guardrails
• Help develop QA strategies to expand future testing capacity, automation, and evaluation coverage as the AI product surface grows

Background

• 5+ years of experience in software QA, with at least 1 year of hands-on testing of LLM-based or AI/ML-powered features
• Strong understanding of QA principles, test case creation/documentation, and best practices for both deterministic and non-deterministic systems
• Hands-on experience with LLM tooling and concepts: prompt engineering, RAG systems, evaluation frameworks (e.g., Promptfoo, Braintrust, LangSmith, DeepEval, Ragas, OpenAI Evals), and LLM APIs (OpenAI, Anthropic, etc.)
• Experience designing automated qualitative evaluation approaches, including LLM-as-judge, rubric-based scoring, semantic similarity checks, and golden dataset regression testing
• Proficiency with test automation tools, with a focus on Playwright
• Strong SQL skills for data validation, test data creation, and verifying data integrity across systems
• Familiarity with token usage, latency profiling, and cost monitoring as quality signals
• Eagerness to learn quickly and a positive, solutions-oriented attitude
• Clear and concise communicator, able to surface issues, blockers, and risks effectively when communicating ambiguous or probabilistic failures
• Self-motivated, proactive, and able to manage time and priorities independently

MDCalc Offers

• Able to make a true difference in medicine: MDCalc is the most broadly used medical reference by physicians, used by over 65% of US attending doctors weekly
• Medical, Dental, & Vision Coverage, with option to extend to your dependents
• Company-sponsored short-term insurance
• Fully-paid 8 week parental leave, after 6 months of employment
• Company-sponsored 401k, after 3 months of employment
• Unlimited vacation for salaried roles - we trust you to take the time you need
• Work from home monthly stipend
• A culture of fun and motivated team members who believe in a greater mission here at MDCalc