Principal AI Engineer

Responsibilities

• Own the end-to-end technical vision for MRO Prodigy's AI layer — a production system that uses RAG, generative AI, and structured data reasoning to automate answers to Healthcare Registry questionnaires.
• Define the AI roadmap for Prodigy, balancing near-term customer commitments against foundational capability investments that scale the product to enterprise maturity.
• Evaluate and make build/buy/integrate decisions for AI capabilities — foundation model selection, embedding strategies, retrieval architectures, and orchestration frameworks — and own the consequences of those decisions.
• Serve as the technical authority on all AI design decisions for Prodigy; produce architecture decision records, set standards, and ensure the architecture is defensible, auditable, and extensible.
• Architect and evolve Prodigy's multi-modal retrieval pipeline, combining unstructured clinical document ingestion with structured EHR/FHIR data to surface accurate, citation-backed answers to registry questionnaire items.
• Design and refine the answer generation layer — prompt engineering, context construction, grounding strategies, and output formatting — ensuring generated answers are clinically accurate and audit-ready.
• Own the question routing and data source classification logic that maps registry questions to the right retrieval path, structured data field, or generation strategy.
• Build and maintain the answer validation and confidence scoring framework, defining the statistics and quality thresholds that govern when answers are auto-accepted versus routed for human review.
• Stay hands-on and close to the work: run direct ideation and feedback loops with Prodigy's end users (abstractors, registry, and quality teams) and with production analytics and monitoring systems — turning real usage signals into prioritized improvements that demonstrably move value, not just model metrics.
• Evolve the feedback loop architecture that captures human corrections and routes them into continuous model improvement — ensuring Prodigy gets measurably better with every customer interaction.
• Define the evals framework for Prodigy: how accuracy is measured, how regression is detected, and what signals trigger retraining or prompt revision.
• Establish guardrails for hallucination detection and factual grounding specific to clinical registry use cases, where answer accuracy has direct downstream compliance implications.
• Architect AI infrastructure across GCP (Vertex AI, BigQuery, Dataflow) and AWS (Bedrock), ensuring the pipeline is scalable, cost-efficient, and operationally observable.
• Collaborate with data engineering to maintain high-quality, well-governed clinical and FHIR data inputs; define feature engineering and chunking strategies that optimize retrieval precision.
• Define MLOps standards for Prodigy: model versioning, deployment gates, rollback procedures, drift monitoring, and audit trail requirements consistent with HIPAA compliance.

Qualifications

• Bachelor's in Computer Science, AI/ML, or related field; Master's or PhD preferred — or equivalent depth proven through shipped AI systems.
• Strong ML / data science / statistics theory foundation with the ability to read research, assess applicability, and execute.
• LLM Engineering & RAG Hands-on LLM integration: prompt engineering, grounding, citation, hallucination mitigation, and output validation at clinical accuracy standards.
• Experience with LangChain, LlamaIndex, or equivalent orchestration frameworks.
• Built confidence scoring and auto-acceptance thresholds that govern when answers route to human review.
• Designed human-in-the-loop feedback systems that capture corrections and feed them back into model improvement.
• Production experience building RAG pipelines — document ingestion, chunking, embedding model selection, vector store management, and retrieval evaluation.
• AI/ML Engineering & MLOps Full ML lifecycle ownership in production: versioning, deployment gates, drift monitoring, rollback, and audit trails.
• Strong Python and software engineering fundamentals — CI/CD, testing, code review.
• Hands-on with vector databases (pgvector, Pinecone, Weaviate, or equivalent) and hybrid search.
• Built evals frameworks that measure accuracy, precision, recall, and F1 to inform product decisioning.
• Cloud & Data Architecture Solid AWS and GCP experience: Bedrock, SageMaker, Vertex AI, BigQuery, Dataflow. Azure familiarity a plus.
• Experience building pipelines over mixed unstructured and structured data sources.
• FHIR/HL7 and clinical document format familiarity strongly preferred.
• Healthcare & Compliance Clinical NLP or healthcare AI experience — medical terminology, document structure, and regulated accuracy standards are not new territory.
• Prefer direct experience with clinical documentation and abstraction workflows.
• Knows how HIPAA applies to AI systems specifically: PHI in prompts and embeddings, data residency, audit logging, de-identification.
• Familiar with AI governance in practice: bias detection, explainability, responsible AI in compliance-sensitive contexts.
• Technical Leadership Has owned AI technical vision before — not just contributed to it.
• Can write an ADR, set an engineering standard, and make it stick across teams.
• Communicates tradeoffs clearly to both engineers and non-technical stakeholders.
• Track record of mentoring engineers and raising AI maturity on a team.