Principal Software Engineer

About the role

We are hiring a Principal Software Engineer to lead the technical direction of a real-time, multi-service clinical platform on Google Cloud. This is not an "architect-from-the-sidelines" role. You will actively design systems, write critical code, ship to production, support live environments, and mentor engineers across UI, backend, and ML.

Responsibilities

• Own architecture and execution
• Lead the technical direction of a real-time, multi-service clinical platform on GCP, while personally building the most critical and performance-sensitive components.
• Build production-grade LLM systems
• Design and operate scalable LLM pipelines (e.g., Claude, Gemini, GPT-4-class) including prompt engineering, structured outputs, evaluation frameworks, retrieval/vector search, OCR-based document understanding, grounding, guardrails, and hallucination mitigation.
• Deliver real-time, high-performance systems
• Arcitect distributed, event-driven, streaming services that support sub-second clinician experiences under heavy load, with strict SLOs and strong fault tolerance.
• Operationalize machine learning
• Lead deployment and lifecycle management of models on Vertex AI, including feature pipelines, model registries, drift detection, A/B and shadow testing, and CI/CD-integrated quality controls.
• Design for scalability and change
• Build flexible APIs, microservices, and data contracts that support rapid iteration, backed by strong testing, observability, and progressive delivery practices.
• Raise the engineering bar
• Drive high-quality code reviews, establish standards across testing, CI/CD, security, and observability, and mentor engineers across full-stack and ML disciplines.
• Own outcomes end-to-end
• Translate ambiguous clinical and business needs into delivered solutions, monitor real-world performance, and continuously iterate based on data.
• Partner across disciplines
• Collaborate with Data Science, Product, Clinical SMEs, AI Governance, and Security teams to ensure solutions meet enterprise standards for compliance, privacy, and safety.

Requirements

• 10+ years of software engineering experience with a strong record of delivering production systems at scale
• 5+ years as a hands-on Staff or Principal-level engineer (actively coding, not removed from implementation)
• Expert proficiency in Python including performance tuning, concurrency, and system optimization
• 5+ years building real-time distributed systems (event-driven, streaming, Pub-Sub) with clear latency and reliability requirements
• 5+ years working in a major public cloud (GCP preferred: Vertex AI, GKE, Cloud Run, Pub/Sub, BigQuery, IAM, etc.)
• Deep experience with microservices architecture, API design (REST), and resilient systems (retries, backpressure, tracing)
• 3+ years building and operating production ML systems (MLOps, monitoring, retraining, deployment pipelines)
• 3+ years developing LLM-powered applications in production, including prompt design, RAG/vector search, evaluation frameworks, and performance optimization
• Experience building modern UI components with React or Angular
• Strong computer science fundamentals, including distributed systems, algorithms, and concurrency

Preferred Qualifications

• Experience operating LLM systems in regulated environments (PII/PHI, governance, security, auditability)
• Familiarity with vector databases (e.g., Vertex Vector Search, Pinecone, Weaviate, pgvector) and advanced retrieval techniques
• Experience designing evaluation frameworks for LLM systems (gold datasets, regression testing, LLM-as-judge)
• Strong background in CI/CD, SRE practices, and SLO-driven engineering
• Experience with healthcare data standards (FHIR, HL7) and compliance frameworks (HIPAA, HITRUST)
• Strong communication skills, with the ability to explain complex systems to both technical and non-technical audiences
• Proven track record of mentoring engineers into senior and staff roles