Principal Engineer AI

About the role

BMO is building the platform capabilities that make enterprise AI safe, governed, and scalable. We are seeking experienced Principal/Senior engineers to build and operate the core infrastructure that governs how AI runs at BMO - the AI Gateway, Policy Engine, Identity Fabric, AI Registry, Guardrails Runtime, and AI Observability.

Responsibilities

• Own capabilities end to end: design, implement, test, ship, and operate production infrastructure.
• Engineer for operability and defensibility from day one: instrumentation, SLOs, latency budgets, failure modes, and runtime evidence built in, not bolted on.
• Implement policy enforcement, guardrails, identity attestation, and audit as first-class engineering concerns: correct, performant, and provable.
• Achieve every capability produces runtime evidence connecting AI activity to policy enforcement, identity, and lineage for model-risk and regulatory review (OSFI E-23, OCC).
• Assess emerging AI infrastructure, foundation-model access patterns, and standards; make deliberate, cost-aware engineering choices.
• Mentor and raise the bar: set engineering standards, review designs and code, and grow depth across the team.
• Partner closely with AI Developer Experience, AI Security and AI SDLC, and the Senior AI Architect in your platform build and operations.

Requirements

• Bachelor's degree in Computer Science, Software Engineering, or a related technical discipline (Master's preferred).
• 8+ years of software/platform engineering experience (Principal), or 5+ years (Senior), with substantial time building and operating shared platform services at enterprise scale.
• Demonstrated experience operating production infrastructure with real SLOs and on-call ownership, ideally in a regulated industry (financial services strongly preferred).
• Depth in one or more of: API gateways / traffic enforcement; policy-as-code and authorization; workload identity / zero-trust; observability and telemetry pipelines; audit/compliance data platforms.

Qualifications

• Strong distributed-systems and platform-engineering fundamentals: latency-sensitive request paths, resilience patterns (circuit breakers, failover), multi-tenancy, and high availability.
• Strong programming skills (Python and/or Go preferred; TypeScript/Java an asset) for building performant services, APIs, and integrations.
• Cloud-native architecture across AWS and Azure: containers/Kubernetes, service mesh, and Infrastructure as Code (CDK, Terraform, CloudFormation/ARM).
• Robust CI/CD, GitOps, and DevSecOps practice; Git-based workflows (Bitbucket/GitHub), Jira, Confluence.

Skills

• Platform Engineering Depth
• Strong programming skills (Python and/or Go preferred; TypeScript/Java an asset)
• Cloud-native architecture across AWS and Azure
• Robust CI/CD, GitOps, and DevSecOps practice
• Policy/Authorization: Cedar, OPA/Rego, policy compilation and distribution, risk-tiered approval workflows
• Identity/Zero-Trust: SPIFFE/SPIRE, mTLS, token exchange, OAuth/OIDC, federated and cross-cloud identity, Entra ID/Agent ID
• Observability/Audit: OpenTelemetry (incl. GenAI conventions), distributed tracing, Dynatrace/Splunk or equivalents, tamper-evident/immutable audit stores, data lineage
• Gateway/Guardrails: API gateway internals, inline enforcement, LLM routing/abstraction, prompt-injection and PII defenses, hallucination detection, AI evaluation
• Registry/Portal: service catalogues, asset registries, lifecycle workflows, federation with external registries

Benefits

GenAI & Governance Context: Working knowledge of GenAI platform patterns: LLM/AI gateways, RAG and agentic patterns, foundation models, embeddings, and guardrails - sufficient to build the infrastructure they depend on. Familiarity with AI/ML platforms (Bedrock, Azure OpenAI, SageMaker, MLflow) and orchestration frameworks (LangChain, LlamaIndex). Grounding in Responsible AI, AI/data governance, privacy, cloud security, and IAM as applied to AI workloads.

Pay

$112,200.00 - $209,000.00

Schedule

Salaried