AI Engineering Lead, Evals and Systems Integration

What Makes Us a Great Place to WorkWe are proud to be consistently recognized as one of the world’s best places to work. We are currently the top ranked consulting firm on Glassdoor’s Best Places to Work list and have earned the #1 overall spot a record seven times. Extraordinary teams are at the heart of our business strategy, but these don’t happen by chance. They require intentional focus on bringing together a broad set of backgrounds, cultures, experiences, perspectives, and skills in a supportive and inclusive work environment. We hire people with exceptional talent and create an environment in which every individual can thrive professionally and personally.Who You’ll Work WithYou will be part of a team building Bain’s integrated suite of agentic decisioning tools that gives large organizations always-on intelligence across their most important growth decisions. The role sits at an intersection few organizations can offer: the analytical depth and client access of a top-tier consultancy, the technical ambition of a product company, and engineering work at the frontier of applied AI.You will work inside Bain’s standing product team, building software that helps the world’s largest companies answer the questions that drive their business: where to invest, where to focus, and how to win. The impact of what you build is direct and immediate, and the platform compounds in value with every release.The platform has two layers working together: an agentic core and a set of decision modules, each solving a distinct commercial problem. The build lifecycle is connected to the real world: as case teams work alongside clients across industries and geographies, that feedback flows straight back to engineering and into the product. Every client engagement makes the platform smarter, broader, and more resilient.This is a startup inside an established company. You get genuine early-stage ownership and real influence over architecture and product direction, backed by Bain’s global scale, client relationships, and leadership commitment; the pace of a startup with the runway of a global firm.We are looking for engineering leads who see how to make the platform’s evaluations sharper and bring that instinct to every sprint without being asked, and who are comfortable with problems that do not have a known answer. The work involves building evaluation frameworks for LLM-based and agentic systems, systems integration testing, and data problems with no off-the-shelf answer. Experience with agentic systems matters: tool and context protocols such as MCP, evaluation harnesses, and entity resolution across messy enterprise data. This is applied AI in production, at scale, with real users and a team depending on how you approach problems that do not have a known answer. The field is moving quickly and the team is built to move with it, so the willingness to keep learning counts for as much as what you already know.Where You’ll Fit Within the TeamThe AI Engineering Lead, Evals and Systems Integration, owns the evaluation framework and systems integration layer across the entire Growth AI Performance System platform. This is a cross-cutting role: you are not evaluating a single agent or a single module, but the coherence, quality, and reliability of the full system: from the data ontology layer through the Agentic Core to the three product modules that clients use. Your work is the quality gate that determines what is ready to ship and what is not.You will work directly across three teams: the Data Ontology and Knowledge team, where you evaluate whether the semantic layer is producing outputs that are accurate, consistent, and consumable by downstream AI systems; the Agentic Core and Interface team, where you evaluate whether agentic workflows are executing reliably, reasoning correctly, and failing gracefully; and the Platform and Security team, where you ensure that evaluation and observability infrastructure is hardened, compliant, and integrated with the platform’s security and audit controls.You will work across the platform and product modules focused on marketing ROI, brand, and pricing strategy, defining what good looks like for each domain, building the evaluation datasets and rubrics that capture it, and enforcing deployment-readiness standards consistently across all three modules before anything reaches a live client engagement.This role has significant client proximity. You will engage with consulting teams after client deployments to understand where the system fell short, translate that feedback into structured evaluation improvements, and present quality and reliability findings to senior client stakeholders. The evaluation framework you build is not just an internal engineering tool: it is part of what makes clients trust what the platform produces.What You’ll DoEvaluation framework design and ownershipOwn and evolve the evaluation framework across the full Growth AI Performance System platform, defining quality standards, scoring rubrics, evaluation datasets, and deployment-readiness thresholds for the data ontology layer, the Agentic Core, and all three product modulesDesign and maintain evaluation datasets and regression test suites that enable systematic, reproducible measurement of system quality across the full stack: from knowledge graph accuracy and entity resolution correctness through to agentic workflow reliability and end-to-end output qualityDefine what ‘good enough to deploy’ means at each layer of the system (semantic layer, agentic orchestration, retrieval pipelines, and product module outputs), and enforce those standards consistently before any change reaches a live client environmentDesign experiments to measure system quality, detect regressions, and surface the root cause of quality failures; apply statistical reasoning and experimental design to distinguish signal from noise in evaluation resultsBuild automated evaluation pipelines that integrate into CI/CD workflows so that quality gates run before every deployment; ensure evaluation is a continuous process, not a pre-release checkpointSystems integration and cross-team evaluationEvaluate the integration points between the data ontology layer and the Agentic Core, validating that knowledge graph outputs are correctly structured, semantically consistent, and reliably consumable by agentic retrieval and reasoning systemsWork with the Data Ontology and Knowledge team to define evaluation criteria for entity resolution quality, taxonomy consistency, and data contract correctness; build the tests and monitoring that catch ontology-layer failures before they propagate downstreamWork with the Agentic Core and Interface team to evaluate multi-agent orchestration, tool use reliability, context passing, and end-to-end workflow correctness; define failure mode taxonomies and ensure the system degrades gracefully when components failWork with the Platform and Security team to ensure evaluation infrastructure is integrated with the platform’s security controls, audit logging, and compliance requirements; validate that evaluation pipelines themselves handle sensitive client data correctlyCoordinate evaluation work across distributed engineering teams; translate evaluation findings into concrete, prioritized engineering improvements and communicate quality status clearly to product and engineering leadershipInternal stakeholder communication skillsEngage with consulting teams after live client deployments to understand where system outputs fell short, tracing quality failures through the evaluation framework, identifying root causes, and feeding structured findings back to the relevant engineering teamsPresent evaluation findings, quality metrics, and system reliability assessments to senior stakeholders including executive leadership; translate technical quality measures into business terms that explain what the platform can and cannot be trusted to doContribute to internal stakeholder where the platform’s evaluation approach and quality standards are explained; build confidence in the rigor of the platform’s deployment process through credible, evidence-based communicationStay current with emerging evaluation methodologies, responsible AI assessment frameworks, and AI-specific risk and safety approaches; lead knowledge sharing on evaluation best practices across the engineering teamAbout YouTechnicalDeep experience designing and running evaluations for LLM-based and agentic AI systems across the full delivery lifecycle, with a track record of building evaluation frameworks that actually change what shipsStrong understanding of systems integration evaluation, able to design tests that validate not just individual components but the correctness and coherence of the interfaces between them, including knowledge graph outputs, API contracts, and agentic context passingStrong Python proficiency; experience building automated evaluation pipelines, regression test suites, and scoring rubrics that integrate into CI/CD workflows and run without manual interventionStatistical reasoning and experimental design: able to design experiments that measure system quality rigorously, interpret results without over-reading noise, and surface root causes of quality failures to engineering teamsFamiliarity with knowledge graph and ontology evaluation, understanding what entity resolution correctness, taxonomy consistency, and semantic layer quality mean in practice and how to measure them programmaticallyExperience with human-in-the-loop annotation and feedback systems; familiarity with preference learning, data labelling strategies, and feedback collection methodologies for LLM and agentic applicationsUnderstanding of security, audit, and compliance requirements for evaluation infrastructure: how to handle sensitive client data in evaluation pipelines, what audit trails need to capture, and how eval integrates with platform security controlsNative AI fluency: active use of AI-assisted coding tools (Cursor, Claude Code, or equivalent) and demonstrated ongoing development of evaluation methodology as the field evolvesDemonstrated commitment to responsible AI practices: risk assessment, bias and fairness evaluation, AI safety principles, and familiarity with AI-specific risks including prompt injection, data leakage, and agentic workflow exploitationExperience8 or more years in AI engineering, ML engineering, or applied AI roles with significant hands-on building responsibilitiesTrack record of building and owning evaluation frameworks for production AI or ML systems, not just contributing to evaluation, but defining the standards, building the infrastructure, and enforcing deployment gatesExperience evaluating systems that span multiple components and integration points, not just model-level evaluation but system-level evaluation across data pipelines, APIs, and agentic workflowsExperience working with or alongside knowledge graph, ontology, or semantic data teams is a strong advantageDirect experience engaging with clients or senior business stakeholders in a technical capacity, presenting quality findings, advising on system reliability, or contributing to client-facing deliveryExperience working in agile, cross-functional product or consulting-adjacent environments; strong troubleshooting and root cause analysis skillsEducationAssociate’s degree / Bachelor’s degree in computer science, engineering, mathematics, or a related technical field; or an equivalent combination of education, training, and experienceU.S. Compensation InformationCompensation for this role includes base salary, annual discretionary performance bonus, 401(k) plan with an annual employer contribution based on years of service and Bain’s best in class benefits package (details listed below).Some local governments in the United States require a good-faith, reasonable salary range to be included in job postings for open roles. The estimated annualized compensation for this role is as follows:In Massachusetts, the good-faith, reasonable annualized full-time salary range for this role is between $198,250 – $237,500; placement within this range will vary based on several factors including, but not limited to experience, education, licensure/certifications, training and skill levelIn Illinois, the good-faith, reasonable annualized full-time salary range for this role is between $189,500 – $227,000; placement within this range will vary based on several factors including, but not limited to experience, education, licensure/certifications, training and skill levelAnnual discretionary performance bonusThis role may also be eligible for other elements of discretionary compensation4.5% 401(k) company contribution, which increases after 3 years of service and is 100% vested upon start dateBain & Company’s comprehensive benefits and wellness program is designed to help employees achieve personal independence, protection and stability in the areas most important to you and your family.Bain pays 100% individual employee premiums for medical, dental and vision programs, offering one of the most comprehensive medical plans for employees without impacting your paycheckGenerous paid time off, including parental leave, sick leave and paid holidaysFully vested 401(k) company contributionPaid Life and Long-Term Disability insuranceFor all other locations, the good-faith, reasonable annualized full-time salary range for this role is commensurate with competitive geographic market rates for this role and will vary based on several factors including, but not limited to experience, education, licensure/certifications, training and skill level.It is unlawful in Massachusetts to require or administer a lie detector test as a condition of employment or continued employment. An employer who violates this law shall be subject to criminal penalties and civil liability.