Real-Time Inference Engineering Lead

About the role

NTT DATA is seeking a Real-Time Inference Engineering Lead to join our team in Charlotte, North Carolina (US-NC), United States (US). This hands-on engineering role will design, build, and industrialize low-latency, resilient model-serving services for real-time predictive AI use cases across public cloud and on-premises environments.

Responsibilities

• Define the target architecture and engineering standards for real-time predictive model-serving services across cloud and on-premises environments.
• Design, build, test, deploy, and operate scalable online inference services that meet latency, throughput, availability, resiliency, and security requirements.
• Establish reusable model-serving patterns for synchronous APIs, asynchronous inference, batch-adjacent processing, and event-driven real-time use cases.
• Build standardized deployment approaches for predictive models, including model packaging, versioning, release promotion, canary deployment, rollback, and retirement.
• Design and implement secure API patterns for inference services, including authentication, authorization, traffic management, rate limiting, auditability, and integration with enterprise systems.
• Engineer Kubernetes-based serving platforms using GKE, OpenShift, and related container orchestration capabilities.
• Implement autoscaling, resource allocation, quota management, capacity planning, and workload-isolation controls for variable inference demand.
• Conduct performance engineering, load testing, stress testing, and failure testing to validate service behavior under expected and peak production workloads.
• Identify and implement latency-optimization opportunities across model initialization, feature retrieval, network paths, API handling, runtime configuration, and infrastructure utilization.
• Define and implement monitoring, telemetry, dashboards, alerts, SLIs, SLOs, and error-budget practices for real-time inference services.
• Partner with ML platform, data engineering, application engineering, security, and operations teams to integrate model services with governed data, feature, network, and identity capabilities.
• Implement CI/CD and automated validation for model-serving services, infrastructure configuration, APIs, performance benchmarks, and release-readiness checks.
• Build operational runbooks, incident-response procedures, support models, and production-readiness artifacts for real-time services.
• Drive reliability improvements through root-cause analysis, capacity reviews, resiliency testing, disaster-recovery planning, and continuous operational improvement.
• Mentor engineers and establish reusable technical documentation, reference implementations, and knowledge-transfer materials for real-time inference capabilities.

Requirements

• 8+ years of software engineering, platform engineering, cloud engineering, SRE, or infrastructure engineering experience.
• 4+ years of experience designing, building, or operating production APIs, distributed systems, platform services, or real-time data and ML workloads.
• Demonstrated experience leading technical design and engineering delivery for highly available, performance-sensitive production services.
• Strong experience with online inference architecture, model-serving frameworks, or predictive-model deployment patterns.
• Hands-on experience designing and operating RESTful, gRPC, or event-driven APIs.
• 4+ years of experience with Kubernetes and container platforms in production, including GKE, OpenShift, or comparable environments.
• Experience with autoscaling, resource management, capacity planning, performance testing, and load testing for distributed services.
• Experience implementing observability, monitoring, dashboards, alerts, SLIs, SLOs, and incident-management practices.
• Experience with CI/CD, Git-based development, automated testing, deployment automation, and production-release practices.
• Strong understanding of resiliency, high availability, fault tolerance, disaster recovery, and operational support for critical services.
• Ability to work effectively with data science, ML engineering, platform engineering, application teams, security, and business stakeholders.

Skills / Knowledge

• Online inference and low-latency model-serving architecture.
• Model deployment, versioning, routing, rollout, rollback, and lifecycle management.
• REST APIs, gRPC, API gateways, authentication, authorization, traffic management, and API observability.
• Kubernetes, GKE, OpenShift, containers, service meshes, ingress, workload scheduling, and autoscaling.
• Performance engineering, load testing, stress testing, benchmarking, profiling, and latency optimization.
• Monitoring, telemetry, distributed tracing, dashboards, alerting, SLIs, SLOs, and error budgets.
• CI/CD, automated testing, deployment automation, infrastructure-as-code, and release controls.
• Resiliency engineering, high availability, capacity controls, incident response, root-cause analysis, and operational runbooks.
• Cloud and on-premises platform operations, networking, identity, data protection, and secure production delivery.

Preferred Qualifications

• Experience with Vertex AI endpoints, KServe, Seldon, NVIDIA Triton Inference Server, MLflow deployments, or comparable model-serving technologies.
• Experience deploying and operating models on GCP, Azure, AWS, private cloud, or hybrid-cloud environments.
• Experience with service mesh, API gateway, traffic-routing, or edge-serving technologies.
• Experience serving high-volume, customer-facing, fraud, risk, personalization, decisioning, or other latency-sensitive predictive models.
• Experience with feature-serving, online feature stores, caching, streaming platforms, or real-time data enrichment.
• Experience with Terraform, Helm, Argo CD, Jenkins, GitHub Actions, GitLab CI, or similar automation tooling.
• Experience in banking, financial services, healthcare, insurance, or another regulated enterprise environment.
• Experience participating in a 24x7 operational support model for high-priority production services.

Pay

The starting hourly range for this remote role is $70–$80/hour. Actual compensation will depend on the candidate’s work location, relevant experience, technical skills, and other qualifications. This position may also be eligible for incentive compensation based on individual and/or company performance.

Benefits

• Medical, dental, and vision insurance.
• Flexible spending or health savings account.
• Life and AD&D insurance.
• Short- and long-term disability coverage.
• Paid time off.
• Employee assistance program.
• 401(k) program with company match.
• Additional voluntary or legally required benefits.