Senior AI Engineer

This role is for one of Weekday's clients Min Experience: 10+ years Location: Bengaluru and Palo Alto Department: Engineering Employment JobType: full-time We are looking for a highly experienced Senior AI Engineer to design, develop, and operate the backend and AI infrastructure required to run intelligent AI agents securely, efficiently, and reliably in production. This role combines hands-on AI engineering with strong backend development, platform engineering, and production operations. You will work across LLM-powered applications, AI agents, model routing, tool integrations, inference services, multi-agent workflows, and evaluation frameworks. You will be responsible for transforming early-stage prototypes into scalable, observable, secure, and production-ready services. The ideal candidate has strong proficiency in Python, practical experience building AI applications and agentic systems, and a solid understanding of the Model Context Protocol (MCP). Experience with Rust-based services, cloud infrastructure, and AI platform engineering will be valuable. Requirements Key Responsibilities AI and Agentic Engineering Design, build, and deploy production-grade AI applications and agent infrastructure using PythonDevelop agent workflows involving tool calling, memory, retrieval-augmented generation (RAG), planning, and multi-agent collaborationIntegrate hosted and self-hosted language models, inference endpoints, and enterprise systemsBuild MCP servers, clients, tools, and gateway integrations to enable secure and governed access to external tools and dataImplement structured outputs, retries, fallbacks, error recovery, and human approval workflowsImprove AI system performance through evaluation, experimentation, tracing, and continuous feedback Backend and Platform Engineering Develop scalable APIs, services, asynchronous workers, and backend components for AI-powered systemsDesign reusable abstractions for models, agents, tools, workflows, policies, and execution environmentsBuild event-driven services for long-running AI workloadsWork with PostgreSQL, Redis, message queues, object storage, and vector databasesContribute to Rust-based platform services and architectural decisions where requiredWrite maintainable, well-tested code with clear interfaces and strong operational visibility AI Infrastructure and MLOps Build repeatable workflows for packaging, deploying, versioning, evaluating, and rolling back AI servicesDevelop real-time and batch inference pipelines and model-serving integrationsMonitor model and agent quality, latency, reliability, errors, token usage, and operational costsManage model, prompt, configuration, and evaluation versionsEstablish evaluation pipelines and feedback loops to improve AI system performanceTransition AI prototypes into secure, scalable, and observable production environments DevOps, Security, and Reliability Own services throughout development, deployment, monitoring, and production supportBuild and maintain CI/CD pipelines, release automation, and infrastructure-as-codeDeploy and operate applications using Docker, Kubernetes, and cloud infrastructureImplement health checks, autoscaling, alerting, graceful failure handling, and rollback strategiesApply best practices for authentication, authorization, secrets management, service identity, and tenant isolationUse logs, metrics, and distributed tracing to troubleshoot issues and improve reliability Required Qualifications 10+ years of professional experience in software engineering, AI engineering, backend development, or platform engineeringStrong proficiency in Python, including experience developing production-grade backend and AI applicationsHands-on experience building and deploying LLM-powered applications, AI agents, or agentic workflowsPractical understanding of MCP (Model Context Protocol), including tool integration, MCP servers, clients, or gateway-based architecturesExperience with LLM APIs, tool calling, embeddings, vector search, RAG, and structured generationStrong backend fundamentals covering API design, asynchronous processing, databases, and distributed servicesExperience deploying and operating production systems using Docker, Kubernetes, cloud platforms, and CI/CDFamiliarity with AI evaluation, observability, monitoring, and inference infrastructureAbility to independently take technical capabilities from design through implementation, deployment, and production supportStrong problem-solving, communication, documentation, and architectural decision-making skills Must-Have Skills Python ProgrammingArtificial Intelligence (AI) EngineeringMCP (Model Context Protocol)LLM Applications and AI AgentsBackend Development and API DesignAI Tool Integration and Agent WorkflowsProduction Deployment and Monitoring Good-to-Have Skills Rust ProgrammingLangGraph, AutoGen, Agno, or similar agent frameworksRAG, embeddings, vector databases, and semantic searchLLM routing, AI gateways, and model-serving infrastructurevLLM, Triton, ONNX Runtime, or Ray ServePostgreSQL, Redis, Kafka, or RabbitMQDocker, Kubernetes, Helm, and Terraform or PulumiOpenTelemetry, Prometheus, and GrafanaMLflow, Weights & Biases, and AI evaluation pipelinesAI security, governance, policy enforcement, and token-cost optimizationMulti-agent orchestration and distributed systems Preferred Domain Experience Candidates with direct experience building AI infrastructure, LLM routers, model gateways, MCP gateways, agent runtimes, workflow engines, inference platforms, AI observability tools, evaluation systems, or AI governance products will be strongly preferred. Experience with GPU-backed inference, multi-tenant SaaS platforms, customer-managed deployments, open-source AI infrastructure, or Rust-based production services will be an additional advantage.