Site Reliability Engineer (SRE) - AI Platform & Cloud

About the role

This is a Software Engineering position at Director level within the Technology division of Morgan Stanley. Since 1935, Morgan Stanley has been a global leader in financial services, always evolving and innovating to better serve clients and communities.

Responsibilities

• Operate, monitor, and maintain the infrastructure supporting GenAI applications (training, inference, feature store, data ingestion, model serving)
• Design and build automation for core platform capabilities, reducing manual toil
• Establish, monitor, and enforce Service Level Indicators (SLIs), Service Level Agreements (SLAs), error budgets, alerting, and dashboards
• Lead incident response, root cause analysis (RCA), postmortems, and systemic remediation
• Perform capacity planning, scaling strategies, workload scheduling, and resource forecasting
• Optimize cost vs. performance tradeoffs in large-scale compute environments
• Harden systems for security, compliance, auditability, and data governance
• Collaborate across teams (cloud engineers, data engineers, infrastructure, security) to ensure safe deployment, rollout, rollback, and integration of new systems
• Define disaster recovery (DR) strategies, backup/restore practices, fault tolerance mechanisms
• Maintain runbooks, operational playbooks, documentation, and training materials
• Participate in on-call rotations and respond to production incidents 24/7 as needed
• Continuously evaluate and integrate new tools, frameworks, or technologies to enhance platform reliability

Requirements

Minimum of 5 years of production experience in SRE / Infrastructure / ops for large-scale systems

Strong programming/scripting skills (Python, Go, Java, or equivalent)

Deep experience with containerization (Docker), orchestration (Kubernetes, etc.), and infrastructure-as-code (Terraform, Helm, CloudFormation, Ansible, etc.)

Familiarity with GPU / AI compute clusters, high-performance data storage, and distributed architectures

Experience with monitoring / observability / logging / alerting tools (Prometheus, Grafana, ELK / EFK, Datadog, etc.)

Solid experience in capacity planning, performance tuning, scaling, and incident response

Demonstrated ability to lead RCAs, deploy fixes, and drive reliability improvements

Experience in regulated environments (financial services, compliance, audit, security) is a strong plus

Excellent communication, documentation, and cross-team collaboration skills

Proven track record of reducing operational toil via automation

Qualifications

Bachelor’s or Master’s degree in Computer Science or related field, or equivalent job experience

Skills

• Understanding of SRE techniques
• Proficiency with Open Telemetry tools including Grafana, Loki, Prometheus, and Cortex
• Good knowledge of Microservice based architecture, industry standards, for both public and private cloud
• Knowledge of data pipeline technologies (Kafka, Spark, Flink, etc.)
• Good knowledge of various DB engines (SQL, Redis, Kafka, Snowflake, etc.) for cloud app storage
• Experience working with Generative AI development, embeddings, fine tuning of Generative AI models
• Experience in high-performance computing (HPC), distributed GPU cluster scheduling (e.g. Slurm, Kubernetes GPU scheduling)
• Understanding of ModelOps/ ML Ops/ LLM Ops
• Experience with chaos engineering, canary deployments, blue/green rollouts

Benefits

Morgan Stanley offers a comprehensive benefits package, including:

• Health insurance
• Retirement savings plans
• Flexible spending accounts
• Employee assistance programs
• Parental leave
• Wellness programs
• Professional development opportunities
• Flexible work arrangements

Pay

The salary range for this position is $200,000 - $300,000 annually, depending on experience and qualifications.

Schedule

The role requires a 24/7 on-call rotation and participation in production incident response.