Lead Cloud Platform Engineer (Data & Execution Platform)

About the role

We are seeking a hands-on Lead Cloud Platform Engineer to implement, scale, and operate cloud-native infrastructure and services that power large-scale data processing systems. This role focuses on translating defined architectures into production-grade platforms that are reliable, observable, secure, and performant.

Responsibilities

• Implement and operate cloud-native platform services for distributed data systems
• Scale fault-tolerant, high-throughput systems aligned with architectural patterns
• Own Spark data pipelines and Airflow orchestration layer and DAG execution
• Tune Spark workloads (partitioning, memory, execution plans, shuffle optimization)
• Troubleshoot Spark jobs and Airflow DAGs across performance and failures
• Operate and optimize Kubernetes-based execution environments, including node group scaling, workload placement, and resource utilization
• Troubleshoot Kubernetes infrastructure and workload issues, including scheduling, networking, and runtime performance
• Leverage developer productivity tools (e.g., GitHub Copilot, LLMs) to accelerate development, debugging, and operational workflows
• Drive operational excellence including monitoring, incident response, and RCA
• Implement observability (metrics, logging, tracing, dashboards, alerting)
• Define and manage SLIs/SLOs for platform reliability
• Deploy solutions using AWS services (EKS, EC2, S3, Lambda, RDS, etc.)
• (Implement secure networking (VPCs, IAM, subnets, load balancing))
• Maintain CI/CD pipelines and deployment automation
• Lead execution across planning, delivery, and cross-team coordination
• Mentor engineers and promote reliability and scalability best practices

Requirements

• Strong understanding of distributed systems (fault tolerance, scalability, consistency)
• Expertise in Apache Spark (tuning, debugging, optimization)
• Expertise in Apache Airflow (DAG execution, orchestration, troubleshooting)
• Strong experience operating Kubernetes (EKS preferred) including cluster scaling and lifecycle management
• Hands-on management of node groups, autoscaling, and capacity planning
• Deep understanding of Kubernetes networking and security (security groups, network policies, ingress/egress)
• Experience with Kubernetes resources (Deployments, StatefulSets, Jobs, CronJobs)
• Familiarity with Custom Resources (CRDs) and advanced configuration via annotations and labels
• Experience monitoring Kubernetes clusters (metrics, logs, events) and integrating with observability tools
• Experience troubleshooting Kubernetes workloads (scheduling failures, resource contention, networking issues)
• Experience with AWS services and cloud-native design patterns
• Proficiency in Python, Java, or Go
• Experience with Docker and Kubernetes
• Experience with observability (metrics, logging, tracing)
• Experience with SLI/SLO-based reliability models
• Practical experience using AI-assisted development tools (e.g., GitHub Copilot, LLMs) to improve code quality, debugging, and productivity
• Networking fundamentals (DNS, TCP/IP, TLS, VPC design)
• Strong troubleshooting and performance tuning skills

Skills

• Strong communication and leadership skills
• Bachelor’s or Master’s degree in Computer Science or related field (or equivalent experience)
• 8 plus years in software, platform, or cloud engineering roles
• Strong experience with AWS cloud platforms
• Mandatory hands-on experience with Apache Spark and Apache Airflow in production
• Experience supporting ETL, data platforms, or workflow execution systems at scale

Benefits

Fidelity is transitioning to a full-time onsite working model through a phased rollout across regions and roles. Currently, some roles and locations require 100% onsite presence, while others require less. Onsite expectations are likely to evolve as the rollout continues. This transition does not apply to fully remote roles.

Pay

Details on pay are not specified in the job description.

Schedule

Details on schedule are not specified in the job description.