Lead Site Reliability Engineer

Responsibilities

• Implement AI systems and automations to assist during ongoing outages and triage potential ones.
• Work with the development teams to ensure they have all the data needed during an outage, including preliminary analysis by AI.
• Provide Tier 3 support for issues across portal services by troubleshooting and resolving technical issues in test and production environments.
• Lead root cause analysis and post-mortem processes, incorporating AI-assisted analysis and pattern detection to ensure continuous improvement.
• Monitor system performance and proactively identify bottlenecks or degradation using AI-driven observability and anomaly detection tools.
• Implement tuning strategies across application layers, databases, and infrastructure.
• Drive initiatives to improve latency, throughput, and resource utilization.
• Deploy improved alerting for Lumen Connect, combining traditional monitoring with AI-based anomaly detection and noise reduction.
• Proactively monitor errors and performance on Lumen Connect, implementing rules to detect deviations and improving together with the teams.
• Design and maintain dashboards, alerts, and metrics using tools like Datadog, AppInsights, CloudWatch, or similar.
• Develop and maintain automation scripts and tools for deployment, scaling, and recovery, leveraging AI-assisted code generation and validation tools.
• Use Terraform, or similar IaC tools, to manage AWS resources.
• Perform an in-depth analysis of the overall system and its dependencies, implementing techniques to increase global availability, reduce reliance on unstable dependencies, and guide ecosystem improvements.
• Champion SRE principles such as SLIs, SLOs, and error budgets.
• Advocate for resilient architecture and fault-tolerant design patterns, incorporating AI-assisted design reviews and architecture evaluation.
• Collaborate closely with software engineers, DevOps, and product teams to align reliability goals.
• Document processes, runbooks, and best practices for knowledge sharing.
• Provide mentorship and guidance on reliability and operational excellence.

Qualifications

• 5 years overall professional experience in SRE, DevOps, or infrastructure engineering roles.
• Experience with Terraform, or similar IaC tools to manage Cloud resources.
• Proficiency in scripting languages (Python, Bash, etc.) and automation frameworks.
• Experience with CI/CD pipelines and tools like GitHub Actions, Jenkins or GitLab CI.
• Solid understanding of monitoring and logging tools (e.g., CloudWatch, ELK, Datadog).
• Familiarity with containerization and orchestration (Docker, Kubernetes).
• Excellent AI and problem-solving skills, and a proactive mindset.