Lead Principal Core Infrastructure Engineer

About the role

Mentors teams and leads the architecture of highly scalable, interdependent distributed systems. Identifies and removes performance/scalability bottlenecks for hyper‑scale workloads; defines scalability requirements with stakeholders; and designs elastic, high‑impact systems while advancing innovation in data plane platforms.

Engineers and oversees fault‑tolerant, in‑service‑upgradable designs; optimizes resilience mechanisms (load‑shedding, throttling, rate‑limiting); and sets SLO‑aligned durability and availability standards across dependent services. Establishes KPIs and advanced telemetry; applies formal verification for complex features; and develops robust replication/synchronization strategies.

Advises and leads resolution of complex production issues, sets operational readiness and SOP standards, and directs incident response and RCAs. Architects advanced security controls, drives remediation and compliance, and delivers enterprise‑level automation (IaC) and change strategies enabling safe, automated patching, updates, and rollbacks.

Responsibilities

System Design & Architecture

System Scalability

• Mentor the team in the architecture and design of highly scalable, interdependent distributed systems, ensuring horizontal and vertical scalability and overall performance, including leveraging distributed state management tools.
• Lead the identification of performance and scalability bottlenecks and recommend solutions to optimize code and/or systems for large-scale data processing and high-throughput requirements to improve performance for hyper-scale systems.
• Lead collaboration with stakeholders to define system scalability requirements, ensuring the defined requirements meet customer expectations.
• Leverage deep expertise to design high-impact, interdependent systems to scale with elasticity (e.g., effectively scaling both up and down).
• Drive innovation in the use of data plane platforms.
• Evaluate whether systems are meeting nonfunctional scalability requirements, and proactively anticipate growing business needs within the business unit.

System Reliability Design

• Design and oversee the implementation of fault-tolerant, interdependent systems capable of withstanding in-service updates by implementing sophisticated redundancy, replication, and automatic failover capabilities.
• Lead the design and implementation of systems that effectively handle service disruptions (e.g., network partitions) by prioritizing consistency, availability, or partition tolerance.
• Guide the optimization of advanced mechanisms to handle network unreliability, including load-shedding, throttling, and rate-limiting.
• Design interdependent systems that are durable and adhere to service level objectives (SLOs), driving standards for availability and durability of other computing services within the organization.

System Reliability Performance

• Define key performance indicators (KPIs) and telemetry to identify risks, gaps, or cyclical dependencies in running, interdependent systems.
• Drive the creation and customization of highly complex dashboards, telemetry systems, and alerting mechanisms, proactively ensuring system health and reliability.

Correctness / Availability

• Maintain expertise in industry standards for verifying correctness and apply existing techniques to interdependent systems.
• Formally verify complex features (e.g., via TLA+) to ensure system design correctness for various interdependent systems.
• Develop advanced strategies for data replication and synchronization, ensuring robust data integrity and availability.

Operational Troubleshooting & Incident Management

• Advise on efforts to diagnose, debug, and resolve complex issues in active, interdependent systems to support ongoing operation.
• Develop and implement comprehensive strategies to prevent interruptions, ensuring no maintenance windows are required for customers and users when resolving issues.
• Maintain expertise in dependencies, dependents, and owned systems to drive effective troubleshooting and performance.
• Set standards for operational readiness and standard operating procedures within the department, and hold third-party partners accountable for meeting those standards.
• Oversee operational support rotations, providing expert guidance in incident response and leading root cause investigations to prevent future occurrences.

Compliance & Security

• Architect advanced security measures to protect data and applications in multi-tenant environments, and lead initiatives to enhance data and application protection.
• Guide the execution of comprehensive remediation plans to address identified security vulnerabilities.
• Ensure cloud infrastructure is in compliance with industry standards and regulations, and guide documentation efforts across projects.

Automation & Change Management

• Develop enterprise-level automation tools and strategies (e.g., Infrastructure as Code (IaC)) and oversee their implementation.
• Drive alignment of change management plans and organizational initiatives for patching, updating, and rolling back applications, and design interdependent systems to allow for automation of these processes.

Core Responsibilities

• Manages and provides direction on timelines, deliverables, and budgets when applicable for critical high-impact projects or initiatives that impact the line of business, ensuring timely completion and adherence to requirements.
• Anticipates and plans for shifts in resources or timelines based on changing business priorities, ensuring optimal outcomes.
• Influences cross-functional leaders and external stakeholders to gain alignment on strategic objectives.
• Fosters partnerships with key business leaders, stakeholders, and/or customers, identifying opportunities for expanding partnerships and promoting long-term organizational success.
• Leads specialized, advanced problem-solving efforts, serving as an escalation point for complex issues.
• Leverages deep industry knowledge and expertise to serve as a thought leader within the organization.
• Develops innovative solutions and drives the implementation of ideas that increase the efficiency and effectiveness of processes, protocols, and workflows across the organization.
• Serves as a subject matter expert regarding talent needs and organizational talent strategy.

Qualifications

Oracle Cloud Infrastructure (OCI) is building the next generation of cloud infrastructure for customers around the world. Our Network Automation team develops the software systems that enable OCI to design, build, scale, and operate its global network. We are seeking a Lead Principal Software Engineer to help build and operate software for network control- and management-plane services.

In this role, you will work on highly available, distributed systems that manage and automate operations across hundreds of thousands of network devices. You will be a hands-on technical leader who can independently drive complex projects, influence architecture and operational practices, and partner across engineering, operations, quality assurance, and vendor teams. You will help improve the reliability, scalability, performance, and operability of critical OCI networking services.

Pay

US: Hiring Range in USD from: $146,300 – $306,400 per year. May be eligible for bonus, equity, and compensation deferral.

Benefits

• Medical, dental, and vision insurance, including expert medical opinion
• Short term disability and long term disability
• Life insurance and AD&D
• Supplemental life insurance (Employee/Spouse/Child)
• Health care and dependent care Flexible Spending Accounts
• Pre-tax commuter and parking benefits
• 401(k) Savings and Investment Plan with company match
• Paid time off: Flexible Vacation is provided to all eligible employees assigned to a salaried (non-overtime eligible) position. Accrued Vacation is provided to all other employees eligible for vacation benefits. For employees working at least 35 hours per week, the vacation accrual rate is 13 days annually for the first three years of employment and 18 days annually for subsequent years of employment. Vacation accrual is prorated for employees working between 20 and 34 hours per week. Employees working fewer than 20 hours per week are not eligible for vacation.
• 11 paid holidays
• Paid sick leave: 72 hours of paid sick leave upon date of hire. Refreshes each calendar year. Unused balance will carry over each year up to a maximum cap of 112 hours.
• Paid parental leave
• Adoption assistance
• Employee Stock Purchase Plan
• Financial planning and group legal
• Voluntary benefits including auto, homeowner and pet insurance