Principal Core Infrastructure Engineer

Responsibilities

• System Scalability
• Lead the development and implementation, and begin to architect, components of scalable distributed systems that support horizontal and vertical scaling to meet system demands, including leveraging distributed state management tools.
• Optimize code and/or systems for large-scale data processing and high-throughput requirements to support hyper-scale systems.
• Define scalability requirements for owned components and ensure design and implementation requirements are met.
• Design systems to scale with elasticity (e.g., effectively scaling both up and down).
• Leverage data plane platforms to effectively handle large-scale data retrieval, storage, and processing.
• Design performance and load testing.
• System Reliability Design
• Build and design fault-tolerant components and systems capable of withstanding in-service updates by implementing redundancy, replication, and automatic failover mechanisms.
• Design systems to effectively handle service disruptions (e.g., network partitions) by prioritizing consistency, availability, or partition tolerance.
• Implement and optimize approaches to handle network unreliability, including load-shedding, throttling, and rate-limiting.
• Design components and systems that are durable and adhere to service level objectives (SLOs), setting expectations for availability and durability of other computing services within the department.
• System Reliability Performance
• Define key performance indicators (KPIs) and telemetry to identify gaps or issues in running systems.
• Build and customize moderately complex dashboards, telemetry systems, and alerting mechanisms to proactively monitor components and system health.
• Correctness / Availability
• Design and implement functional and correctness requirements for feature sets and/or systems in new or existing systems.
• Design complex test scenarios (e.g., fault-injection, brown-out) to evaluate system correctness.
• Implement data replication and synchronization techniques to maintain data integrity and availability.
• Operational Troubleshooting & Incident Management
• Take a proactive role in diagnosing, debugging, and resolving issues in active components and systems to support ongoing operation, and mentor others in these processes.
• Implement strategies to prevent interruptions, ensuring no maintenance windows are required for customers and users when resolving issues.
• Maintain expertise in owned components and systems to ensure effective troubleshooting and performance.
• Meet operational readiness expectations through design and implementation.
• Serve in operational support rotations, providing guidance in incident response and root cause investigations.
• Compliance & Security
• Implement robust security measures to protect data and applications in multi-tenant environments, including encryption techniques and access controls.
• Execute remediation plans to address identified security gaps.
• Ensure cloud infrastructure is in compliance with industry standards and regulations and that documentation is up to date.
• Automation & Change Management
• Develop and maintain automation scripts and tools (e.g., Infrastructure as Code (IaC)) to manage cloud infrastructure.
• Create and adhere to change management plans for patching, updating, and rolling back applications, and begin designing systems and components to allow for automation of these processes.

Qualifications

• 8+ years of experience in security engineering, application development, or related field
• 6+ years of experience with systems/application development
• 4+ years of experience with distributed systems or network programming
• Demonstrated ability to write code in Java or similar OO languages
• Experience building automated network and security solutions
• Strong security and network experience, particularly with focus in network, security and identity and service architecture and design patterns
• Proficient with data structures, algorithms, operating systems
• Experience with distributed systems: familiarity with CAP theorem, Consensus, messaging, High Availability
• Strong collaboration and communication skills
• Experience in building new cloud products, specifically security and network related
• Experience with Windows and Linux