Product Assurance Engineer

Responsibilities

• Design and execute workload-driven stress tests, focusing on realistic production traffic patterns.
• Instrument, observe, and correlate metrics to characterize and pinpoint system issues.
• Drive chaos-monkey-style tests to simulate hardware failure modes and control plane outages.
• Investigate availability and reliability gaps and produce reproducible failure reports.
• Work with engineers across the software stack to triage findings and verify fixes and mitigations.
• Improve the assurance infrastructure itself, including scripts, harnesses, chaos tooling, and automated regression suites.

Requirements

• Deep familiarity with troubleshooting traffic issues at different network layers and characterizing performance and reliability of network data paths.
• Comfortable with low-level debugging, reading packet captures, and reasoning from raw protocol state.
• Enjoy hands-on work with test infrastructure.
• A track record of finding non-obvious reliability bugs in complex systems.
• Proficiency in at least one scripting or systems language (Python, Rust, Go) for building test harnesses and automating workload generation.
• Enjoy writing: clear bug reports, knowledge articles, and technical documentation.

Qualifications

• Experience with rack-scale networking and understanding of our implementation.
• Knowledge of Hell is other networks and its impact on customer problems.
• Understanding of the Network Behind the Network and the specific challenges of our service processor network.
• Experience with Systems Software in the Large and a multi-year effort to develop full-system update.
• Experience with When Nine Nines Isn’t Enough and a multi-year search for an evasive failure.
• Experience with RFDs: The Backbone of Oxide and our writing-intensive culture.

Skills

• Scripting or systems language proficiency (Python, Rust, Go).
• Low-level debugging skills.
• Experience with stress testing and reliability testing.
• Ability to write clear bug reports and technical documentation.

Benefits

• Premium healthcare coverage covering 100% of premiums for employees and dependents.
• $275,000 USD base salary for all employees, regardless of location.

Pay

$275,000 USD

Schedule

Remote first, with workday overlaps with Pacific Time for at least four hours. Flexible working hours.