Member of Technical Staff - Debugger Generation

Company: Infinity Artificial Intelligence Institute
Team: Systems / AI Infrastructure
Location: San Francisco (on-site)
Type: Full-time

About the role

The mission is to build a debugger that refuses to let program state disappear. When an accelerator run hangs, a race condition fires, or a result fails to reproduce, the state explaining the issue typically evaporates the moment execution moves on. Traditional debugging involves instrumenting more, re-running, and narrowing in on increasingly detailed dumps—often an endless loop for non-deterministic failures. Numerical bugs are even harder to localize manually, as corruption usually originates far upstream of where it surfaces.

A real debugger solves this by freezing the entire program, including every value’s physical location, and allowing bidirectional execution. This role focuses on building the agent that generates such a debugger for new chips, leveraging continuous checkpointing with restore-to-checkpoint: snapshot the run as it proceeds, then jump back to the instant before a hang or numerical degradation to inspect memory without rewriting code.

The same frozen-state representation accelerates optimization. Restore to any point, modify an inference pass, measure the effect, and undo it by restoring—eliminating full rebuilds and reruns. This collapses the rewrite-and-rerun cycle into checkpoint-and-restore, yielding 10–20x speedups in experimentation. For numerical bugs (often deterministic given inputs), the loop closes cleanly: restore, scan for NaN/inf, diff tensors against a known-good pass, ablate precision changes, and re-run.

This debugger must run directly on the chip (no simulator) and generalize across hardware, from AI accelerators to mobile SoCs like Snapdragon. The near-term target is the d-Matrix Corsair debugger, currently limited to a single breakpoint and no checkpointing. The goal is to add real breakpoints, continuous checkpointing, and restore-to-intermediate-state functionality, enabling AMPs to run inside the debugger with Corsair state rewound to any computation point. This is the same chip Ignition (Infinity’s bringup agent) took from first hardware access to tensor-parallel matmuls across 32 compute units in 10 hours and to three frontier models end-to-end in 10 days, now live as the Infinity d-Matrix Cloud.

Responsibilities

• Continuous checkpointing and restore: Capture full program state at varying granularity, roll back to any saved point, and build reverse-execution machinery (invert computations where possible, replay forward from checkpoints otherwise).
• Breakpoints and state inspection: Halt computations to surface every variable, its value, and physical location, with metadata tying values back to their nodes in the inference computation graph.
• In-debugger optimization: Enable changes to inference passes to be applied, measured, and ablated in place, running experiments at checkpoint speed instead of rebuild speed.
• Parallel debugging: Capture and reproduce interactions between parallel execution units (e.g., gangs) to debug race conditions where single-threaded debuggers fail.
• Differential debugging: Diff suspect inference passes against known-good references to localize numerical or structural divergences automatically.
• Breadth across silicon: Extend debugger capabilities to diverse hardware, including AI accelerators and mobile SoCs like Snapdragon/Hexagon DSP, running on real silicon.
• The Corsair build: Add real breakpoints and continuous checkpointing to the existing d-Matrix debugger, enabling AMPs to run inside it with state restored to intermediate points.
• Agent-facing interface: Expose debugger functionality so agents can pause, read values/locations, and disprove hypotheses through the debugger (not throwaway scripts), accelerating bug resolution.

Requirements

• Genuine debugging depth: Experience building debuggers or relying on them intensely, ideally tracking bugs down to silicon.
• Program state on hardware: Understanding of memory models, execution ordering, and the costs of capturing/restoring state on massively parallel machines.
• Time-travel/record-replay: Experience with time-travel, record-replay, or checkpoint-restore systems (or a strong desire to build one).
• Agent-first tools: Instinct for designing interfaces optimized for agents, not just human users.
• Languages: Fluency in Python and a systems language (Rust, C, or C++).

Nice to have

• Built or worked on time-travel/record-replay debuggers (e.g., rr, GDB reverse debugging, WinDbg Time Travel).
• Firmware, bare-metal, on-chip debug, or JTAG experience.
• Worked on AI accelerators or mobile SoCs (e.g., Snapdragon/Hexagon DSP).
• Hunted numerical bugs in ML (e.g., NaN-at-layer-forty issues).
• Familiarity with checkpoint-and-restore systems (e.g., CRIU).

About the company

Infinity is an early-stage AI infrastructure research company building the software layer to make non-NVIDIA chips competitive for AI inference. Instead of relying on scarce human kernel engineers, Infinity uses AI to automatically generate, test, and optimize low-level code for chip efficiency. The company has signed or is negotiating design partnerships with d-Matrix, AMD, AWS Trainium, Microsoft (Maia and Nexus), Qualcomm, and others. Founded by Jeremy Nixon (former Google Brain; co-founder of AGI House with Andrej Karpathy), Infinity has raised $15M from investors including the founder of Intercom, the VP of AI at AMD, and the founder of MLCommons. Headquarters are in San Francisco.