The Heartbeat Light That Masked Another Processor

Two emulated Origin 300 chassis, joined by one NUMAlink cable, came up as a single machine. The firmware found 8 CPUs on 2 nodes, passed its diagnostics and handed over to IRIX, and IRIX got most of the way through starting its services before it stopped at a line that looks like a networking problem:

NFS server localhost (nsd) not responding

It was not a networking problem. Underneath it the kernel was waiting on one processor that would never answer.

A cross-call nobody picked up

When a multiprocessor kernel changes a page mapping it has to tell every other processor to drop its cached copy of the old one, a TLB shootdown. It sends each processor an interrupt and waits until they have all acknowledged. One processor on this machine never acknowledged, because its interrupts were masked, and it had not masked them itself.

A convenience that was not neutral

The Bedrock node controller exposes a small register window per processor. Our model knew one device inside it, the console path to the L1 controller, and forwarded every other address in the window straight through to the processor-interface registers, which hold each processor’s interrupt masks and its cross-call bits. The model’s own address map described that forwarding as an emulator convenience. Nothing had visibly broken, so nothing had shown it was wrong.

A public Linux header for the Itanium machines SGI later built around the same node controller says what is really there: one LED register per processor, a status light for each CPU, at offset 0xc0 in that window. The firmware writes progress codes to it. IRIX writes a heartbeat to it, a periodic tick meant for a status light.

Through the model’s forwarding, that heartbeat landed on the interrupt-mask register of a different processor. A traced four-processor boot showed it plainly: each heartbeat from one processor rewrote the next processor’s interrupt mask, and the same stores raised and cleared cross-call bits hundreds of times. On smaller populations the machine got away with it. On eight, it took only one heartbeat arriving at the wrong moment to leave a processor masked while a shootdown waited for it, and the whole machine stopped behind the one processor that could not hear.

Fixing the window, not the symptom

The fix gives the window its LED registers at their published offsets, so a heartbeat is a heartbeat. The rest of the window, where nobody has yet established what lives, is no longer forwarded anywhere: it is an unmodelled stand-in that logs every access, so an unknown write shows up in a log instead of turning into a live register somewhere else. With that in place, IRIX boots across both chassis to a login, and its inventory lists eight R14000 processors and a gigabyte of memory.

The lesson is about unknown address space. Aliasing it onto a register file you do understand feels harmless, because the aliased registers are real and well tested. But every store the guest makes to the unknown part then lands on something that matters, and the guest has good reasons to write there that you have not discovered yet. An unknown region is safer answered as unknown, loudly, than borrowed from a neighbour.