Debugging a Lost Processor Pair on the Origin 2000

On a fraction of eight-processor boots, a whole node came up short. The two processors of one node would meet at bring-up, one of them would go on without the other, and the master would disable both. The machine that reached a login did so with six processors, not eight, and only some of the time.

The handshake and its one-millisecond margin

The two processors of a node rendezvous while the machine comes up. The second reaches its comparator match and takes an alive hold, parked, waiting for the first to set a bit in a node scratch register and raise an interrupt. On the real machine the second processor spends about a millisecond on the real-time counter between its match and its read of that bit. The hold is there to bridge exactly that millisecond: it keeps the second processor parked past its match when the first is briefly late, so the two still meet.

What ended the hold too early

In the model the hold ended whenever work was queued on the held processor, on the same footing as a stop or a deliverable interrupt. The trouble is that work queued on a processor is mostly not the guest’s. A memory-map change anywhere in the machine queues a translation commit on every processor, and the stock firmware makes dozens of those while a node’s two processors meet, as each processor arms and leaves its private firmware-data window and aims its wide address windows. So the second processor’s alive hold ended on that unrelated bookkeeping, before the signal, in every traced boot that later lost the node.

Once the hold ended early, the second processor read the scratch bit, found it clear, and went on without its partner. It set the next interrupt one exchange too soon, withdrew its post, and seconds later switched its text mapping to a firmware copy nobody had written. The master found the node in that state and disabled both of its processors. A hold that yields to unrelated work bridges nothing on a busy machine, which is precisely when the bridge is needed.

A hold that outlasts the bookkeeping

The fix lets the hold outlast queued work. The work runs, and the hold resumes behind it by re-queueing itself on the same processor, so the queued commit runs first and the hold is taken up again before the processor executes another instruction. A stop or a genuinely deliverable interrupt still ends it, as they should. The millisecond the firmware allows is now bridged across a host thread that is briefly off its processor, which is the real event the hold was written for.

One smaller change went in alongside it. The hold’s end-trace had said “preempted” for a stop, for queued work and for a deliverable interrupt all alike, so a boot trace that lost a node could not say what had ended the hold. It now names which, so the next such trace answers the question rather than leaving it to be guessed. With the hold bridging what it was meant to bridge, the node keeps both its processors, and the machine stopped losing a pair this way.