Bringing Up a Second Processor on the Origin 200

The stock SGI PROM now runs past its own diagnostics and loads sash, the standalone shell that installs and boots IRIX and the thing I’ve been pointing this project at since the firmware first printed a banner. Both of the PROM releases we support reach their matching sash, and they do it on two processors.

I expected the work of getting the second processor going to be about the processor. It was mostly about the emulator’s scheduler, which until this point I’d had no reason to look at, and which it turned out the guest could see.

The system controller arbitrates between processors with a signal, and the firmware’s side of that exchange has a hold: it asserts, and keeps asserting, for 250 microseconds. The controller rotates its own signal every 300 microseconds. Those two numbers are the whole contract, and on real hardware two processors sit inside it without any trouble.

QEMU runs several guest processors on one host thread by round-robin, switching between them on a timer, and that timer’s default period is 100 milliseconds. That is four hundred times the firmware’s hold. So a virtual processor would begin its 250-microsecond assert, get switched away long before finishing it, and the controller would rotate the signal underneath it. When the processor resumed, the thing it had been holding for had moved. Nothing about that looks like a timing bug from the guest’s side. It looks like a machine whose system controller doesn’t answer, which is a much worse place to start looking.

The period had been tried before I got to it, at 10 milliseconds and again at 1 millisecond, and both times the boot came back looking exactly the same. It’s easy to read that as “the period isn’t the problem,” and for a while I did. But neither trial had actually tested the boundary, because both values still sat far above the device timer. Ten milliseconds is still forty times the firmware’s hold. The experiment had been run twice, at two different values, and both times the answer was the same for a reason that had nothing to do with the value, which is the exact shape that talks you out of the right idea.

At 10 microseconds it changed. All four processors completed holds of 250 to 254 microseconds against the same modelled device timing, and the contract was satisfied. That told me the mechanism was right, but 10 microseconds is also about a hundred thousand scheduler timers a second, and the machine drowned in them. A two-processor run at that period was still short of I/O completion after twenty minutes, so the value that proved the mechanism was plainly not the value to keep.

100 microseconds still sits inside the 300-microsecond arbitration window, and at that period the same run reached I/O completion, relocation, memory setup, and both processors’ interrupt test in about two minutes. That is what the launcher selects now. The generic period is an exposed parameter with its own regression test, and the 100-millisecond default is preserved for every other machine. This one asks for something unusual, and I would rather it say so than have the emulator quietly change for everybody.

There was a second clock problem hiding underneath the first, pointing the other way. Some of this work runs under fixed instruction counting rather than wall-clock device timing, because a deterministic clock is what makes a multiprocessor test reproducible. That choice produced its own false picture. A single-processor control run under fixed instruction counting reached the relocated virtual mapping without ever writing the RAM pages behind it, executed the zero-filled memory it found there, and dropped into the PROM’s power-on diagnostic mode. Read cold, that looks like a memory-aliasing bug, or a NUMA bug, or a relocation bug, three plausible and expensive things to go chasing. The same command under wall-clock device timing populated every copy page and carried on. The failure was an artifact of the scheduler clock, and said nothing about the memory model at all.

That is the project’s oldest lesson arriving in new clothes. A wrong model fabricates evidence about the very thing it models, and the evidence looks like the subject rather than like the model. The only tell I had was that the fault seemed too structural for how recently it had appeared, which is a weak thing to steer by, but it was the one that turned out to matter.

And through part of this the progress-code listener reported nothing at all, which reads as a machine that has stopped. The machine had not stopped. The Origin 200 reports its early progress at a different pair of addresses from its larger siblings, so the listener was watching the wrong two registers while the firmware wrote diligently to the right ones. The sequence had been intact the whole time. The Hub’s indicator decode gained the second pair, and the boot now observes all thirty-three of the progress codes it is meant to.

That is three separate things in one milestone, the arbitration hold, the fixed-icount artifact, and the silent listener, and all three were the measuring apparatus rather than the machine. I think that is worth counting, because the ratio usually isn’t that high, and a run of it is exactly how a project talks itself into believing the machine is worse than it is.

Where that leaves things: two processors reach sash on one node. Under fixed instruction counting, two-node Linux brings up four processors across two 64 MiB nodes and both crosstalk endpoints, and a process pinned to a processor on the first node can move a page to the second, ask where that page lives, and read its contents back. That is the ccNUMA property this whole machine is shaped around, exercised for the first time on something I built.

The four-processor firmware boot has moved on since, and it moved into the same territory this note is about. With the system-controller fix in, the four-processor boot reaches the CrayLink global barrier and wedges, and the mechanism is a scheduling one again. A processor slice bounded-polls its Hub pending bank waiting for a peer’s edge, exhausts the firmware’s loop inside a single round-robin slice, and the peer that would produce the edge never gets scheduled to produce it. An ordered handoff, plus a Hub assist that yields only when a re-read comes back unchanged, carries a deterministic regression that fails without it and passes with it. That acceptance run is not captured, and the honest reading of the attempts is the one in the project’s own notes: ten boots that never reached the barrier phase, and archived serials that point at a single timing-sensitive defect wearing three faces rather than at a boot lottery. A later run turned up a further shape, the second node’s firmware-copy verify reading zeroes, and how that one relates to the first is still open.

The IRIX half of the two-node milestone is untouched. There are a couple of four-processor observations in the history, one timeout with no serial output and one shell reached with four processors, but neither is a retained check and neither is evidence.