Three Bugs That Weren't

A lot of the debugging on this project goes on the machine I wrote. A surprising amount of the rest goes on things that were never wrong: a guest doing exactly what it was designed to do, firmware faulting on purpose, and a tidy diagnosis the machine itself contradicted. Each one looked exactly like a real bug right up until I asked a different question, which is why they stuck with me more than the real defects did.

A hang that was a question

With the disk-I/O defect fixed, an OpenBSD 6.9 install ran clean twice over, a fresh disk image each time, both distribution sets to 100 percent, through to the installer’s congratulations and a clean shutdown. Then I booted the installed kernel standalone for the first time, and it printed:

boot device: 'unknown' unrecognized.

and went silent. No further console output, ever, in that state.

What followed was careful and produced a precise wrong answer. I sampled the processor’s registers over and over across a second, spaced out on purpose so I would not catch a fast loop by luck. Every sample showed the same program counter, no variance at all. Resolved against the installed kernel’s own symbols, it was the second instruction of the crosstalk bus driver’s byte-read accessor, the load itself, not a branch. Disassembly ruled out a guest-side loop: that accessor is a four-instruction leaf with nothing to loop on, and its caller, the console register reader, calls it exactly once. The SCSI fix could not be the cause, because those commits touch only the adapter model, nothing in the I/O controller or console or bridge or processor. And it could not be a regression in something long-tested, because this was the first time the installed kernel had ever booted standalone. Every prior success had used the install ramdisk, whose root filesystem is baked into the kernel image.

All of that is correct. The conclusion I drew from it was wrong. The guest was not stuck. It was asking a question. A kernel started directly, with no firmware-supplied boot-device string, cannot know where it booted from, so it prints exactly that line, prompts for a root device, and waits. It is documented OpenBSD behaviour, predicted by its own source, and this project’s own notes had captured and explained the exact string months earlier. The trouble was that the harness driving the boot only ever watched for a login prompt, and a machine waiting at an unanswered prompt and a machine wedged on a lost completion produce the same silence. Having just spent days on a real hang is exactly the state in which the next silence reads as another one.

The discriminator was cheap and there the whole time: what the guest was waiting on. It is the same discipline as reading the first exception in a fault loop instead of the state at the wedge, and I missed it for the same reason people usually do.

There is a smaller thing tangled in this one, with a one-line fix. My repeated checks of the live transcript kept landing just short of the prompt, because a prompt has no trailing newline. It is waiting for one, which is the whole reason a prompt does not end its own line. Line-oriented and byte-truncated views both hide it the same way, not by cutting content but by cutting exactly the content that would have told me I was looking at a question rather than a finished statement. Switching to a display that marks line ends made the unterminated line visible as text instead of absent, and cost nothing. The harness fix was an explicit answer for the root-device prompt plus a bare-newline fallback for whatever came next, and the fallback earned itself immediately: the very next prompt asked for a swap device. It also once nudged a slow-printing startup line that happens to end in a colon, harmlessly, and that asymmetry, between wrongly answering a real prompt and harmlessly nudging a non-prompt, is why I left it broad rather than tuning it tighter.

Buried in the same investigation is the mistake I would most want back, because I published it before catching it. I had claimed the emulator’s monitor could read the guest’s target address cleanly while the guest’s own load never retired, which would mean the two access paths disagreed and would point at a real defect in the device. I had even resolved the address against a memory-map dump showing the right serial-port region. It was the wrong address. The value I was reading sat in node memory, not in the device, and reading node memory before anything has written it returns zeros no matter what the I/O controller is doing. The disparity was an artifact of comparing a real device value against a read of the wrong location. The guest’s actual target, through this machine’s crosstalk translation for the bridge’s widget, is a completely different address, and reading that returns a real, valid line-status value, which I verified against the still-live instance before I corrected the paragraph in place. The device answers correctly. There was no bug. When two access paths appear to disagree, the thing to do is print the address each one actually used. Both my reads were honest. Only one of them was of the thing I was arguing about.

Firmware that faults on purpose

The project’s other track loads the real, extracted SGI firmware image and tries to get it to its diagnostic prompt. Loaded correctly, it runs into C code in its own main routine, drives real traffic on the on-board serial bus, and then hits a cascading data-bus error. Following the rule that the last exception in a fault loop is the least informative one, I went for the first exception of the cascade, deep in a long trace. The emulator’s own output there was:

sgi-bridge: write of register 0x7c which this model does not implement; raising a bus
error rather than answering
mips: store of 4 bytes at 0x920000000800007c (physical 0x2000800007c) reached no
device; raising a bus error

A four-byte write to offset 0x7c in the bridge’s widget space. The obvious reading is that I had missed a register. I had not. This project’s own bridge datasheet has its register map jumping from 0x70 straight to 0x80, and SGI’s published chip specification has the identical gap, so 0x7c is reserved. The same specification says what real silicon does with a reserved offset, in one sentence: any access to an undefined address area raises an address error. So real hardware faults here too, and the model’s policy of bus-erroring on an unimplemented bridge register matches the documented behaviour exactly.

Which turns the question around. Why is the firmware writing to a reserved offset? Because it is a deliberate self-test, and I could see the whole shape of it in the trace: the firmware writes the sentinel 0xdeadbeef to that offset to provoke the chip’s address-error detection, then reads the bridge’s interrupt status register to confirm the invalid access was reported. The test is the test for that documented behaviour, and every axis matches, the offset, the sentinel, and the mechanism. It runs as part of the firmware’s widget discovery, unconditionally, not behind any operator command.

I did want to be sure the binary I was running actually executed that write, rather than my inferring it, because the check is not present in every firmware build. The trace settled it. Bounding it to a narrow window around the fault, the store’s destination decodes to 0x7c, and the standard two-instruction constant idiom right before it leaves the sentinel in the register being stored. This fork boots the stock, unmodified firmware, so a store that actually executes is a store that is in the shipped binary. The execution is the proof, and I did not need to date the image.

So there is no bridge-model fix to make. What is still open is one layer up, whether the firmware’s own fault-recovery path takes the branch it registered before the write, and a narrower question about one field in the routine that dumps bridge error state. That is where the diagnosis goes next.

The tidy theory this replaced

Before any of that, this same track had a root cause that explained everything: a translation fault at a named flash offset, routing through a global-pointer-relative load into the node controller’s halt routine, bypassing the normal boot and entering the diagnostic loop with a bad status value. It accounted for the silent console, a missing initialisation bracket, and the fault-handler entry, all at once. It was wrong, and the way it was wrong is the point.

The firmware image is a container, not raw code. It carries a header and a single segment starting 4 KB in, which I confirmed by comparing the payload byte for byte against the image rather than inferring it from matching sizes. The emulator had been loading the container header as the reset vector’s first instructions, which is exactly why it produced a reserved-instruction fault at the offset that sits in the middle of the container’s magic string. Everything I had measured downstream of that was describing a machine executing the wrong bytes. The translation fault does not reproduce at all once the image loads correctly. The reading that the firmware’s configuration block was a zero placeholder had been reading the container header. At the right offset it holds a real mode word, the configuration block’s own magic tag, and plausible processor and node-controller clock frequencies for a 180 MHz machine. And the halt hypothesis is not merely withdrawn, it is contradicted: the machine runs tens of thousands more logged lines after the point where the halt was supposed to have stopped it, including serial-bus traffic the firmware runs after the check the theory turned on. The halt routine even carries a source comment saying it is never reached, and a boot that calls it and then executes tens of thousands more instructions did not halt.

I keep that distinction on purpose. Withdrawn means the evidence turned out to be about the wrong subject, and the question reopens if better evidence arrives. Contradicted means a measurement says otherwise, and it does not reopen. The document that carried the old account opens with a withdrawal banner now and keeps the old reasoning beneath it, labelled, because I acted on it for a day and it explains work that would otherwise look unmotivated. Its own summary is the durable part: a wrong model fabricates evidence about the thing it models. Every trace was real, every exception was really taken, and the analysis was careful. What was wrong was the subject. The tell, in hindsight, is that the account explained everything at once, and a theory that tidy is generated by the bug rather than constrained by the hardware.