Origin 200: The Road to a Root Shell

Two days of wall-clock time, a fork of QEMU, and a ladder of milestones. This is the route from a QEMU monitor prompt on an empty machine to an authenticated root shell on OpenBSD, booted off a SCSI disk through an emulated 1997 SGI I/O fabric.

It was made almost entirely of walls. That is the interesting part.

A machine with nothing in it

The first commit that mattered gave the fork a machine type, -M origin200, that started and reached the QEMU monitor. Nothing was in it. The processor model came next: a new processor definition plus small deltas in the MIPS target, with the R10000’s own quirks, its configuration register layout, an internal-only second configuration register the guest never reads, a masked branch-diagnostic register where only the prediction-mode field is writable, a 64-entry translation buffer, a cycle counter that ticks at half the processor clock, 40-bit physical addressing and 44-bit virtual addressing. The bar was a suite of bare-metal test ROMs, and the model was then handed to an independent second-opinion review that found five real defects in it.

Then the bare machine: node memory at the bottom of the node’s window, and the boot flash as one region aliased into both of its views, the reset vector’s view through the compatibility segment, and the Hub’s special boot window. Its test asserts the firmware configuration block through both flash views, all ten entry points, the flash rejecting writes without faulting, and memory still answering underneath the flash overlay.

Printing before any device exists

The next milestone is the one that reads oddly on the roadmap and is the best decision in it: OpenBSD boots and prints, with no device model in the machine at all.

OpenBSD reaches its first console output by calling the firmware’s output routine, not by touching a serial chip. So I synthesize the artifacts the real firmware leaves behind, a directory at a fixed offset in node memory that locates everything else, the board and component graph the guest reads its own hardware out of, a global data area with the node table, the callback vector table, and a set of firmware environment variables, and implement the callbacks as C handlers behind a trap region, where a load from an address is a firmware call and the operation is the offset.

That got the guest through every synthesized structure end to end and stopped it at its first access to real hardware: a byte write to the console serial port at crosstalk widget 8, offset 0x62_0178 plus three, setting the line to eight data bits, no parity, one stop bit. Three printed lines, and every firmware structure exercised with nothing to blame but the firmware.

The milestone was renamed three times because the guest kept falsifying its name. “The OpenBSD banner” turned out to need a serial chip. So did “device-free console switch”. And the console-switch line itself turned out to be printed by the console the guest had just selected. A name describing what a milestone demonstrates survives whatever the guest turns out to touch next. A name describing the work does not.

One diagnosis from that stretch is worth carrying on its own. The faulting routine in the first console-switch attempt was reported as a crossbar write, and got relayed to three people as “the crossbar”. It is the crosstalk bus driver’s byte accessor, which a child device calls to reach its own registers. The name identifies the bus. Decoding the address identified the target: the console serial port, three bytes into the base my own firmware had advertised, with the value that sets eight-none-one. A whole increment was about to be planned around modelling a crossbar the guest had never touched. What caught it was a teammate asking for the raw offset.

The banner, and the console address that was wrong three times

Adding the I/O controller’s first serial port took the guest from three printed lines to thirty-one. It moves its console off the firmware routine onto the real chip inside its console-initialisation routine, and its console table has no firmware entry to fall back on, so this needed one serial port and nothing else. The banner followed from the same chip with no second device.

OpenBSD 6.9 (GENERIC-IP27) #0: Thu Apr 22 03:53:59 UTC 2021
real mem = 268435456 (256MB)
mainbus0 at root: Origin 200
cpu0 at mainbus0 nasid 0: MIPS R10000 CPU rev 0.0 180 MHz, R10000 FPU rev 0.0
cpu0: cache L1-I 32KB D 32KB 2 way, L2 1024KB 2 way

The guest names the machine, the processor, the clock and the caches, all read out of the configuration graph I synthesized. What that proves is that the encoding round-trips, offsets, widths and the enabled flag are right, and a kernel reading a component I had not enabled would print zeros. It does not prove the values match real hardware, because the values are mine. Change them in the model and the machine together and the guest prints the new ones and looks equally correct.

There is exactly one exception, and it is the only field on that screen checked against anything outside the project: the processor identification number. The guest turns it into the string R10000 using its own table, which I did not write and cannot edit.

The console’s own address was wrong three times, and each wrong value passed a test. The address has three independent components and the guest announces a decision about only one of them. The first wrong value was 0x178, the serial port’s offset inside the controller’s serial block, which shares its low twelve bits with the correct 0x20178, and twelve bits is exactly the width of the check a guest uses to pick which controller generation it is talking to. So a machine carrying 0x178 identifies its console correctly and then writes to an address nothing answers. Correcting it to twenty bits fixed that and was still wrong: the value had dropped a bus term living in the other half of the guest’s split, where an offset check cannot reach.

Two things stuck with me from that. I stopped trusting a check that passes a value I already know is wrong: the twelve-bit discriminator here would have happily certified the broken base it existed to catch, because a wrong value sharing its low twelve bits with the right one sails straight through. And where I could compare the whole address instead of some property of it, I started doing that, because a whole-value comparison has nothing weaker left to pass on.

The evidence that would have caught the first one had been in the project’s own reference library the whole time: a real machine’s captured console address, recorded in full, and filed under the claim that it “has 0x178 in its low bits”. True, weaker, and precisely the reduction that produced the bug. Anyone checking the datasheet against the library found agreement.

Also from that milestone, and cheap to get wrong: the two serial ports sit one byte apart inside a window whose neighbours are thirty-two bits wide, so the ports have to declare their access width rather than inherit a default. Assume symmetry there and you get a console that emits nothing.

Autoconf, and two gaps behind one symptom

Getting from a banner to the guest enumerating its own hardware needed the crossbar, the bridge, a PCI bus and a disk adapter. The first attempt modelled the crossbar, which is the device the guest’s panic named, and the panic did not move.

The reason is that the synthesized firmware had never programmed the address-translation entry that makes the guest’s probe reach a crossbar in the first place. Two gaps, one symptom, and the two were not symmetrical. The firmware’s translation-entry write was specified, small and merely unimplemented. The crossbar was a full datasheet with nothing modelled at all.

The cheap gap was not the visible one, so the natural order, model the device the guest asked for, ends with the boot stopped in the same place and more work done. Doing the redirect first is also diagnostic: with it programmed and no crossbar, the probe reaches a widget that does not answer, which is a different and more informative failure than a probe that reaches nothing. After that I tried to order the work so each next failure would tell me more than the one before it, and let cost break the ties.

The half no machine could meet

The autoconf milestone was deliberately written with two halves: the guest must attach the crossbar, the bridge, the PCI bus and the serial chip, reach its root-device prompt, and take at least one real device interrupt through the full chain.

Only the first half was met, and the second was not merely unmet, it was impossible. A search across every one of the machine’s own files found exactly one interrupt call, the one that initialises the processor’s own lines. The Hub declared no output line. The console chip contained no reference to an interrupt at all. The console is polled, so the interrupt half was untested rather than passed, and the difference is invisible in a boot transcript. The milestone looked like it validated something it never touched.

Splitting the milestone’s wording is what stopped it being recorded as complete. Closing it took a bare-metal test ROM that drives a PCI device’s pin and follows the whole path: the Bridge composes a packet, the fabric carries it, the Hub decodes it, and the processor’s cause register shows the line. The ROM asserts that the Hub holds the interrupt number the Bridge sent rather than merely some bit, and that a packet aimed at another node arrives nowhere, so the node field is proven used rather than present.

Walls, one register at a time

With the bridge fitted, the guest walked into the real fabric and the walls came one at a time. Each was a register block the I/O controller presents and the guest reads during attach: a status register in the controller’s main window, the ethernet block, the serial port’s own DMA control block, the PS/2 keyboard-controller block, and the 128 KB of static RAM the ethernet engine buffers into. That last one produces a line the guest prints from a size probe:

iec0 at ioc0: 128KB SSRAM, address ff:ff:ff:ff:ff:ff

The size is the same one all thirty-one captures from real Origin 200 hardware print. The address is all ones because the one-wire part that would supply a real ethernet address is deliberately not modelled yet, and OpenBSD’s fallback fills the field with ones and prints it.

Then the real-time clock, which the guest reaches over a byte-wide expansion bus behind the I/O controller. With the clock modelled and the bridge fitted by default, OpenBSD reached its root-device prompt on the ordinary machine.

The clock also produced two lessons that had nothing to do with clocks. Its register byte order looked like a disagreement between guests and turned out to be an artifact of the reference corpus: two Linux checkouts straddling a 2020 fix to how that architecture does byte-wide register access, with all three guest drivers agreeing on the wire address once the right tree was read. And a test that failed intermittently by exactly one second turned out to be a real defect in the device rather than a race in the test: the clock measured its offset against the host clock and applied it to whichever clock the machine had been told to use, so a guest that set the time read back something else whenever the two differed. No booting guest was affected, because the default machine uses the host clock on both paths. The fix made a rare failure deterministic.

The disk, and a real installer

Two SCSI adapters sit on that bus on a real Origin 200, the machine’s own hardware inventory names both, so the emulator fits two. They would not come up. Their mailbox registers read wrong after reset, which stops the driver dead.

The cause was in the Bridge: its per-device windows needed an address-only correction on narrow accesses to a little-endian device sitting on a big-endian machine. The fix scoped that correction by PCI vendor and device identity, exempting SGI’s own I/O controller, mirroring what OpenBSD’s bridge driver does with its own device hook. It went through a design review before it was written and an implementation review after it landed, with three separate mechanisms mutation-tested. A second, subtler bug came out of the same work: a value-side swap that made the mailbox test pass by luck, through a coincidental endianness agreement. The full suite caught it.

With both adapters attaching, the OpenBSD 6.9 install ramdisk ran its interactive installer against the emulated adapter and extracted the base and compiler sets onto a SCSI disk. Megabytes of real DMA write traffic through the modelled fabric, driven by a real vendor installer rather than by any test I could write.

It halted partway through, reproducibly, and that story is told in the hang that needed two writers.

The login

With the completion-erasure defect fixed, the installed kernel booted from the disk the installer had written. Every line below is verbatim from the committed transcript, and this is an excerpt with lines dropped: the copyright block, the memory summary, the clock, both bus attachments, each adapter’s NVRAM and firmware-revision lines, the one-wire bus, the interrupt assignments, the second serial port, the PS/2 controller, the ethernet interface and its transceiver, and the software-RAID layers are all in the full record. Marked because an unmarked cut reads exactly like a complete sequence, which has caught this project three times in its own reference library:

ARCS64 Firmware
Found SGI-IP27, setting up.
Machine is in M mode.
Initial setup done, switching console.
[... copyright block and memory summary dropped ...]
OpenBSD 6.9 (GENERIC-IP27) #0: Thu Apr 22 03:53:59 UTC 2021
mainbus0 at root: Origin 200
cpu0 at mainbus0 nasid 0: MIPS R10000 CPU rev 0.0 180 MHz, R10000 FPU rev 0.0
xbow0 at mainbus0 nasid 0: Bridge revision 3
xbridge0 at xbow0 widget 8: Bridge revision 3
xbpci0 at xbridge0 bus 0: 33 MHz PCI bus
pci0 at xbpci0 bus 0
qlw0 at pci0 dev 0 function 0 "QLogic ISP1020" rev 0x05: irq 0, xbow irq 62
sd0 at scsibus0 targ 1 lun 0: <QEMU, QEMU HARDDISK, 2.5+>
sd0: 2048MB, 512 bytes/sector, 4194304 sectors
qlw1 at pci0 dev 1 function 0 "QLogic ISP1020" rev 0x05: irq 1, xbow irq 61
ioc0 at pci0 dev 2 function 0 "SGI IOC3" rev 0x00
com0 at ioc0 base 0x20178: ns16550a, 16 byte fifo
com0: console
dsrtc0 at ioc0: DS1742W
root on sd0a swap on sd0b dump on sd0b
Automatic boot in progress: starting file system checks.
/dev/sd0a (18cc54198c86b2cb.a): file system is clean; not checking

and then, after the daemons come up:

login: root
Password:
OpenBSD 6.9 (GENERIC-IP27) #0: Thu Apr 22 03:53:59 UTC 2021

Welcome to OpenBSD: The proactively secure Unix-like operating system.
...
origami# id
uid=0(root) gid=0(wheel) groups=0(wheel), 2(kmem), 3(sys), 4(tty), 5(operator), 20(staff), 31(guest)
origami# uname -a
OpenBSD origami.my.domain 6.9 GENERIC-IP27#0 sgi
origami# df
Filesystem  512-blocks      Used     Avail Capacity  Mounted on
/dev/sd0a      3985660   1534604   2251776    41%    /
origami# halt -p
syncing disks... done
System Power Down.

The full transcript is committed as a tracked golden, with a provenance header stating what it is and the one edit made to it, a shell prompt-redraw escape sequence removed as terminal noise.

What that login is and is not

It is the installed kernel, not the install ramdisk. uname says 6.9 GENERIC-IP27, and the boot is a standalone boot of a system the emulator installed.

Root is a real filesystem on a real disk image the installer wrote through the modelled adapter. df reports 41% used off /dev/sd0a. It is not a synthetic image dropped in from the host.

It is an authenticated session, not a login banner. id returns root’s groups read out of the mounted filesystem, and the machine then halted cleanly.

The version is 6.9, and this is not the same guest as the banner work. The console, autoconf and root-prompt milestones were all reached with OpenBSD 7.9 GENERIC-IP27. The disk boot is OpenBSD 6.9 GENERIC-IP27, the last release before the platform was retired and the one with an official signed installer. Both are true, and a sentence that says only “OpenBSD” now spans two releases.

The root device was supplied at the prompt, not by firmware. This is a bare kernel start with no firmware-supplied boot-device string, so the kernel reports its boot device as unknown and asks, and the test harness answers sd0a and a bare newline. Those lines are left in the committed transcript rather than stripped, because what was sent is part of the proof. It belongs to the firmware-synthesis work, and the honest phrasing of the milestone is “booted to an authenticated root shell on the installed filesystem, with the root device supplied at the prompt”, not “installed and then rebooted autonomously into multiuser”, which is a stronger sentence than the record carries.

What the boot proves

It is worth being exact about what a boot like that proves, because it looks like more than it is. It shows that every structure the guest reads is reachable and parses: the parameter block, the firmware call vector, the output routine, the memory descriptors, the hardware directory, the configuration graph, the global data area, the boot-flash block, and every Hub register in the path. It does not show their contents are right. The console descriptor here parsed cleanly three times carrying three different values, two of them wrong. Only something that compares a value proves a value, which is why every structure in this machine carries its own register-level test instead of resting on the boot.

And a boot only ever tells me the guest did something on my machine, never that the guest does it. That second claim needs a source or a capture from real hardware, because everything I learn about the guest arrives through a machine I wrote, and my own defects and the guest’s behaviour come down the same wire.