Five Reasons the Origin 200's Ethernet Stayed Dead

An OpenBSD 6.9 GENERIC-IP27 system, booted from its own disk on the emulated Origin 200 all the way to an authenticated root login, now takes a DHCP lease on the I/O controller’s ethernet and moves a file in each direction. The lease and the transfers are the first real network traffic this machine has carried.

Before that, the interface was dead. Not slow, not lossy, dead in both directions, for months of project time. What made it expensive to fix is that it was dead for five separate reasons, and any one of them alone was enough. Fixing four of the five would have produced exactly the same silence as fixing none, so for a long stretch nothing I did to it changed anything I could see, which is a demoralising way to work.

The milestone rests on a scripted session against the installed system. It asks for a lease, fetches a file from the host, sends one back, and accepts an inbound connection to a listener running inside the guest. Each payload is 65,536 bytes, and each leg is checked by digest and by byte count independently on both sides. The four DHCP handshake messages, discover, offer, request, and ack, are counted out of a packet dump of the wire rather than read from the client’s own output, and that distinction turned out to matter more than once: it is the difference between the guest believing it has a lease and a lease having actually happened. I ran the whole session three times and it passed every leg every time, and someone else reproduced it on a later build under their own conditions.

Here are the five.

The first is the one I would send anyone to read. The transmit walk erased the interrupt for the reply it had just caused. QEMU’s user-mode networking builds its answer to a request inside the same send call that carried the request, so the reply re-enters the same device a few frames below the guest’s doorbell write, lands in the receive ring, and raises the receive interrupt. The transmit walk had copied the interrupt-status register into a local before sending and wrote that local back when it finished, which erased the bit the reply had just set. Every answer arrived inside a transmit, so every answer lost its interrupt. From the guest’s side that looked baffling: dhclient sent seventeen discovers and never saw an offer, while twelve byte-perfect offers sat in the ring behind it, and the guest read the status register thirty-eight times across the run without once catching the receive bit.

The second: frames were leaving the machine malformed, because the engine miscounted a descriptor. A transmit descriptor is a command word, a length word, two eight-byte buffer pointers, and then its inline data. The engine counted one pointer where there are two, and read the inline area from offset 16 instead of 24, so every frame went out with the second pointer stuck in front of it and its own last eight bytes missing. On the wire that reads as eight leading zeroes, a destination address of all zeroes, and an ethertype of 0xffff. Nothing parses that, so nothing ever replied. Both numbers that fix it, the offset and the 104-byte inline area, fall out of the two guest drivers independently, one computing them and the other declaring an array of exactly that size, which is a reassuring way to have a constant confirmed.

The third: the engine was raising the wrong interrupt source. It announced every delivered frame on the receive-threshold source and never raised the receive timer at all. The timer is the per-frame source. Both drivers arm it and load its idle timeout with zero, and both say in their own comments that zero means interrupt immediately on reception. The threshold is a low-watermark warning on how many receive buffers are still free, which is a supply warning, not a delivery notice. OpenBSD survived the substitution only because it happens to enable both sources. Linux enables the timer and nothing else, so on this model a Linux guest would have found status and enable disagreeing on every frame it received. The requirement had been written down before the fix: the datasheet rows that grant this freedom were rewritten first, and the model was corrected to match afterwards.

The fourth: a write to the interrupt-enable register did not re-derive the line. The interrupt line is level-triggered from status and enable together, so the enable register decides it exactly as much as the status register does. But only the acknowledge path, a delivery and a transmit completion, recomputed the line. An enable write fell through to plain storage, so a source that had set while the enable was clear could not raise the line when the guest later enabled it. That is not a corner case, it is on every interface bring-up: OpenBSD stops the interface, writes the receive enables, and writes this register last, and a frame delivered in that gap lands in exactly the window.

The fifth is the least glamorous and the one with the widest reach. The chip could not be given a network at all unless the Origin’s own board code reached in and did it. The ethernet is a child device that held the network properties itself, and nothing outside could name them. That made the interface unreachable on any other machine, which matters beyond test convenience: this part shipped as a plain PCI card as well as soldered onto a base I/O board, and a card nobody can attach a network to is not a model of the card. The properties are aliased onto the chip now, so the ordinary -device sgi-ioc3,netdev=... works.

There is a condition that travels with all of this, and it is not a detail to drop: every one of those passing runs was at a 128 MiB fitting. At 256 MiB the installed system does not reach a login at all, and the reason has nothing to do with the network. It is a memory-map defect. A declared memory bank is quartered across four slots on a 128 MB stride, so what the guest is told exists and what the machine actually backs come apart by an amount that depends on the fitted total:

128 MB is the only fitting that clears both defects, which is why I ran the repeatability series there. An earlier series at 480 MiB had completed the inbound leg once in four attempts, and I kept that record rather than replacing it, because it is the only account I have of how the failure looked: every failure struck after the listener had started, and none of them was a delivery failure. So the pass rate improved because the machine was fitted correctly, not because the network was repaired. That sentence is in the review, and it is the kind of distinction that decides whether a milestone can honestly be declared. The network worked at 480 MiB too. What did not work was the machine underneath it. That memory map is the project’s wall now.

The measuring apparatus itself needed strengthening before I could believe it, and two things were wrong with it, both caught by the people who had built it. The transfer check had reported a pass on two empty files: its author had changed what the payload was made of without re-asking what the check would still catch, and a digest comparison passes perfectly when both sides are empty. The note written up afterwards refuses the easy lesson. The thing worth carrying is not “also check for empty files,” it is that the property being checked was weaker than the property that had to hold. It checks length as well as digest now. The other: the regression suite did not cover the interrupt source Linux depends on. Both of its assertions about a delivered frame had been written against the guest that happened to be in front of the author, so a regression that left Linux unable to receive would have sailed through the whole suite. That is the same defect this milestone had just spent a session fixing, latent in the very test written to prevent its return. It is guarded now, and the guard was checked the only way worth trusting, by running it against an engine with the fault still in: it fails, on the exact bit.

An independent read of the model then turned up eight more things, one of them serious and none of them blocking. A guest could hang the emulator with two register writes. Both ring indices are byte offsets that can only legitimately hold a descriptor number shifted left by seven, but the consume index fell through to plain storage, so a guest could write an unaligned value and leave the transmit walk stepping 0, 128, 256 and never landing on the index that would stop it. The register write never returns. Neither driver we run does this, but it is reachable from unprivileged code inside a guest, so it was fixed rather than filed.

None of this buys as much as it sounds like. Serving a boot image is a longer reach than moving a file, so the network-boot route is nearer than it was, not open. Which route the firmware takes by default, and what has to be set to send it at the network, is still unestablished. Linux networking is unblocked and unexercised, which are different states: the model raises the source Linux dispatches on and the suite guards it, but no Linux guest has booted since the fix landed, and the only thing that closes that gap is a boot. And the machine still stalls where it always stalled, on the node controller’s one-wire bus, with the register modelled and nothing built behind it yet.