QSOE v0.2-rc, in three release candidates

v0.1 was a singular machine. One userspace, booting on two kernels, at one console, on the board it was tuned for. Everything difficult about it was internal — the spawn-from-disk deadlock, the libc seam, getting one set of binaries to come up under both Skimmer and seL4. v0.2 is the cycle where that machine stopped being singular. A second machine appears on a wire; a second console appears on the screen; a second silicon target, the SpaceMiT K3, stops letting the code assume the machine underneath it. None of the three is finished, which is why v0.2 is a candidate and not a release. But the shape of each is in the tree now, tagged rc1 through rc3, and the through-line is the same one every time: a thing that used to be assumed unique — a node, a terminal, a connection id, a hart — has to be made to mean the right thing once there are two of it.

The userspace is still a single body. libc.so and quser are shared source; only the seam differs — taskman and a handful of kernel calls, per kernel. So most of what follows had to land twice and identically, because a session, a /dev/tty, or a COPY relocation that behaves differently under the two kernels is not portable, and portability across the two is the whole reason the seam exists. Two threads run underneath all three candidates and I have written about both before: the tree now builds end to end with qjmcc, the Johnson–Magnusson C compiler, kernel and userspace alike; and the K3 bring-up that started in v0.1's tail continues here, mostly as a sequence of places where real silicon contradicts something QEMU let stand. I will not re-cover either except where v0.2 depends on it. The current set is lq 0.17, nq 0.21, libc 0.10, quser 0.9.

v0.2-rc1

The headline is the first networking this system has ever had — not a stack, not sockets, but the wire beneath both. QSP carries QSOE's own message passing between machines, so that a MsgSend can eventually cross a segment without the application knowing. rc1 is the format and the discovery: frames ride raw Ethernet under a local-experimental EtherType with no IP and no routing layer, the header is a flat forty-eight bytes parseable by a few hundred bytes of code on a microcontroller, and station ids are globally unique and assigned by hand — they are printed on the front panel — which deletes the per-boundary node-descriptor translation QNX has to carry. devn-virtio serves /dev/net/en0, the manager reads the net's roster from /etc/qsp.conf, announces itself, learns who else is out there, and publishes them under /net, so that ls /net is the answer to "who is on the wire." The emulator grew a two-machine mode to develop against — a second NET=1 NODE=2 ./emu.sh in another terminal is a second machine on the same layer-2 segment, no hardware asked for. What is emphatically not here yet, and rc1 says so plainly, is circuits and transactions: two machines find each other and nothing more.

The kernel earned nq 0.19 for the two primitives a transparent relay will need. MsgReadv and MsgWritev had held a syscall number, a libc trampoline, and a public prototype since the message table was laid out, with no implementation behind them — and the dispatcher panics on an empty slot, so calling either took the machine down. They are filled now because they are exactly what lets a server reach into a sender parked mid-rendezvous: read its send buffer at an offset, scatter into its reply buffer at an offset, without completing the exchange. That is what will let a relay move a multi-megabyte MsgSend onto a wire one frame at a time rather than buffering the whole thing up to the transfer ceiling. Alongside them, the resource-manager framework's control() method was finally made reachable — the vtable slot had existed since the framework was written and the object model makes it normative, but no request decoded into it and libc offered no way to send one, so a device had no channel at all for answering a question about itself. A network driver knowing its own hardware address and having no way to say so is what made that intolerable.

For QSOE/L the rc1 story is shorter and more embarrassing: it boots again. taskman had never applied R_RISCV_COPY relocations, so a non-PIE executable that imports a data object from libc.so — stdin, stdout, stderr, __environ — read it as NULL, and the two resource servers that dereference such a pointer died on the way up. /usr never mounted and boot fell to the rescue shell. The fix taught the shared relocation walker to read the defining object as currently loaded rather than as it sits in its file, because for a pointer like stderr the file holds zero and the real value is written in at load time by that object's own relative relocation. That lifted QSOE/L to 0.15. The same two-day sweep turned up five under-declared build dependencies of one shape — a rule that looks current because its declared inputs are older than its output, while the real input is a whole subtree — each of which had at some point shipped a stale binary that built clean and behaved as though the code just written did not exist. Unglamorous, and worth an evening each; the tree is stricter for it.

v0.2-rc2

rc2 is where the emulator's charity runs out. Most of it is correctness that QEMU had been quietly covering for, surfaced the moment the code met real hardware or a real environment.

The sharpest case is on the K3. An NVMe controller read and wrote flawlessly and never once raised an interrupt: config space worked, BARs worked, DMA landed byte-perfect, the filesystem mounted — and every message-signalled interrupt went nowhere. The root port is a PCI-to-PCI bridge, and its prefetchable window came out of firmware covering the entire low four gigabytes, with the AIA interrupt files sitting inside it. A bridge forwards a transaction downstream when its address falls in one of its windows, regardless of which direction it arrived from, so every MSI-X the disk sent upstream landed in that window and was bounced back down as a peer-to-peer request nothing answered. What hid it is that this board's memory begins above four gigabytes: every DMA transfer was addressed above the window and sailed through, while every interrupt was addressed below it and was rejected. The four-gigabyte boundary had been sorting data from interrupts the whole time. The fix closes the two windows QSOE never allocates from, with the architected base-greater-than-limit encoding Linux uses on the same board — and it closed the same latent hole under the Ethernet and wireless controllers, neither yet asked for an interrupt.

The kernel owed three things, and rc2 is nq 0.20 for them. A thread that ended while attached to an interrupt was left in the kernel's attachment table, and the next fire panicked the machine with a schedule of a dead thread — which inverts the entire point of running drivers in user mode, since any process allowed to attach a source could halt the machine by attaching one and exiting. Teardown now happens at the moment the thread is doomed, before it is even marked a zombie. Second, posix_spawn's environment argument was parsed, validated, and then dropped on the floor, so every process on the system started with an empty environment and nothing said so; it was found while writing a sysinit fragment that exports the station id — login set USER and HOME, spawned a shell, and the shell had neither. Third, and subtler, a data object an executable copies out of libc.so was left as two objects that drifted apart the instant either side wrote: the program read its own .bss copy while the library wrote its own through the GOT. ELF's rule is that the copy becomes the definition and every other reference rebinds to it, including the library's own — the missing half of the relocation work rc1 began. __environ is where it showed, and it is why login could set a variable and then hand the child an environment that was still empty. Both kernels carried the bug, QSOE/L by one route and QSOE/N by another, and both are 0.16 and 0.20 respectively for the same rule.

The rest of rc2 is instrument and honesty. A driver aiming an MSI-X message needs the hart its interrupt source was bound to, and the only way to ask had been a blocking taskman round trip in a path a driver was just told not to block in; the kernel now returns the hart with the attach, and libc reads it back — while QSOE/L, where seL4 hands out an interrupt capability with no hart in it, answers UNKNOWN, which is deliberately not zero, because a caller that took a plausible zero at face value would target hart 0 and program an entry that looks correct and never delivers. slog grew a severity threshold tested at the call site before the record is built rather than after it is formatted and sent, so a suppressed debug line costs a compare instead of a clock read, a format, and a MsgSend; SLOG_LEVEL raises it per process, which only became possible once the spawn fix gave a process an environment to read. And the video console became a console rather than a display: a read with nothing to give now defers instead of returning end-of-file, the lie about capability that had sent every getty into a banner-print loop — so a login prompt can park on the screen and wait for a keyboard that does not exist yet. Underneath all of it the relay learned to carry a real exchange, circuits and transactions over the wire, with real Ethernet behind them: the SiFive FU740 on my desk, station 2, now has a working interface through its Cadence GEM, and the K3 through its DesignWare MAC, each finding its own controller in the device tree rather than by anything compiled in. Still an rc: the keyboard is unwritten, qsh's background operator and pipelines are degenerate, and slog's severity is honoured at one end and ignored at the other.

v0.2-rc3

rc3 is the candidate where the machine becomes something a person can sit down at. The one-line summary in the changelog is two consoles a person can actually work at, and a kill that ends a process rather than the one thread that noticed the signal — and both halves of that are harder than they sound.

Take the kill first, because it was a genuine hazard. A default-terminate signal was delivered on the process's signal thread, which called _Exit and terminated only itself; the process was reaped on paper while every other thread kept running, still holding whatever it had claimed. A shell killed while parked reading a console left that console owned forever — every later getty was refused, exited, was respawned, and the screen was gone for good on behalf of a process that no longer existed. The fix is ThreadDestroy(-1), meaning every other thread of the calling process, which is the meaning QNX gives it, wired into _Exit; and cross-thread destroy now accepts a target parked anywhere it can block, not only one parked in a receive — which is exactly the thread, parked in a read, that mattered and could not be killed before. Paired with it, the kernel now sends an unblock pulse naming the receive id when a parked client goes away, and the resource-manager framework handles it in one place, so a server waiting on a keystroke rather than on its client finally learns that the client is gone instead of discovering it only when it tries to reply, which for a console may be never.

The consoles rest on sessions. taskman grew a refcounted session — leader, process group, controlling terminal — that every spawn inherits and that setsid() breaks a process out of; /dev/tty is answered from the asking process's own session rather than from a global alias, which is the only way it can mean the right thing on a machine with two of them. Before this, a login on the screen printed its password prompt down the serial line. The terminal is carried as a path rather than a descriptor, because a QSOE file descriptor is a raw connection id straight onto a resource manager and cannot be mapped back to the name it was opened under — the name being the thing the pathmgr resolves — which is a real divergence from procnto, and worth stating rather than papering over. The terminal driver holds the foreground process group and signals it directly, since it is the one thing that sees the interrupt character arrive, so a Ctrl-C now reaches the running command rather than the shell blocked in waitpid. /proc reports the session, group, and terminal, and ps gained a TTY column with no flag, because on a machine with two consoles the first question about a process is which one it belongs to, and until now nothing could answer it.

Two long-standing keying mistakes fell out of the same work. A resource manager must key its per-open state on the server-side connection id the kernel stamps into every message, not the client's own descriptor number, which is unique only inside the process holding it — so two processes' descriptor 3 had been landing on one control block; the kernel had been sending the right id all along and taskman was dropping it. And the receive-id generation counter was sixteen bits, wide enough to reach the sign bit, at which point an ordinary message arrived looking like an error and its sender stayed reply-blocked forever while the server served everyone else — fifteen bits now, so the top bit stays clear. Both are the sort of defect that stays invisible until something finally trusts the field, and both got a regression test that provokes the exact state rather than a proxy for it.

And the screen is finally typeable. devu-dwc3 brings up a USB host controller, walks the hubs, drives a HID boot keyboard, and injects decoded keystrokes into /dev/con1 — push, not pull, so keyboard and console need know nothing about each other's startup order. It starts from a fragment on the disk rather than the boot archive, which looked wrong and is not: a console needs something to type into, and until /usr is mounted there is no getty and no login, so the half that draws is useful early and the half that types only after the mount. The same reasoning moved the network and USB drivers out of the archive entirely — some 340 KB no boot could use before the mount, and one driver that had been shipping twice.

That is v0.2 across three candidates: a wire, a second board's hard truths, and a pair of consoles with sessions behind them. What stands between here and v0.2 proper is written on each rc's own tin — the shell's job control, the far end of slog's severity, and enough hours on real hardware to trust that the emulator is no longer covering for anything. The next post is likely to be one of those rather than a fourth candidate.