QSOE/N v0.18: a second board, and a second compiler
QSOE/N boots to an interactive shell on the SpacemiT K3 — the second piece
of silicon it has ever run on, after the SiFive Unmatched. And the entire
operating system — kernel, taskman, libc.so, the runtime linker, every
program under /bin and /sbin — now builds with a second compiler,
jmcc, and boots on it too.
Neither was on the roadmap as "portability work". Both were the same lesson delivered twice: stop assuming the thing you were tuned for. The K3 broke every assumption baked into the hardware — one machine, one DRAM base, one interrupt geometry. jmcc broke every assumption baked into the toolchain — position-independent code, hard-float, GCC's particular generosity about relocations. Each found bugs the other never would have, and by the end the system is more honestly portable than it was before either one started.
The K3 does not resemble the machine we were tuned for
The Unmatched taught QSOE/N how to leave the emulator. The K3 taught it that leaving the emulator once is not the same as being portable.
It is a hostile target in all the interesting ways. Its DRAM starts at
4 GiB, not at zero. It numbers its harts 0..15. It is heterogeneous:
eight X100 application cores and eight A100 AI cores, the latter carrying a
quarter of the L2 and a narrower ISA. And it does not boot alone — it runs
management firmware on companion cores, an OpenSBI and an esos and two
RT24 remoteproc images, all of it described in a device tree that lists
seventeen /reserved-memory children before the kernel has reached its
first page table.
Getting from power-on to a shell prompt on that machine was a sequence of boot blockers, each found the same way: on the board, by a silence where output should have been.
A board is now a first-class concept. QSOE/N gained a Kconfig board
choice — several boards buildable from one tree — and stopped hard-coding
the machine it was first tuned for. The board turns out to be a purely
link-time property: KERNEL_LOAD_PA arrives from a linker --defsym, and
a missing one is a link error rather than an image quietly built for the
wrong DRAM base. That mattered more than it sounds: the old identity-map
loop started from a compile-time constant, and on a part with DRAM at
4 GiB it would have mapped two gigabytes of megapages straight over the
taskman and initrd regions and panicked at the first user blob.
The load address was off by one hex digit, and the board went dark.
The K3 image wanted 0x102200000 — 4 GiB plus 34 MiB, safely above the
vendor carve-outs. It was written 0x100200000, one digit short: 4 GiB
plus 2 MiB, which lands exactly on the reserved region the firmware
gives to a companion core. head.S zeroes some 15 MB of BSS from the load
address, so the firmware was overwritten before the first console call —
and then the kernel tried to speak to the console through firmware that
no longer existed. No output, no panic, no clue. Just a board that had
gone quiet.
The interrupt geometry is genuinely different, not just relocated. The
K3 is an AIA machine — APLIC plus IMSIC — and two assumptions that had
never been wrong on QEMU virt were wrong here. The IMSIC file stride is
not one page: the K3 declares riscv,guest-index-bits = 6, which makes the
real stride 256 KiB, and every hart above hart 0 would have had its
interrupts written into the wrong page — nothing faulting, the interrupts
simply never arriving, which reads as a broken device rather than a broken
address. And the S-domain APLIC used to be found by way of the console
UART; now it is identified by its msi-parent, which is what actually
defines it, so a board whose serial port we do not recognize no longer
takes the interrupt controller down with it.
The console itself had to be learned. The K3's UART is the xscale
lineage — spacemit,k1-uart — with its own register shift and width, all
of it now published to the userspace driver so the driver stays
board-independent.
The reserved-memory tables were sized for a simpler world. Eight exclusions was plenty for QEMU virt and the FU740. The K3's seventeen reserved children, plus the FDT and initrd and firmware span, needed three times that before the first page table exists. The tables now hold 64 exclusions, and overflow is a deliberate panic that names the region that did not fit — because with a table this size, the useful question is which node was being registered, not that a count was reached.
And the device tree lies about how much memory there is. The firmware
hands over a static blob declaring 8 GiB; the board has 16 GiB starting at
4 GiB. A bootloader that allocates the initrd high, through EFI's
AllocateMaxAddress, then puts it around 19 GiB — perfectly good memory
that the old pmap flatly refused to boot on, because it insisted the
initrd sit inside the RAM the kernel manages. It never needed to. That
requirement is gone; the check that remains is the one that matters — an
initrd overlapping the firmware or the kernel image is memory already in
use.
Sixteen cores, and only half of them belong to real-time work
The heterogeneous split needed a policy, and the policy QSOE/N chose is to
refuse to have one. The kernel discovers which harts are the AI cores —
by the device tree's own cpu-ai property, not by matching a vendor model
string, because a name is the wrong thing to hang a scheduler decision on —
records the distinction, and publishes it three ways: to per-hart kernel
code, to userspace through the sysmap, and to the operator as a single boot
line. Then it enforces nothing. Placement of interrupt service and
hard-real-time threads is a decision made where the latency requirements
are actually known. The kernel's whole job here is to make the difference
visible, because a machine that is quietly slower on half its cores is
not something anyone should have to discover by benchmark.
Loading skimmer-k3.elf ...
Loading modpkg.cpio ...
================================================
QSOE/N microkernel ("Skimmer") v0.18
================================================
fdt: no PCI host bridge -- PCI surface disabled
timer: timebase 24000000 Hz, tick 24000 cycles, scan hart 0
FDT: aplic@0xe0804000 srcs=512 imsic@0xe0400000 ids=511 gidxbits=6 stride=0x40000
pci-ecam@0x0 size=0x0 buses=0..0
ram@0x102000000 size=0x3fe000000
initrd@0x4f8cd0000 size=0x138200
physmem: 1 bank(s), 29 exclusion(s)
ram [0x102000000 .. 0x500000000) fdt memory@ node
excl [0x100200000 .. 0x100400000) fdt reserved-memory
excl [0x100800000 .. 0x100c00000) fdt reserved-memory
... (25 more reserved-memory / firmware exclusions) ...
excl [0x4f8cd0000 .. 0x4f8e08200) initrd (modpkg cpio)
excl [0x102000000 .. 0x103186000) firmware + kernel
intc: selected aia backend
[hart 0] primary up (Sv39 high-half), aia
Bringing up secondary harts via SBI HSM...
auxiliary harts (not for IST / hard-RT): mask 0xff00
[hart 1] secondary up (Sv39), aia
[hart 2] secondary up (Sv39), aia
... (12 more secondaries) ...
[hart 8] secondary up (Sv39), aia
All harts online.
Spawning taskman U-mode thread...
Handing primary hart over to LWKT scheduler.
================================================
QSOE/N taskman starting
================================================
taskman: cmdline: ''
[init] starting slogger...
[slogger] alive, pid=3
[init] starting pci-server...
[pci-server] alive, pid=4
[pci-server] ecam_init failed
[init] syncing system clock from hardware...
rtc: this board carries no battery-backed clock the kernel recognizes
[init] starting devc-ser8250...
[devc-ser8250] 16550 initialized @ vaddr 0x0000000020000000
[devc-ser8250] IRQ thread spawned, tid=25
[devc-ser8250] /dev/ser1 registered (chid=65544)
[init] repointing /dev/console -> /dev/ser1...
[init] no mainfs= on the command line
[init] root filesystem unavailable -- entering interactive shell.
Use 'shutdown' to power off.
[/]# echo "Hello from QSOE/N 0.18 running on SpacemiT PicoITX/K3!"
Hello from QSOE/N 0.18 running on SpacemiT PicoITX/K3!
[/]#Every line the post argues for is in that capture. gidxbits=6 stride=0x40000 is the IMSIC geometry, derived rather than assumed — 0x40000 is 256 KiB, the real per-hart file stride. The managed bank opens at 0x102000000, above the first 32 MiB of vendor firmware; the initrd sits near the top of RAM at 0x4f8cd0000, memory the old pmap would have refused. The seventeen reserved-memory nodes expand — with the firmware, initrd and kernel spans — into the 29 exclusion(s) the walk logs, comfortably under the raised ceiling and well past the old one of eight. mask 0xff00 marks harts 8–15 as the auxiliary A100 cores, held back from interrupt and hard-real-time work. And the last four lines are the whole of “Not in this release,” captured live: no ECAM, no mainfs=, no root filesystem — and then a shell, and a machine that answers back.
jmcc: the whole OS, built by a compiler that trusts nothing
Running the entire build through jmcc — the Johnson-Magnusson C Compiler, now a selectable toolchain alongside GCC — did to the software what the K3 did to the hardware. GCC had been quietly carrying the system on a set of conveniences: it built everything position-independent, it reached globals through the GOT, it papered over a whole class of relocation. jmcc does none of that, and every place the system had leaned on GCC's habits, it fell over — usefully.
The loader was silently skipping R_RISCV_COPY relocations. It never
mattered while every binary was PIE and reached imported data through the
GOT. A non-PIE executable importing a data object from a shared library —
devb-nvme reaching stderr in libc.so — got a private copy that read
as zero, and faulted on the first setvbuf. Now the loader binds COPY
relocations eagerly, and the storage driver comes up.
Worker threads were running with the wrong global pointer. The thread
trampoline lives in libc.so and set gp to its own __global_pointer$,
but gp is process-global and must be the main executable's — invisible
under PIE, fatal for a non-PIE qsh whose SIGCHLD handler then read its
job list out of libc's data segment and crashed on the first child exit.
The trampoline now carries the creating thread's gp forward instead of
recomputing it.
There were smaller ones — taskman's stack grown to four pages because
jmcc's frames spill everything to memory and overflow a single page; a
per-ABI multilib query so the soft-float runtime links against the
soft-float image; the discovery that jmcc writes no dependency files, which
had let a raised CONFIG_MAX_CPU_COUNT rebuild one translation unit and
not another, leaving a loop over sixteen harts reading an array still sized
for eight. Every one of them was a latent bug that GCC's defaults had been
hiding. The fully jmcc-built QSOE/N now boots the whole stack — slogger,
the PCI server, the serial and NVMe drivers, the /usr mount, sysinit — to
the login prompt, at parity with GCC.
Not in this release
The K3 boots to an interactive shell. It does not boot to login:.
There is no getty, no login, no shared command suite on the K3 yet, and the
reason is simple and worth stating plainly: its storage is UFS, or NVMe
behind a PCIe controller QSOE/N does not yet recognize. The Unmatched
reaches its root filesystem; the K3 does not, so there is nowhere to mount
/usr from and nothing to spawn a shell out of a filesystem for. That is
the next piece of work, and it is named here rather than left for someone
to find missing.
Comments
Post a Comment