I Wanted to Build QSOE 0.4 on Azure Linux 4.0 -- Here's What Happened

QSOE is built every day on Debian. The 0.4 release was tested from a fresh clone on Debian too, and that's exactly the problem: a build that has only ever seen one distro silently learns that distro's habits. So on Monday evening, a day after the 0.4 release, I tried something different: a brand-new Azure Linux 4.0 virtual machine in QEMU, an 8 GB disk, a minimal install, and nothing else. RPM packages, tdnf, no apt anywhere.

The rule for the evening was simple: clone, type make, and write down every single thing that breaks. I had Claude Code keep the log while I typed the errors in.

The obvious ones

A minimal image really is minimal. Before the first line of QSOE was compiled, I had installed git, make, wget, tar, pip3 and, a bit later than I should have, the host gcc. None of that is surprising. What was pleasant: xz, mtools, dosfstools and cpio were already there.

The seL4 side of the build has a host check that lists everything it needs in one go. It prints Debian package names, though, and three of them are spelled differently in the RPM world:

device-tree-compiler  ->  dtc
libxml2-utils         ->  libxml2
python3-yaml          ->  python3-pyyaml

With those translated, everything came straight from the Azure Linux repository:

sudo tdnf install cmake ninja-build dtc libxml2 \
    python3-jinja2 python3-ply python3-pyyaml

Free Pascal: not in the repo, but easy

QSOE's package tool, qupkg, is written in Pascal, and the host copy of it assembles the system image. Azure Linux has no Free Pascal package, but freepascal.org ships a self-contained x86_64 tarball with an install.sh. Five minutes.

Remember this one, it comes back later.

Our first real bug: a Debian path

python3: cannot open /usr/lib/python3/dist-packages/alldefconfig.py

QSOE uses kconfiglib for its configuration, and both kernels' Makefiles had its location hard-coded to where Debian puts Python modules. RPM distros use site-packages under a versioned directory. pip3 install --user kconfiglib plus make KCFG_LIB=<that directory> got past it. (It has to be on the command line: the Makefiles assign it with :=, which beats an exported variable.)

The cross-compiler: Bootlin to the rescue, almost

As expected, there is no riscv64-linux-gnu- cross toolchain in the Azure Linux repository. Bootlin publishes ready-made ones, and their riscv64-lp64d--glibc--stable-2026.08-1 (GCC 15.3.0) unpacked into /opt without fuss.

Bootlin's tools are called riscv64-buildroot-linux-gnu-*; QSOE expects riscv64-linux-gnu-*. The first idea was a set of renaming symlinks, and it failed in a way I hadn't seen before:

riscv64-linux-gnu-gcc.br_real: No such file or directory

Bootlin's gcc is a small Buildroot wrapper. It runs the real compiler, <the name it was called by>.br_real, from its own directory. Call it by a new name, and it looks for a binary that doesn't exist. The fix was two-line shell scripts instead of symlinks:

#!/bin/sh
exec /opt/riscv64-lp64d--glibc--stable-2026.08-1/bin/riscv64-buildroot-linux-gnu-gcc "$@"

One per tool, generated by a loop. After that, QSOE was compiling.

An emulator to build a kernel?

Then the seL4 configure step for the QEMU target stopped:

Failed to determine QEMU version (qemu-system-riscv64)

seL4 runs qemu-system-riscv64 at configure time, only to dump the virt board's device tree. We assumed Azure Linux wouldn't have a RISC-V QEMU and prepared a workaround (dump the DTB on another machine, hand it to cmake as QEMU_DTB). Then a quick tdnf search qemu showed qemu-system-riscv right there. Lesson: search first, assume later.

Still, a build that needs an emulator installed is odd. Shipping that DTB in the tree would remove the dependency.

The bug that hid itself

Next, building the host qupkg failed with nothing but:

make[2]: *** [Makefile:29: build/qupkg] Error 1

No compiler message at all. Our top-level Makefile runs that build with >/dev/null, and Free Pascal prints its errors to stdout. So the one line that explained everything went straight into the void.

It got better. Running the build by hand said "Nothing to be done": the compiler had written the binary and then failed, and make doesn't delete a target whose recipe failed. A second make would have happily carried on with a binary from a failed build. Deleting it and rebuilding finally showed the culprit:

Warning: "crtbegin.o" not found, this will probably cause a linking failure
Warning: "crtend.o" not found, this will probably cause a linking failure

qupkg is built with -Sew (warnings are errors), so these were fatal. And here is why Free Pascal had to be remembered: its install.sh writes /etc/fpc.cfg with a pointer to gcc's library directory, where those two files live. I had installed Free Pascal before gcc. No gcc, no directory, no line. One appended -Fl line in /etc/fpc.cfg fixed it. The real lesson is about order: gcc first, then Free Pascal.

The last two

lz4 was missing and is in the repo. Then:

note: oggenc not found -- no audio-test.ogg (apt install vorbis-tools)
mkpkg: pkg/qsoe-audio-samples.manifest:17: build/audio-test.ogg is not built

vorbis-tools isn't packaged for Azure Linux. And the Makefile treats oggenc as optional (it prints a note and moves on), while the audio samples package lists the Ogg file as mandatory. The two parts of the build disagree with each other, and it took a distro without vorbis-tools to notice. I copied the 15 KB test file in from my Debian machine, ran make once more, and:

Build succeeded. Both kernels, the shared userspace, the packages and the images, on a distro QSOE had never seen before.

The recipe

For anyone on Azure Linux, and very likely Fedora, since the package names mostly match:

sudo tdnf install git make gcc wget tar python3-pip cmake ninja-build dtc \
    libxml2 python3-jinja2 python3-ply python3-pyyaml qemu-system-riscv lz4
pip3 install --user kconfiglib pyfdt
# Free Pascal 3.2.2 tarball from freepascal.org -- AFTER gcc
# Bootlin riscv64-lp64d glibc toolchain in /opt,
#   plus exec wrappers named riscv64-linux-gnu-*
# audio-test.ogg from a machine with vorbis-tools (for now)
make -j KCFG_LIB=<kconfiglib's site-packages directory>

What we found in our own house

The missing packages were the expected part. The interesting part is the list of things that are our bugs, invisible on Debian:

  • kconfiglib's location hard-coded to the Debian path (and menuconfig checks for it with dpkg);
  • every "missing tool" hint says apt install;
  • the QEMU-target seL4 kernel needs QEMU just to be configured;
  • host tools built with a bare cc instead of a settable compiler;
  • qupkg's build output discarded, errors included;
  • a failed qupkg build leaves a usable-looking binary behind;
  • oggenc optional in one place, mandatory in another.

Seven small things, each of which would cost a newcomer an evening. They're on the list for 0.5.

Building QSOE on another distro took one evening and found seven bugs. One honest caveat: I built the images but didn't boot them in that VM. Booting comes next.

Comments

Post a Comment

Popular posts from this blog

QSOE project v0.1 released

The first QSOE release, v0.1, is out. It bundles both kernel variants, the boot loader, the userspace, and libc into a single numbered drop, and consists of the following components: QSOE/N v0.17 , with the custom "Skimmer" kernel. QSOE/L v0.14 , which works together with seL4 version 15. mr-bml v0.5.1 , a GRUB-derived boot loader with support for Multiboot 3, RISC-V Linux-style kernels, and kernels with an EFI stub. quser v0.5 , offering an mksh-based shell (qsh). libc v0.6 . The two variants share the same userspace above the seam: quser, qsh, and libc are identical, with only taskman and libc.so differing per kernel. That shared userspace is the whole point — one QNX-compatible environment over two very different microkernels. The milestone behind this release is on the QSOE/L side: it now boots from NVMe storage to an interactive login shell on the SiFive Unmatched (FU740). Spawning the first program off a mounted disk had exposed a pair of deadlocks — taskman blocked ...
Read more

A free QNX-like operating system (2003)

I came across an old post of mine on the comp.os.qnx newsgroup, dated August 7, 2003. The subject line was "A free QNX-like operating system," and under it I announced RadiOS — a hobby effort to build a free, QNX-compatible microkernel OS for x86, with the microkernel and task manager written entirely in NASM assembly. About 70% of the required system calls were in place by then, along with parts of libc, a console resource manager, and an ATA/IDE driver; there was no filesystem manager yet. The post asked whether anyone wanted to help hack on it, and signed off with "73!". The thread is here: https://groups.google.com/g/comp.os.qnx/c/srdvVoik37M The fuller picture survives on the old SourceForge page, still up at radios.sf.net . RadiOS targeted system-call compatibility with QNX Neutrino 6.1; its Task Manager mirrored the QNX process-manager message format, and its libc was derived from QNX 6.1 code. By mid-2004 it ran to roughly 45,000 lines, almost all NASM, w...
Read more

seL4 for SpacemiT K3

Image
The QSOE 0.3 notes say the K3 is QSOE/N only. That was true of the release set and, for the three days before it, false of the lab: on the evening of 14 September seL4 16.0.0 booted to user space on the SpacemiT K3 — 16 harts, at the first attempt — and by the small hours of the 16th QSOE/L was logged in on it, over the serial line and on the screen, with the NVMe, the USB keyboard and the network attached and, once up, nothing from the kernel on the console at all. The kernel is k3sel4 , version 16.0.0-qsoe-k3-0.2 . This post is about what it took, because seL4 16.0.0 could not have booted on this board: the SoC has none of the things the RISC-V kernel assumes, and one thing it does not expect. seL4 16.0.0 assumes the K3 has k3sel4 does a PLIC no PLIC anywhere: an S-mode APLIC in MSI delivery mode feeding 16 IMSIC interrup files (the RISC-V AIA) a driver for both, behind the kernel’s six-function interrupt-controller contract the SBI v0.1 legacy calls, for everything ...
Read more