QSOE audio: Crystal Clear

QSOE's 0.4 milestone has a one-line definition of done: continuous playback is audible on an HDMI audio extractor. Not "the driver loads", not "the counters look right" — audible. The extractor, a small FeinTech box that takes HDMI in and gives optical S/PDIF out, arrived on Tuesday. Two evenings later I have heard QSOE play music on two RISC-V boards, under both of its kernels, and for the first time ever on a system built on seL4. This is the story of those two evenings, because almost nothing went the way the plan said, and every detour taught something.

Tuesday: the Unmatched

The first board was the SiFive HiFive Unmatched, with an NVIDIA GK208 card in its PCIe slot. The card's HDMI output carries audio through an HD Audio controller that sits beside the GPU; the chain is GPU → HDMI → extractor → TOSLINK → an old JVC receiver.

Linux went first, to prove the chain. It was silent. The receiver had not been asked to select its digital input since the summer of 2021, and the way to ask it turned out to be a jog control I had forgotten existed. Once it listened, mpv played an internet radio station, and that was the first sound I have ever heard from any RISC-V system. Linux has its own trap here, worth writing down: the ALSA control called IEC958 Playback Switch gates the digital output, and it starts off.

Then QSOE/N, the native system on my own kernel. Its HD Audio driver, deva-hda, must first put the GPU's output into HDMI mode with its audio packets enabled — the firmware brings up the picture, not the sound — and then a 440 Hz test tone played for three seconds, clear and sinusoidal. A 30-second tone followed, then Herbie Hancock from an Ogg Vorbis file. Flawless. The 0.4 criterion, met.

For about ten minutes.

The gap

I ran sloginfo on the video console, the text scrolled, and the music stopped for more than a second and a half. It came back when the scrolling ended. Every time.

The obvious suspect was the scheduler: the console driver and the player shared a hart, and scrolling a text screen by rewriting video memory is not cheap. But "obvious" is a hypothesis, not a measurement, and the kernel could not yet say how much CPU time a thread had used. So I built that first: one clock read per context switch, the running total reported per thread through ps -H. It cost almost nothing and answered at once. The player was using about 16% of a hart where decoding needs about 3%.

Five times the work, for the same music. The cause was in the C library. QSOE's port of musl's stdio has no readv(), and in its absence the read path fetched exactly as many bytes as the caller asked for. The Vorbis decoder reads its file through fgetc — one byte at a time. On Unix that costs a system call per byte; on a microkernel, where a read() is a message to the filesystem server and a reply back, it is a full IPC round trip per byte. With a busy thread of equal priority on the same hart, each reply waited for the next scheduler tick, and the decoder ran at about a twentieth of real time. The ring drained, and the music stopped.

The fix was to read a buffer's worth and serve the small reads from it, as stdio has always been meant to. Afterwards the player used 3.3% of its hart, and the music played straight through a burst of scrolling and a stress test. The per-thread CPU counter stayed in the kernel: it paid for itself in one evening. The investigation became a chapter of the project's development story, called "One Byte at a Time".

Late that night came the second lesson. Under QSOE/L, the seL4-based variant, the player's counters were perfect — every period played, every interrupt on time — and there was no sound at all. The GPU's HDMI output was off: the video BIOS had found no display at that boot (a long cable and a marginal EDID read, since fixed on the firmware side). No link, no audio packets, however perfect the counters. From that night on the rule is simple: audio works when I hear it, not when the numbers say so.

Wednesday: the K3

The second board was the SpacemiT K3 Pico-ITX. It sends audio down its DisplayPort output: an I2S controller feeds the DP transmitter, and a small audio DMA engine feeds the I2S controller from 16 KiB of on-chip SRAM. The chain: DP → a docking station's HDMI port → the extractor. Linux played through it on the first try.

QSOE/L did not. The first stream after boot hung at once: the DMA engine fetched its first descriptor, finished, and stopped, with its current-descriptor register reading 0x0000000b — not an address, garbage. I had seen exactly this signature five days earlier, when seL4's device memory was handed to the driver out of order and the driver wrote its descriptors into the wrong physical pages. So the first suspect was the same: the way QSOE/L's task manager carves device memory out of seL4's untyped objects. The kernel was asked. Every frame the task manager carved was checked with Page_GetAddress against the address it was meant to cover. Every single one was right.

Then QSOE/N hung in the same way, on the same board. Not an seL4 problem, then, and not a carving problem. After a power cycle it played — but badly: a heavy distortion, as if the signal passed through a relay chattering at about ten hertz. The counters, again, were perfect.

I tried the obvious things. A shorter ring, a longer period, only half the SRAM. The rattle did not follow any of them. I compared our clock tree, dividers, I2S format and DisplayPort audio fields against the vendor's Linux drivers, field by field: identical. Reading more code was not going to find it.

The dump

So I did what finally works on a board that runs two operating systems: ask the one that works. A tiny read-only tool dumped every register of the audio path from Linux while it played — the DMA channel, its descriptors in SRAM, the I2S controller, its clock control, the DisplayPort audio block — and the QSOE driver learned to dump the same registers in the same format. Then diff.

The descriptors were byte-for-byte identical. Same addresses, same lengths, same chaining. And yet under QSOE the DMA engine had fetched garbage from that very memory.

When the CPU and a device disagree about what is in memory, it is the cache. On the K3, that SRAM, mapped by its physical address, is cacheable to the application cores — and the DMA engine does not look into the CPU's caches. Everything the driver wrote there, descriptors and audio alike, reached the SRAM only when the cache happened to evict it. A first descriptor the engine could not follow; stale periods played back as a ten-hertz rattle; and "it works after a power cycle" was nothing but eviction luck. Linux never saw it because it maps that SRAM uncached.

The fix is a few lines. Our audio library, which moves the samples into the ring, now tells the driver which bytes it has just written; the K3 driver writes those cache blocks back with the RISC-V cbo.clean instruction, and does the same for its descriptors. The block size comes from the device tree, and on a board whose DMA sees the caches — like HD Audio on PCIe — nothing is done at all.

Herbie Hancock's "Doin' It", on QSOE/N: crystal clear.

Then a warm reboot — no power cycle this time, which was itself the test of the explanation — into QSOE/L, and the same track again. Crystal clear. The first time music has played on a QSOE system built on seL4.

The numbers for "Gentle Thoughts", played on QSOE/L afterwards: 20,277,760 frames, 7 minutes 2 seconds, 0 dropped, 0 late wake-ups, the worst interrupt-thread delay 62.5 microseconds.

What the two evenings taught

• The ear is the oracle. Twice the counters said everything was perfect while the truth was silence or a rattle. Counters measure what the software believes; a listener measures what happened.
• Measure before theorizing. The CPU counter, the per-frame address check and the register dump each ended an argument that reading code had kept alive.
• The other operating system on the board is a reference instrument. A dump from a working Linux, diffed against ours, found in minutes what an evening of comparing source had not.
• Old rules still bite. Buffer your reads; clean your caches before DMA. Neither is new. Both were the whole story.

Next

A resampler in the player: the K3 plays only 8, 16 and 48 kHz, so 44.1 kHz recordings are refused today. The same cache maintenance for the VisionFive 2's PWM DAC, whose DMA is no more coherent than the K3's. And the next real work on QSOE/L's interrupt latency: during a network copy alongside playback, 108 period wake-ups came more than 1.4 ms late, the worst 21 ms. The four-period ring absorbed every one of them and not a frame was lost, but on a quiet system the worst case is 62.5 microseconds, and closing that gap is what the next milestone, the real-time package, is about.

The investigation was done working with Claude Code, which ran the board over its serial console, built the instruments and chased the numbers, while I listened. The two fixes are in QSOE's C library and userspace at gitlab.com/qsoe, Apache-2.0, like the rest of QSOE.