How QSOE started

It was the first of April, and work on QRV — my port of QNX to RISC-V — was in full swing. A couple of weeks earlier I had put up an open petition asking QNX Software Systems and BlackBerry to re-license the historical 2007–2009 QNX Neutrino source under Apache 2.0. Around this time a thought settled in: there's a real chance it gets ignored completely. I needed a plan B.

I've kept the L4 microkernel in the back of my mind for a long time. I first read about it in the early 2000s, along with its various implementations — the ones with the nutty names (Hazelnut, Pistachio), and the one with the funny name, Fiasco, from the group in Dresden. I met those people in person about ten years ago. I never really stopped thinking about L4 after that; I knew about L4Linux and the rest. And I got particularly interested when I learned about seL4.

What seL4's authors did still strikes me as a genuinely novel approach: a real microkernel with a machine-checked proof of functional correctness. That is Computer Science and Engineering in the most respectable sense of both words. So the plan-B question more or less wrote itself: what if I implemented a QNX-like OS on top of seL4?

The name

I came up with it on that same walk to the office, somewhere around a restaurant called Da Giuseppe. QSOE works on several levels at once, which is why I kept it:

• QSO + e, for established. In ham radio a QSO is a contact between two stations, and QRV? asks "are you ready?". So: QRV? — sure; QSO, established.
• QNX-Style Open Environment.
• Quick and Secure Operating Environment.

The first reading turned out to be apt, since connections are exactly what the system is built around — but that's a topic for another post.

I wasn't the only one

I wasn't the first to try this. Andrew Warkentin in Canada had been building a QNX-style system on seL4, called UX/RT, for a long time, and had bumped up against seL4's IPC design from the inside — its fixed, deliberately small message size eventually pushed him to fork the kernel.

He also had a long public exchange with Gernot Heiser on the seL4 mailing list about IPC performance. Warkentin's benchmark — a client hammering a bit-bucket server — suggested QNX's IPC came out faster. Heiser's reply was that the benchmark wasn't measuring raw seL4 IPC at all: it ran inside a VM, used the syscalls the wrong way, and on the QNX/Linux side mostly timed a libc function call. Done properly, a round-trip on x86 is under ~900 cycles — only about 20% above the bare hardware cost of a kernel trap and a page-table switch, and in fact an order of magnitude ahead of QNX, not behind it.

The deeper point in that thread is the one that shaped how I think about the design: seL4 IPC is a protected procedure call into another process, not message passing in the classical sense. You use it for control; bulk data moves through shared memory.

April was nothing; the work began on May 12th

The git log is honest about April: two commits, both housekeeping. One script to clone the seL4 sources, and the work of extracting the bits and pieces I'd need. That is all of April.

The actual work started on May 12th. Through May I cut the first eight versions, 0.1 through 0.8. The system booted statically-linked user applications — most importantly a shell — and ran decently. seL4 underneath, a musl libc patched to live on it, a small runtime, and the first version of taskman: the single user-space server that does what QNX hands to procnto — process, memory and path management plus system services, all in one program.

One environment, two kernels

And right around there I asked the question that doubled the size of the project: what if QSOE had not one kernel, but two?

What if I implemented a QNX-like microkernel of my own — the funny name Skimmer came later — and arranged things so that the userspace, and all the resource managers, were shared between QSOE-on-seL4 and QSOE-on-Skimmer?

That is the design the project still runs on:

• QSOE/L — on seL4.
• QSOE/N — on Skimmer.

The userspace is 100% identical across both. The libc is 85%-or-more identical; only the thin layer that talks to the kernel differs. The one component that is genuinely different in each variant is taskman — which is exactly where the difference belongs, since it's the program that speaks to the kernel directly.

Skimmer

The seL4 variant gets multiprocessing essentially for free: seL4 already supports SMP, so it's a matter of selecting the SMP configuration and building. Skimmer is different. It's a kernel I write myself, so its concurrency model isn't a build option — it's the central design decision. This is where the old big-kernel-lock problem actually matters, and this is where I borrowed from DragonFly BSD.

DragonFly split from FreeBSD precisely over how to do SMP, and rather than locking shared kernel state ever more finely, it leaned on two mechanisms: light-weight kernel threads (LWKTs), each CPU running its own scheduler, and message ports (msgports) for handing work between subsystems instead of sharing state behind locks. That is the model I took for Skimmer — a message-passing kernel at its core, which sits comfortably next to the QNX-style message passing QSOE already does in user space.

It came together quickly. Skimmer's v0.1 through v0.3 were done in a couple of days. And from early on I embedded a serious stress test into the system itself — not a bolt-on harness, but part of the kernel — so that the concurrency could be put under real load rather than taken on faith.

Closing QRV, opening the umbrella

By mid-May it was clear that QRV was reaching its end, and that I wanted to put my effort into a free, Apache-2.0 system instead — the same license I had been asking BlackBerry for. QRV was finished on May 26th — exactly three months after I started it. The next day, May 27th, I started the "Umbrella" project: the larger QSOE that ties the two variants together, with Kconfig-style selection of the kernel (Skimmer or seL4), selection of which userspace programs to build, and so on.

That's how it started. I'll use this blog to write down where it goes from here.