- Spectrum Discuss - spectrum-os.org

This Week in Spectrum, 2025-W38
by Alyssa Ross 23 Sep '25

23 Sep '25

Whoops — I'm a bit late again this week. Was really busy yesterday. Given how late I was last week though, there's not all that much anyway. As usual, we acquired some build fixes and development environments from Demi. Some small ones were applied — my favourite is making sure that early kernel logs don't get lost in the development environment[1] — and bigger things like automatically generating file lists for images[2], and build performance improvements[3] are still under discussion. We also continued discussing the udev switch, and I expect to see a new version of that posted soon. [1]: https://spectrum-os.org/git/spectrum/commit/?id=1c471fc4191958dfbffbdb5ad2d… [2]: https://spectrum-os.org/lists/archives/spectrum-devel/20250903-genfiles-v1-… [3]: https://spectrum-os.org/lists/archives/spectrum-devel/20250919-less-dirname… [4]: https://spectrum-os.org/lists/archives/spectrum-devel/20250913-udev-v1-0-ea… I spent some time debugging a Rutabaga crash in QEMU. When I submitted a fix, I learned that it had already been fixed a few days prior[5], but I also discovered that Rutabaga has been extracted from crosvm to a standalone project, which I think is fantastic news. They're even adding a proper Meson build (no wrapped Cargo)[6]. Excited to review that. [5]: https://github.com/magma-gpu/rutabaga_gfx/pull/9 [6]: https://github.com/magma-gpu/rutabaga_gfx/pull/13 Just as I've been writing this, Yureka has submitted a new version of the XDP forwarding series[7]. We're aiming to have the XDP forwarder program committed soon, but actually using it will have to wait for the rest of the new network stack, and for a couple of Nixpkgs fixes[8][9], which we also made some progress on this week. [7]: https://spectrum-os.org/lists/archives/spectrum-devel/20250923132012.28013-… [8]: https://github.com/NixOS/nixpkgs/pull/444996 [9]: https://github.com/NixOS/nixpkgs/pull/436206 I've been working on a couple of talks I'll be giving soon. The first is "Rethinking the compartmentalization experience" at the Qubes OS Summit in Berlin on Friday at 12:30 UTC[10], and the other is "An upstream-first approach to application virtualization" at XDC in Vienna on Tuesday at 08:30 UTC. These are actually the first talks I'll ever have given about Spectrum, because I've historically wanted to avoid publicity until I was confident Spectrum would be able to deliver on its promises. As we're approaching an initial release, this is becoming less of a concern, and by now we've learned a lot that's going to be useful to share regardless. As always, get in touch if you want to chat at either of these events. [10]: https://cfp.3mdeb.com/qubes-os-summit-2025/talk/UPMP38/ [11]: https://indico.freedesktop.org/event/10/contributions/436/

1 0

This Week in Spectrum, 2025-W37
by Alyssa Ross 17 Sep '25

17 Sep '25

Decided I'd rather get this out late than miss another week. There was no This Week in Spectrum for 2025-W36, because I was too tired after KVM Forum and NixCon to get it out on time, and in the last week I was in the hospital for several days for various tests, including when I'd usually send this out. Anyway, with two and a bit weeks to cover, this is going to be a long one. KVM Forum --------- Was nice to put faces on names from mailing lists — will make it easier to approach people online to discuss new KVM things, I think. The most interesting talk for me was "IOMMU in rust-vmm, and new FUSE+VDUSE use cases"[1], in which I learned: • virtiofsd is going to get support for additional backends, e.g. NFS and cloud storage. • A vhost-user↔VDUSE bridge is in development, which will allow a vhost-user device to be used by the kernel running that device. This raises the possibility of moving device implementations out of the host kernel. • virtio-vhost-user didn't end up getting through standardization and implementation not for any fundamental reason, but just because nobody was sufficiently motivated to make it happen. This also came up again just yesterday on the virtualization list[2]. It would be nice to see some momentum again towards moving device implementations into VMs. [1]: https://lore.kernel.org/virtualization/CAJSP0QWV+=+v9Z0wU9qJcdToDKBnWiGKzOV… [2]: https://www.youtube.com/watch?v=qsFc234tzz4&pp=ygUOaW9tbXUgcnVzdC12bW0%3D NixCon ------ KVM Forum and NixCon partially overlapped, so I only managed to make the second half of the latter. I didn't see any talks, but had some good conversations, mostly in the areas of documentation (which I hope we'll get a lot of work done on next year) and funding / business model stuff, which will be very important to get right to keep the project going after NGI ends. Speaking of which: NGI Zero Commons Business Circle -------------------------------- As recipients of an NGI Zero grant, we have access to various support services[3]. One of these is called "Business Circle", and provides business consulting and mentorship. I had an initial call with them after I got home after NixCon. Not much to report yet, but it does sound like we'll have lots of support available with legal etc. stuff if we need it to take the project forward. [3]: https://nlnet.nl/NGI0/services/ Cloud Hypervisor ---------------- Demi and I are currently working on sandboxing services on the Spectrum host — best effort defense in depth, since these services are part of our TCB anyway. One easy win there, I thought, would have been enabling Cloud Hypervisor's support for locking itself down with landlock, preventing it from accessing any non-allowlisted files. This should have been just a case of turning it on, and passing it a couple of extra paths to allow access to so VMs were allowed to have new devices passed through to them after creation, but I ended up finding quite a few bugs in the implementation in Cloud Hypervisor which had to be fixed first. The default rules were overly strict, resulting in VSOCK being non-functional if enabled[4], and in it failing to start at all on many aarch64 machines[5], and I also found that if landlock was requested, but the kernel didn't support it, Cloud Hypervisor would just start anyway, without landlock[6]. I've fixed all of these upstream now, and conveniently, a new Cloud Hypervisor release was scheduled for a couple of days afterwards, so these changes have already made their way into Spectrum. I haven't committed the Spectrum-side landlock stuff though, as I want to test it a bit more first. [4]: https://github.com/cloud-hypervisor/cloud-hypervisor/pull/7334 [5]: https://github.com/cloud-hypervisor/cloud-hypervisor/pull/7331 [6]: https://github.com/cloud-hypervisor/cloud-hypervisor/pull/7335 rust-vmm -------- The next step towards sound support in Spectrum is running vhost-device-sound on the host, but we can't currently do this robustly (and without polling), because vhost-device-sound doesn't support any form of socket activation or readiness notification. I've been working on adding support for the former, but it turned out not to be possible to implement with the current interface of the vhost-user-backend crate. In fact, I discovered, all the vhost-device devices create a new socket each time they're ready to accept a connection, meaning that, after the end of a connection, there's a window where a new connection can't be accepted. The right thing to do here is to reuse a socket between requests, and making that change will conveniently also make it straightforward to optionally pass in a socket provided by the service manager rather than creating it in the vhost-device program, so I changed the vhost-user-backend API[7], and am now waiting for a release, at which point I'll submit my changes to vhost-device-sound to update vhost-user-backend, fix the race between connections, and add support for using an externally provided socket. [7]: https://github.com/rust-vmm/vhost/pull/321 udev ---- Demi has been working on switching the host system from mdevd to systemd-udevd, with the motivation being that systemd-udevd comes with a big database of device quirks, which are likely to be beneficial to us in our aim of broad hardware support. This was originally done as part of her experiment with booting Spectrum with systemd[8], but has now been extracted into a standalone series[9], which is currently awaiting review. One of the complications of this, as I understand it, is that unlike every other service we run, we're not confident it's okay to start services that depend on udev as soon as the udev control socket is created, so we needed an adapter from systemd's readiness notification protocol to s6's. [8]: https://spectrum-os.org/lists/archives/spectrum-devel/20250904-systemd-v1-0… [9]: https://spectrum-os.org/lists/archives/spectrum-devel/20250913-udev-v1-0-ea… xdp-forwarder ------------- Yureka submitted a new version of her program for forwarding packets between two network interfaces[10], which will be part of our new networking stack. The idea here is that we can get the program reviewed and committed without having to wait for everything to be ready to switch to it. There were some more review comments from Demi and me, but nothing major, so I expect we'll see this part of the networking work finished soon. [10]: https://spectrum-os.org/lists/archives/spectrum-devel/20250906141228.235763… Ergonomics and other small improvements --------------------------------------- Demi has been submitting a lot of small fixes and improvements for the Spectrum development experience. Between conferences and the hospital I haven't had a chance to look at many of these yet, but I did apply a few of the most straightforward ones already. Of course, there have also been all the usual recurring things like server updates, Nixpkgs reviews and contributions, and so on, but this is getting far too long already, so I'm going to leave it here. Hopefully at the end of this week I'll be able to get back to regular updates of reasonable length…

1 0

[ANNOUNCE] cloud-hypervisor-48.0-spectrum0
by Alyssa Ross 14 Sep '25

14 Sep '25

I've updated Spectrum's Cloud Hypervisor patches (which add virtio-gpu support) to support Cloud Hypervisor 48.0. More information is available at <https://spectrum-os.org/software/cloud-hypervisor/>.

1 0

This Week in Spectrum, 2025-W35
by Alyssa Ross 01 Sep '25

01 Sep '25

Quiet week this week. Funding stuff is all sorted (for now), so I've been having a bit of a break (did get through a bit of an email backlog, though), and Demi and Yureka have both been carrying on with their respective bits — getting the stuff we want to keep from the systemd proof of concept, and working on the driver VM part of the new networking stack respectively. This week, I'll be at the KVM Forum, and later NixCon. I haven't been to KVM Forum before, but I'm expecting it to be a very useful opportunity to get info about virtualization stuff that's often not very well documented and difficult to see the whole picture of. I'm making a list of things I want to try to find out about, but if there are any suggestions, or if you want to meet with me at either of these events, let me know!

1 0

This Week in Spectrum, 2025-W34
by Alyssa Ross 25 Aug '25

25 Aug '25

This week, we've seen the first results from Yureka's work on Spectrum's new networking stack posted to the mailing list[1]. The draft submission modifies the networking VM to forward packets using XDP between physical interfaces attached to the VM and a virtual interface that will be provided by the host system. She has also been working on getting the required dependencies into Nixpkgs: updating[2] libbpf to a version that includes her recently accepted bugfix[3], and fixing a musl build issue[4][5]. [1]: https://spectrum-os.org/lists/archives/spectrum-devel/20250823222134.177241… [2]: https://github.com/NixOS/nixpkgs/pull/435918 [3]: https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git/commit/?id… [4]: https://github.com/libbpf/libbpf/pull/919 [5]: https://github.com/NixOS/nixpkgs/pull/436237#discussion_r2296245797 Demi has been experimenting with using the systemd service manager on the Spectrum host system. The main motivation was systemd's service hardening capabilities. I think we will probably stick with s6 for now, having discovered that we can likely get a similar level of hardening with standalone sandboxing tools, and just creating users for VM services at VM import time, but I wouldn't be surprised if we revisit systemd again in future, and am open to the switch at some point in future if it will be of benefit to Spectrum. I also understand that her work has resulted in a number of service manager-independent improvements to Spectrum that we should see posted soon. As for me, it's been recovering from the second (final) round of dental surgery, the usual Nixpkgs and server maintenance, and most of all getting the new grant all finalized, which I'm hoping is now basically done. I'm looking forward to actually getting back to proper work soon. :)

1 0

This Week in Spectrum, 2025-W33
by Alyssa Ross 18 Aug '25

18 Aug '25

Going to keep it very brief this week because I'm once again recovering from a wisdom teeth extraction (the last one!). • Yureka's libbpf fix[1] has been applied upstream, and she's now working on exposing vhost-user-net devices from her userspace router. • Demi has been doing some exploratory work on using systemd on the Spectrum host system[2]. We want to evaluate whether the extra complexity would be justified to get access to its extensive sandboxing features. • Apart from recovering, I did a routine test build with a new Nixpkgs, and then spent a while bisecting a kernel regression that broke virtio-gpu[3] (which it turned out had already been discovered and reverted, but my report at least got it tracked by regzbot as well). We also did some work sorting out the new funding. [1]: https://lore.kernel.org/all/20250814180113.1245565-2-yuka@yuka.dev/ [2]: https://github.com/DemiMarie/spectrum/compare/main...b4/systemd [3]: https://lore.kernel.org/all/kgasjsq2s4pshravsinycfihdfjhdts5iz2fox42aejs4xq…

1 0

This Week in Spectrum, 2025-W32
by Alyssa Ross 11 Aug '25

11 Aug '25

Demi and I figured out the last known blocker for her Cloud Hypervisor generic vhost-user device[1] — reads and writes of the VIRTIO configuration space weren't being forwarded to the backend — and she's now seen it work with virtiofsd. Another of her Cloud Hypervisor PRs was just merged[2], so I'm hoping generic vhost-user will be quite soon too, and then we'll be able to start easily adopting more device types in Spectrum, starting with virtio-sound. She also submitted a few small Spectrum improvements[3][4][5][6]. [1]: https://github.com/cloud-hypervisor/cloud-hypervisor/pull/7221 [2]: https://github.com/cloud-hypervisor/cloud-hypervisor/pull/7223 [3]: https://spectrum-os.org/lists/archives/spectrum-devel/20250807-no-require-k… [4]: https://spectrum-os.org/lists/archives/spectrum-devel/20250807-no-erofs-xat… [5]: https://spectrum-os.org/lists/archives/spectrum-devel/20250808-envrc-v1-1-4… [6]: https://spectrum-os.org/lists/archives/spectrum-devel/20250807-host-dev-fd-… Yureka has continued working on her program that will forward traffic on network devices out of driver VMs to the host userspace router, and the code is now public[7]. It now handles multiple physical interfaces per driver VM, which will be important if we want to use both of them and they're in the same IOMMU group. [7]: https://cyberchaos.dev/yuka/xdp-forwarder As for me, it was mostly the usual Nixpkgs maintenance — start of a new staging cycle — and preparation for the new grant. I also fixed a few Spectrum bugs — a couple of compatibility fixes for applications, and a harmless error during the build. I've been at WHY2025 this week too, and having some good conversations about future opportunities for Spectrum. I'll be here until Wednesday morning — if you are too and would like to talk about Spectrum, or get a sticker, DECT QYLS. :)

1 0

This Week in Spectrum, 2025-W31
by Alyssa Ross 04 Aug '25

04 Aug '25

First, some big news: Spectrum's application for funding from the NGI Zero Commons programme has been accepted. The final amount granted is still undetermined, but I expect that Demi's, Yureka's and my work on new feature development Spectrum will be funded for at least the next year. (Maintenance and administrative work is generally not covered — that's where GitHub Sponsors / Liberapay donations come in for me.) This is the first time we'll have funding that covers more than just me, so it should mean that we'll be able to really accelerate our pace of development, as we've already started to see with Demi and Yureka's work over the past couple of months, for which I'm grateful to them for having been willing to make a head start before we had this confirmed. There's still more work to do, agreeing the exact amount of funding, and what work will be covered, so expect a lot of our time (especially mine) to be occupied with that in the near future. Next, Demi's work on PipeWire support in guests is now in the repo[1]. This doesn't mean that we have audio support in Spectrum quite yet, because there's also some host-side work to do to expose a virtio-sound device. In that direction, I added vhost-device-sound to Nixpkgs[2], and I started working on adding socket activation support to it. [1]: https://spectrum-os.org/git/spectrum/commit/?id=6c945fc8b81ab15866567816007… [2]: https://github.com/NixOS/nixpkgs/pull/429376 Yureka is now able to send and receive ethernet frames to a Wi-Fi interface in a VM. This is an important step towards our goal of moving routing to host userspace, connecting VMs running applications with VMs running network drivers. I also spent some time on Nixpkgs maintenance. A test build of Spectrum against staging identified some build regressions, so I've been working through those[3][4][5]. I also tested a crosvm update, and release candidates of QEMU and Meson, which proved worthwhile because I found a regression in Meson affecting Spectrum's tests[6]. [3]: https://github.com/NixOS/nixpkgs/pull/429588 [4]: https://github.com/NixOS/nixpkgs/pull/429585 [5]: https://github.com/NixOS/nixpkgs/pull/429816 [6]: https://github.com/mesonbuild/meson/issues/14869

1 0

https://spectrum-os.org blocking Tor: is this intentional?
by Demi Marie Obenour 30 Jul '25

30 Jul '25

I noticed that https://spectrum-os.org doesn't work from Tor. Is this intentional? -- Sincerely, Demi Marie Obenour (she/her/hers)

2 1

This Week in Spectrum, 2025-W30
by Alyssa Ross 28 Jul '25

28 Jul '25

This week, Demi and I continued the review of the patches adding PipeWire to guests, now on their seventh revision[1], with some preparatory patches having already been applied[2][3]. She also opened a couple of Cloud Hypervisor PRs[4][5], one of which adds support for a generic vhost-user device. This would be a very exciting feature for Cloud Hypervisor to gain, because it would mean using new kinds of vhost-user device with Cloud Hypervisor wouldn't require any Cloud Hypervisor modifications. I believe (but we haven't confirmed yet) that this should allow using a vhost-user-sound device with Cloud Hypervisor straight away, and eventually allow us to drop our patches for GPU support altogether, although that also requires some extra vhost-user messages to be standardised (an effort which is still ongoing[6]). [1]: https://spectrum-os.org/lists/archives/spectrum-devel/263f81f2-9e86-4bb1-be… [2]: https://spectrum-os.org/git/spectrum/commit/?id=b3cb2511d3612a1d58246fabdc6… [3]: https://spectrum-os.org/git/spectrum/commit/?id=560fd878ba1bbd8df0fe28488e7… [4]: https://github.com/cloud-hypervisor/cloud-hypervisor/pull/7221 [5]: https://github.com/cloud-hypervisor/cloud-hypervisor/pull/7223 [6]: https://lore.kernel.org/qemu-devel/20250609144729.884027-1-aesteve@redhat.c… As for me, aside from reviewing and applying some of Demi's fixes, I updated Spectrum's pinned Nixpkgs[7] and released a new version of the Cloud Hypervisor patches[8], and did the usual round of server updates. Sometimes, updating the pinned Nixpkgs is easy, otherwise, due to a change that looked entirely inconsequential upstream, it can take a lot of work, and this has been one of the latter variety, so I'm glad it's finally done. Since then, and having just returned home from a couple of months of travelling, I've been feeling very tired. A few weeks ago[8], I mentioned that I'd been thinking I might have to calm things down a bit for myself to make sure I'm on the right side of burnout, and I think now is probably the time for that, so I'd expect there won't be much direct progress from me for the next week or two, although I should still be keeping up with emails — feeling like I'd come back to a mountain of them would be much worse than just dealing with them roughly as they come in — and doing what I can to make sure nobody else is blocked. [7]: https://spectrum-os.org/git/spectrum/commit/?id=d0a5f0abdc356f1bba676d489ba… [8]: https://spectrum-os.org/lists/archives/spectrum-discuss/87o6twmg2p.fsf@alys… See you soon!

1 0