- Spectrum Discuss - spectrum-os.org

This Week in Spectrum, 2025-W35
by Alyssa Ross 01 Sep '25

01 Sep '25

Quiet week this week. Funding stuff is all sorted (for now), so I've been having a bit of a break (did get through a bit of an email backlog, though), and Demi and Yureka have both been carrying on with their respective bits — getting the stuff we want to keep from the systemd proof of concept, and working on the driver VM part of the new networking stack respectively. This week, I'll be at the KVM Forum, and later NixCon. I haven't been to KVM Forum before, but I'm expecting it to be a very useful opportunity to get info about virtualization stuff that's often not very well documented and difficult to see the whole picture of. I'm making a list of things I want to try to find out about, but if there are any suggestions, or if you want to meet with me at either of these events, let me know!

1 0

This Week in Spectrum, 2025-W34
by Alyssa Ross 25 Aug '25

25 Aug '25

This week, we've seen the first results from Yureka's work on Spectrum's new networking stack posted to the mailing list[1]. The draft submission modifies the networking VM to forward packets using XDP between physical interfaces attached to the VM and a virtual interface that will be provided by the host system. She has also been working on getting the required dependencies into Nixpkgs: updating[2] libbpf to a version that includes her recently accepted bugfix[3], and fixing a musl build issue[4][5]. [1]: https://spectrum-os.org/lists/archives/spectrum-devel/20250823222134.177241… [2]: https://github.com/NixOS/nixpkgs/pull/435918 [3]: https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git/commit/?id… [4]: https://github.com/libbpf/libbpf/pull/919 [5]: https://github.com/NixOS/nixpkgs/pull/436237#discussion_r2296245797 Demi has been experimenting with using the systemd service manager on the Spectrum host system. The main motivation was systemd's service hardening capabilities. I think we will probably stick with s6 for now, having discovered that we can likely get a similar level of hardening with standalone sandboxing tools, and just creating users for VM services at VM import time, but I wouldn't be surprised if we revisit systemd again in future, and am open to the switch at some point in future if it will be of benefit to Spectrum. I also understand that her work has resulted in a number of service manager-independent improvements to Spectrum that we should see posted soon. As for me, it's been recovering from the second (final) round of dental surgery, the usual Nixpkgs and server maintenance, and most of all getting the new grant all finalized, which I'm hoping is now basically done. I'm looking forward to actually getting back to proper work soon. :)

1 0

This Week in Spectrum, 2025-W33
by Alyssa Ross 18 Aug '25

18 Aug '25

Going to keep it very brief this week because I'm once again recovering from a wisdom teeth extraction (the last one!). • Yureka's libbpf fix[1] has been applied upstream, and she's now working on exposing vhost-user-net devices from her userspace router. • Demi has been doing some exploratory work on using systemd on the Spectrum host system[2]. We want to evaluate whether the extra complexity would be justified to get access to its extensive sandboxing features. • Apart from recovering, I did a routine test build with a new Nixpkgs, and then spent a while bisecting a kernel regression that broke virtio-gpu[3] (which it turned out had already been discovered and reverted, but my report at least got it tracked by regzbot as well). We also did some work sorting out the new funding. [1]: https://lore.kernel.org/all/20250814180113.1245565-2-yuka@yuka.dev/ [2]: https://github.com/DemiMarie/spectrum/compare/main...b4/systemd [3]: https://lore.kernel.org/all/kgasjsq2s4pshravsinycfihdfjhdts5iz2fox42aejs4xq…

1 0

This Week in Spectrum, 2025-W32
by Alyssa Ross 11 Aug '25

11 Aug '25

Demi and I figured out the last known blocker for her Cloud Hypervisor generic vhost-user device[1] — reads and writes of the VIRTIO configuration space weren't being forwarded to the backend — and she's now seen it work with virtiofsd. Another of her Cloud Hypervisor PRs was just merged[2], so I'm hoping generic vhost-user will be quite soon too, and then we'll be able to start easily adopting more device types in Spectrum, starting with virtio-sound. She also submitted a few small Spectrum improvements[3][4][5][6]. [1]: https://github.com/cloud-hypervisor/cloud-hypervisor/pull/7221 [2]: https://github.com/cloud-hypervisor/cloud-hypervisor/pull/7223 [3]: https://spectrum-os.org/lists/archives/spectrum-devel/20250807-no-require-k… [4]: https://spectrum-os.org/lists/archives/spectrum-devel/20250807-no-erofs-xat… [5]: https://spectrum-os.org/lists/archives/spectrum-devel/20250808-envrc-v1-1-4… [6]: https://spectrum-os.org/lists/archives/spectrum-devel/20250807-host-dev-fd-… Yureka has continued working on her program that will forward traffic on network devices out of driver VMs to the host userspace router, and the code is now public[7]. It now handles multiple physical interfaces per driver VM, which will be important if we want to use both of them and they're in the same IOMMU group. [7]: https://cyberchaos.dev/yuka/xdp-forwarder As for me, it was mostly the usual Nixpkgs maintenance — start of a new staging cycle — and preparation for the new grant. I also fixed a few Spectrum bugs — a couple of compatibility fixes for applications, and a harmless error during the build. I've been at WHY2025 this week too, and having some good conversations about future opportunities for Spectrum. I'll be here until Wednesday morning — if you are too and would like to talk about Spectrum, or get a sticker, DECT QYLS. :)

1 0

This Week in Spectrum, 2025-W31
by Alyssa Ross 04 Aug '25

04 Aug '25

First, some big news: Spectrum's application for funding from the NGI Zero Commons programme has been accepted. The final amount granted is still undetermined, but I expect that Demi's, Yureka's and my work on new feature development Spectrum will be funded for at least the next year. (Maintenance and administrative work is generally not covered — that's where GitHub Sponsors / Liberapay donations come in for me.) This is the first time we'll have funding that covers more than just me, so it should mean that we'll be able to really accelerate our pace of development, as we've already started to see with Demi and Yureka's work over the past couple of months, for which I'm grateful to them for having been willing to make a head start before we had this confirmed. There's still more work to do, agreeing the exact amount of funding, and what work will be covered, so expect a lot of our time (especially mine) to be occupied with that in the near future. Next, Demi's work on PipeWire support in guests is now in the repo[1]. This doesn't mean that we have audio support in Spectrum quite yet, because there's also some host-side work to do to expose a virtio-sound device. In that direction, I added vhost-device-sound to Nixpkgs[2], and I started working on adding socket activation support to it. [1]: https://spectrum-os.org/git/spectrum/commit/?id=6c945fc8b81ab15866567816007… [2]: https://github.com/NixOS/nixpkgs/pull/429376 Yureka is now able to send and receive ethernet frames to a Wi-Fi interface in a VM. This is an important step towards our goal of moving routing to host userspace, connecting VMs running applications with VMs running network drivers. I also spent some time on Nixpkgs maintenance. A test build of Spectrum against staging identified some build regressions, so I've been working through those[3][4][5]. I also tested a crosvm update, and release candidates of QEMU and Meson, which proved worthwhile because I found a regression in Meson affecting Spectrum's tests[6]. [3]: https://github.com/NixOS/nixpkgs/pull/429588 [4]: https://github.com/NixOS/nixpkgs/pull/429585 [5]: https://github.com/NixOS/nixpkgs/pull/429816 [6]: https://github.com/mesonbuild/meson/issues/14869

1 0

https://spectrum-os.org blocking Tor: is this intentional?
by Demi Marie Obenour 30 Jul '25

30 Jul '25

I noticed that https://spectrum-os.org doesn't work from Tor. Is this intentional? -- Sincerely, Demi Marie Obenour (she/her/hers)

2 1

This Week in Spectrum, 2025-W30
by Alyssa Ross 28 Jul '25

28 Jul '25

This week, Demi and I continued the review of the patches adding PipeWire to guests, now on their seventh revision[1], with some preparatory patches having already been applied[2][3]. She also opened a couple of Cloud Hypervisor PRs[4][5], one of which adds support for a generic vhost-user device. This would be a very exciting feature for Cloud Hypervisor to gain, because it would mean using new kinds of vhost-user device with Cloud Hypervisor wouldn't require any Cloud Hypervisor modifications. I believe (but we haven't confirmed yet) that this should allow using a vhost-user-sound device with Cloud Hypervisor straight away, and eventually allow us to drop our patches for GPU support altogether, although that also requires some extra vhost-user messages to be standardised (an effort which is still ongoing[6]). [1]: https://spectrum-os.org/lists/archives/spectrum-devel/263f81f2-9e86-4bb1-be… [2]: https://spectrum-os.org/git/spectrum/commit/?id=b3cb2511d3612a1d58246fabdc6… [3]: https://spectrum-os.org/git/spectrum/commit/?id=560fd878ba1bbd8df0fe28488e7… [4]: https://github.com/cloud-hypervisor/cloud-hypervisor/pull/7221 [5]: https://github.com/cloud-hypervisor/cloud-hypervisor/pull/7223 [6]: https://lore.kernel.org/qemu-devel/20250609144729.884027-1-aesteve@redhat.c… As for me, aside from reviewing and applying some of Demi's fixes, I updated Spectrum's pinned Nixpkgs[7] and released a new version of the Cloud Hypervisor patches[8], and did the usual round of server updates. Sometimes, updating the pinned Nixpkgs is easy, otherwise, due to a change that looked entirely inconsequential upstream, it can take a lot of work, and this has been one of the latter variety, so I'm glad it's finally done. Since then, and having just returned home from a couple of months of travelling, I've been feeling very tired. A few weeks ago[8], I mentioned that I'd been thinking I might have to calm things down a bit for myself to make sure I'm on the right side of burnout, and I think now is probably the time for that, so I'd expect there won't be much direct progress from me for the next week or two, although I should still be keeping up with emails — feeling like I'd come back to a mountain of them would be much worse than just dealing with them roughly as they come in — and doing what I can to make sure nobody else is blocked. [7]: https://spectrum-os.org/git/spectrum/commit/?id=d0a5f0abdc356f1bba676d489ba… [8]: https://spectrum-os.org/lists/archives/spectrum-discuss/87o6twmg2p.fsf@alys… See you soon!

1 0

[ANNOUNCE] cloud-hypervisor-47.0-spectrum0
by Alyssa Ross 23 Jul '25

23 Jul '25

I've updated Spectrum's Cloud Hypervisor patches (which add virtio-gpu support) to support Cloud Hypervisor 47.0. More information is available at <https://spectrum-os.org/software/cloud-hypervisor/>.

1 0

This Week in Spectrum, 2025-W29
by Alyssa Ross 21 Jul '25

21 Jul '25

The main thing that's been happening this week is a lot of back and forth between Demi and me on PipeWire in VMs. The series is now in its fifth iteration[1], and is shaping up well. I'm happy that we're discussing it on the list so much, because it will make it easy to refer back to in future for context of why decisions were made. For my part, I've been mostly trying to find solutions to problems Demi has been encountering as she gets set up to work on Spectrum. To that end, I fixed a couple of build issues that occurred when KVM wasn't available[2][3], removed the need to remember to run "make clean" after modifying the list of installed packages to avoid ending up with a broken build[4], and spent some time trying to debug why QEMU's virtio-gpu implementation doesn't seem to work nearly as well as crosvm's, despite them being built on the same code. (I haven't made much progress with the latter though — I have a gut feeling it's a memory safety problem, but running Valgrind on QEMU has so far been painful.) I also figured out and fixed the Cloud Hypervisor bug[5] I mentioned last week, so once we update Cloud Hypervisor, the file chooser portal no longer intermittently fails to configure itself, and I did the usual server updates and Nixpkgs maintenance. Yureka has been making progress with forwarding packets out of a network driver VM to host userspace, where we plan to do routing with the aim to later move it into a VM as well. She's also started working on implementing a vhost-user backend, which is how the host userspace router will get packets to and from the application VMs. Sounds like it's still very much in the experimental phase, so there's no public code to link to for any of this as yet. [1]: https://spectrum-os.org/lists/archives/spectrum-devel/a7325ee5-eb7f-458c-9b… [2]: https://spectrum-os.org/git/spectrum/commit/?id=1b33fcabdad8e4b76587848ad29… [3]: https://spectrum-os.org/git/spectrum/commit/?id=89b80752a275a4ca1818a49d86f… [4]: https://spectrum-os.org/git/spectrum/commit/?id=c1e1449b2173913f08828bafc62… [5]: https://github.com/cloud-hypervisor/cloud-hypervisor/commit/ec8fceb4a6a537c… Until next week! I'll be at BornHack until Wednesday morning, so if you are too and want to talk about Spectrum, get in touch. :)

1 0

This Week in Spectrum, 2025-W28
by Alyssa Ross 14 Jul '25

14 Jul '25

This week, I finally finished the networking fixes I've kept mentioning for the last few weeks. Now, if the VMM for the network provider VM exits for whatever reason, it's possible to recover without rebooting the whole system. I've also fixed running AppImages, which had recently regressed, and added a test for that functionality to ensure it doesn't happen again. Along the way came various improvements to Spectrum's integration test framework. As I work on tests and the test framework, each new test is easier than the last, and it's a great feeling! Now, I'm looking into a Cloud Hypervisor bug that causes Spectrum's File Chooser portal implementation to intermittently not work. When that's fixed, I of course already have a Spectrum regression test ready for that too. In this case, because it only doesn't work some of the time, having a test that can (usually) reproduce it has made debugging so much easier than trying to reproduce by hand. I investigated a couple more newly introduced Nixpkgs regressions that affected Spectrum. One was a crosvm segfault on startup, caused by minijail not taking into account that kernel headers could include features not available on the running kernel. They already had a fix, and tagged a new release including it when I asked, but sadly it seems that the regression will be reintroduced next time a new kernel version is added[1]. :( [1]: https://groups.google.com/a/chromium.org/g/minijail/c/7YR25MbzGG0 Demi continued working towards sound in Spectrum. She sent another revision of her patch to run PipeWire in guests[2], and sent a large batch of security fixes upstream to PipeWire[3], some of which have already been accepted[4]. She also continued her work fixing unsoundness in Cloud Hyprevisor with a PR[5] to the rust-vmm vfio crate, and started a discussion in the review thread for the specification of the virtio-media device about its security properties[6]. [2]: https://spectrum-os.org/lists/archives/spectrum-devel/638beeaa-2351-4f51-81… [3]: https://gitlab.freedesktop.org/pipewire/pipewire/-/merge_requests/2423 [4]: https://gitlab.freedesktop.org/pipewire/pipewire/-/merge_requests/2423#note… [5]: https://github.com/rust-vmm/vfio/pull/103 [6]: https://spectrum-os.org/lists/archives/spectrum-devel/67378956-e1dd-4d7f-9b… See you next week!

1 0