- Spectrum Discuss - spectrum-os.org

[ANNOUNCE] cloud-hypervisor-50.0-spectrum0
by Alyssa Ross 20 Dec '25

20 Dec '25

I've updated Spectrum's Cloud Hypervisor patches (which add virtio-gpu support) to support Cloud Hypervisor 50.0. More information is available at <https://spectrum-os.org/software/cloud-hypervisor/>.

1 0

Taking a break from This Week in Spectrum
by Alyssa Ross 19 Nov '25

19 Nov '25

Hi all, We have a funding deadline at the end of the month, so to keep my workload manageable I'm going to take a break from writing This Week in Spectrum until we're done with that. Lots happening though! Userspace router, Flatpak VMs, system updates, persistent storage, less stuff running as root.

1 0

[ANNOUNCE] cloud-hypervisor-49.0-spectrum0
by Alyssa Ross 10 Nov '25

10 Nov '25

I've updated Spectrum's Cloud Hypervisor patches (which add virtio-gpu support) to support Cloud Hypervisor 49.0. More information is available at <https://spectrum-os.org/software/cloud-hypervisor/>.

1 0

This Week in Spectrum, 2025-W44
by Alyssa Ross 04 Nov '25

04 Nov '25

It has been a very busy week. System updates -------------- We finally got the first posting of Demi's work on system updates using systemd-sysupdate[1]! There are still quite a few issues to work through, but it's going ins a promising direction. The most significant of those issues is the following: our current installer, eos-installer[2], works by copying a whole disk image to the target disk. Our combined image allows either booting that image directly, or booting into a system that runs eos-installer with that image. For an updatable system, the partitions that we install need to have room for the system to grow in future, and we also need to have two copies of those partitions for A/B updates. Changing these things causes the image to become unreasonably large. eos-installer supports compressed disk images, but if we did that, we'd lose the ability to directly boot the image to try it out. Demi also points out that eos-installer's approach is incompatible with systems that have e.g. firmware partitions that need to be preserved for the system to boot. Really, then, we're going to need an installer that can work at the partition level rather than the disk image level, so minimally sized partition images can be copied into future-proof bigger partitions in the installed system, B partitions can be created empty without needing a template, and existing partitions on the system can be left alone. This will also mean we get natural dual boot support. I was disappointed to learn that an off the shelf installer that works in this way, analogous to eos-installer, doesn't seem to exist, though. I asked what GNOME OS does, because I knew they previously used eos-installer and must have encountered this problem, and what we were able to figure out on Matrix is that they're in the process of writing a new installer for themselves. I plan to reach out to them to find out about their plans and see if this is something that could also be shared between image-based distros. In the meantime, though, we'll probably just get rid of the combined image, and have one non-updatable live image for testing Spectrum, and an installer image that installs a compressed disk image using eos-installer. [1]: https://spectrum-os.org/lists/archives/spectrum-devel/20251029-updates-v1-0… [2]: https://github.com/endlessm/eos-installer "Try Spectrum" test ------------------- The updates submission, which took the approach of compressing the images and breaking the "Try Spectrum" function of the combined live/installer image, made me realise that there was no test coverage for that, so I wrote an integration test for it[3]. This was shortly before we realised dropping the combined image for the time being is probably inevitable, but even so it led to lots of nice improvements to the integration test infrastructure that will still be useful going forward. [3]: https://spectrum-os.org/git/spectrum/commit/?id=e0f5aad7901d150b1abe381ffd9… udev ---- I changed the script that assigns network devices to VMs to be idempotent[4], which was the last thing that needed to happen before Demi's patches to switch the host system to use udev[5] can be applied, so that should happen very soon. [4]: https://spectrum-os.org/git/spectrum/commit/?id=1d666582892fbdf812662b4490c… [5]: https://spectrum-os.org/lists/archives/spectrum-devel/20251103-udev-v5-0-5c… nixpkgs updates --------------- I finished bisecting the kernel regression we encountered when trying to update Nixpkgs last week. A fix is on the way[6], and I was also able to identify a workaround that could let us keep up to date in the meantime[7]. With this, the update passed all tests and I committed it. A few days later though, it was reported to me that the installer no longer started. I'm bisecting this now, and once we've figured out how to fix it, I'll be able to use all those nice integration test framework improvements to write a regression test for this, too. I used to do more manual testing of Nixpkgs updates, but really to be able to deliver updates at the pace we want, we're going to have to rely a lot less on manual testing and a lot more on automated tests, so going forward I'm going to be trying to get Nixpkgs updated more frequently, with less manual testing, and a focus on adding automated regression tests when something goes wrong. This will likely mean more of this sort of issue in the short term while we build up our test coverage, but a more robust approach to updates in the long term when we've built up our test suite. Yureka has also been continuing to work on identifying Spectrum-related build regressions in Nixpkgs quickly, which will also be important to enable a more rapid pace of updates. [6]: https://lore.kernel.org/netdev/20251030144438.7582-1-minhquangbui99@gmail.c… [7]: https://spectrum-os.org/git/spectrum/commit/?id=9a03e3ea12a40e03735132ed2ed… What a pace, right!?

1 0

This Week in Spectrum, 2025-W43
by Alyssa Ross 27 Oct '25

27 Oct '25

Hi! I'm back after being unwell for a couple of weeks. As such, I'll be covering things that have happened since the last update. Demi fixed the BPF x86_64 build failure mentioned last time by avoiding the use of userspace headers in BPF compilation environments[1]. While she was at it, she also submitted a refactor and possible small performance improvement, which is still awaiting review. [1]: https://spectrum-os.org/git/spectrum/commit/?id=db54efac07deee32f39bdec4e4b… [2]: https://spectrum-os.org/lists/archives/spectrum-devel/20251021-fix-forwarde… Yureka fixed some recently introduced build regressions in Nixpkgs[3], including a fix for iproute2 which was applied upstream[4], and then submitted a patch to update Spectrum's pinned Nixpkgs[5]. I am absolutely delighted with this, because it's the first time anybody else has taken on this regular maintenance work. Unfortunately, when I tested the Nixpkgs update, I found that the networking integration test no longer passed. A still-running bisect indicates this is probably due to the update to Linux 6.17. If so, it's probably been caught by somebody else and fixed already, so which means hopefully the fix will just be to update to an even newer Nixpkgs and cross our fingers no new build regressions have been introduced in the meantime. Yureka has also been working on automatically testing Spectrum builds against newer Nixpkgs versions, which I hope might pave the way for semi-automatic updates at some point, and/or automated notifications of regressions. [3]: https://spectrum-os.org/lists/archives/spectrum-devel/82249ddc-ae1a-4e3a-a6… [4]: https://git.kernel.org/pub/scm/network/iproute2/iproute2.git/commit/?id=25b… [5]: https://spectrum-os.org/lists/archives/spectrum-devel/20251012222756.20947-… I've been working on running Flatpak applications in VMs. This is a bit more complicated than the AppImages we already support, because rather than relying on the system to provide libraries, Flatpaks are installed to a shared OSTree repository, so that they can be hermetic while still not duplicating huge amounts of common libraries between applications. We don't want to just pass through the whole Flatpak repository to application VMs, because then any application would be able to see which other applications were installed, so to do Flatpak VMs we need to determine which paths are actually needed by an application, and pass through just those. I've written a program that sets up bind mounts for a Flatpak application and its runtime to pass through to a VM (but not extensions yet), and also the glue in the Spectrum application VM to actually run a passed through Flatpak. I haven't had a chance to do a full end to end test yet, but I think everything should just about work. Thanks to work from Demi[6], file modes in Spectrum images are now fixed, rather than being affected by the build environment. As well as being an important reproducibility fix, this will be important to start running things as non-root users, where modes actually become relevant. [6]: https://spectrum-os.org/git/spectrum/commit/?id=a972ba0ce2be8fb32f4b44a7080… Demi has been working on implementing OS updates using systemd-sysupdate, and just got to the point where everything works! Hopefully we'll see that on the list soon.

1 0

This Week in Spectrum, 2025-W40
by Alyssa Ross 06 Oct '25

06 Oct '25

It was quite a week. Yureka and I were at XDC from Monday until Wednesday. My talk, "An upstream-first approach to application virtualization", was on Tuesday, and the video is now online[1]. I didn't learn that much at the conference that was directly relevant to Spectrum, but I was interested to learn from a lightning talk that kmscon has been revived and was recently repackaged in Fedora[2]. I'm still getting used to public speaking again, so the talk wasn't as smooth as I'd have liked, but it was okay. After XDC, I stopped by the Systems Meetup Dresden[3] and did a lighting talk. This one went much better, but wasn't recorded, so you'll just have to believe me. While we were away, All Systems Go[4] (AIUI basically the systemd conference) took place in Berlin, and that meant a lot of Linux mobile/desktop people were in town who are not normally. We were able to get together over the following days, and in particular we had some good conversations about what the future of Flatpak/Portals might look like. It's great to be in the room for this sort of thing because it means Spectrum's needs can be considered from the start. The designs being considered would solve our biggest pain points, so I'm optimistic, but there's of course the question of momentum / developer attention to make it happen. Hopefully we can contribute to that. [1]: https://video.tuwien.ac.at/events/xdc/v/Ow9xtIyJeTt?t=01h13m49s [2]: https://github.com/Aetf/kmscon/ [3]: https://ukvly.org/ [4]: https://all-systems-go.io/ In between the talks at XDC (and during some of the talks about graphics drivers I had no hope of understanding), I was pleasantly surprised to find myself in a particularly productive state — it must have been the conference atmosphere. It didn't make sense to try to work on anything big, so I went through my backlog of small Spectrum improvements. Image rebuild times during development have gone down by a further non-trivial two seconds on a fast laptop, more code has been integrated into the single Meson build system I aspire to, and a few other smaller things. I also managed to completely clear my review queue, although it of course started filling up again immediately after. Demi's patch to automate the lists of files to include in images is finally finished and committed, and her udev changes made progress, but they've been blocked on further review from me, since with so much stuff on I haven't had a chance since leaving XDC. She's also made a bit of progress on system updates using systemd-sysupdate, about which we'll hopefully have more to say soon. Finally, we've been discussing for a couple of days now what to do about a recently introduced build failure on x86_64. I'm already late getting this update out, so I'm not going to go into detail, but basically it's very unclear what the right way to build BPF programs that include kernel UAPI headers is, or if there even is one. The common way of using a distro's linux-headers/libc package is not correct, and likely to lead to either loud compilation failures (as we're currently experiencing) or silent hidden bugs[5]. If we can't find a good solution, we can fall back to copying the definitions we need out of the headers, but I'm hoping it won't come to that. Investigation continues. This has also highlighted the need for pre-push testing on all supported architectures — the reason this build failure got past me is that it only happens on x86_64, and I was working on aarch64. I'll try to get that addressed. [5]: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org… Hopefully next week will be less busy, and I'll be able to get my head down a bit and do some programming.

1 0

This Week in Spectrum, 2025-W39
by Alyssa Ross 29 Sep '25

29 Sep '25

Sent this to the wrong list at first, oops! I've got a lot going on. >.< Alyssa Ross <hi(a)alyssa.is> writes: > The big news this week is that review of Yureka's XDP forwarding program > is complete, and has now been committed to the Spectrum repo[1]. We > also saw the usual couple of small cleanups from Demi, several new > versions of her script to automatically generate file lists for > Makefiles[2], and just a few hours ago, a new version of the udev > switch[3]. > > [1]: https://spectrum-os.org/git/spectrum/commit/?id=535b6bec89a1aae6b0ee9579f9b… > [2]: https://spectrum-os.org/lists/archives/spectrum-devel/20250927-genfiles-v6-… > [3]: https://spectrum-os.org/lists/archives/spectrum-devel/20250924-udev-v2-0-60… > > On Friday I gave a talk at the Qubes OS Summit called "Rethinking the > Compartmentalization Experience"[4]. Tomorrow (Tuesday, 08:30 UTC) I'll > be doing another talk at XDC called "An Upstream-first Approach to > Application Virtualization", which will be streamed[5], and on the way > back home I'll be stopping by Systems Meetup Dresden[6], where I'll do a > lightning talk. > > [4]: https://www.youtube.com/live/HwisqKFEQ0g?feature=shared&t=16880 > [5]: https://live.video.tuwien.ac.at/watch?l=hsJNWfHvxZPnonNLxYNU5Z > [6]: https://ukvly.org/

1 0

This Week in Spectrum, 2025-W38
by Alyssa Ross 23 Sep '25

23 Sep '25

Whoops — I'm a bit late again this week. Was really busy yesterday. Given how late I was last week though, there's not all that much anyway. As usual, we acquired some build fixes and development environments from Demi. Some small ones were applied — my favourite is making sure that early kernel logs don't get lost in the development environment[1] — and bigger things like automatically generating file lists for images[2], and build performance improvements[3] are still under discussion. We also continued discussing the udev switch, and I expect to see a new version of that posted soon. [1]: https://spectrum-os.org/git/spectrum/commit/?id=1c471fc4191958dfbffbdb5ad2d… [2]: https://spectrum-os.org/lists/archives/spectrum-devel/20250903-genfiles-v1-… [3]: https://spectrum-os.org/lists/archives/spectrum-devel/20250919-less-dirname… [4]: https://spectrum-os.org/lists/archives/spectrum-devel/20250913-udev-v1-0-ea… I spent some time debugging a Rutabaga crash in QEMU. When I submitted a fix, I learned that it had already been fixed a few days prior[5], but I also discovered that Rutabaga has been extracted from crosvm to a standalone project, which I think is fantastic news. They're even adding a proper Meson build (no wrapped Cargo)[6]. Excited to review that. [5]: https://github.com/magma-gpu/rutabaga_gfx/pull/9 [6]: https://github.com/magma-gpu/rutabaga_gfx/pull/13 Just as I've been writing this, Yureka has submitted a new version of the XDP forwarding series[7]. We're aiming to have the XDP forwarder program committed soon, but actually using it will have to wait for the rest of the new network stack, and for a couple of Nixpkgs fixes[8][9], which we also made some progress on this week. [7]: https://spectrum-os.org/lists/archives/spectrum-devel/20250923132012.28013-… [8]: https://github.com/NixOS/nixpkgs/pull/444996 [9]: https://github.com/NixOS/nixpkgs/pull/436206 I've been working on a couple of talks I'll be giving soon. The first is "Rethinking the compartmentalization experience" at the Qubes OS Summit in Berlin on Friday at 12:30 UTC[10], and the other is "An upstream-first approach to application virtualization" at XDC in Vienna on Tuesday at 08:30 UTC. These are actually the first talks I'll ever have given about Spectrum, because I've historically wanted to avoid publicity until I was confident Spectrum would be able to deliver on its promises. As we're approaching an initial release, this is becoming less of a concern, and by now we've learned a lot that's going to be useful to share regardless. As always, get in touch if you want to chat at either of these events. [10]: https://cfp.3mdeb.com/qubes-os-summit-2025/talk/UPMP38/ [11]: https://indico.freedesktop.org/event/10/contributions/436/

1 0

This Week in Spectrum, 2025-W37
by Alyssa Ross 17 Sep '25

17 Sep '25

Decided I'd rather get this out late than miss another week. There was no This Week in Spectrum for 2025-W36, because I was too tired after KVM Forum and NixCon to get it out on time, and in the last week I was in the hospital for several days for various tests, including when I'd usually send this out. Anyway, with two and a bit weeks to cover, this is going to be a long one. KVM Forum --------- Was nice to put faces on names from mailing lists — will make it easier to approach people online to discuss new KVM things, I think. The most interesting talk for me was "IOMMU in rust-vmm, and new FUSE+VDUSE use cases"[1], in which I learned: • virtiofsd is going to get support for additional backends, e.g. NFS and cloud storage. • A vhost-user↔VDUSE bridge is in development, which will allow a vhost-user device to be used by the kernel running that device. This raises the possibility of moving device implementations out of the host kernel. • virtio-vhost-user didn't end up getting through standardization and implementation not for any fundamental reason, but just because nobody was sufficiently motivated to make it happen. This also came up again just yesterday on the virtualization list[2]. It would be nice to see some momentum again towards moving device implementations into VMs. [1]: https://lore.kernel.org/virtualization/CAJSP0QWV+=+v9Z0wU9qJcdToDKBnWiGKzOV… [2]: https://www.youtube.com/watch?v=qsFc234tzz4&pp=ygUOaW9tbXUgcnVzdC12bW0%3D NixCon ------ KVM Forum and NixCon partially overlapped, so I only managed to make the second half of the latter. I didn't see any talks, but had some good conversations, mostly in the areas of documentation (which I hope we'll get a lot of work done on next year) and funding / business model stuff, which will be very important to get right to keep the project going after NGI ends. Speaking of which: NGI Zero Commons Business Circle -------------------------------- As recipients of an NGI Zero grant, we have access to various support services[3]. One of these is called "Business Circle", and provides business consulting and mentorship. I had an initial call with them after I got home after NixCon. Not much to report yet, but it does sound like we'll have lots of support available with legal etc. stuff if we need it to take the project forward. [3]: https://nlnet.nl/NGI0/services/ Cloud Hypervisor ---------------- Demi and I are currently working on sandboxing services on the Spectrum host — best effort defense in depth, since these services are part of our TCB anyway. One easy win there, I thought, would have been enabling Cloud Hypervisor's support for locking itself down with landlock, preventing it from accessing any non-allowlisted files. This should have been just a case of turning it on, and passing it a couple of extra paths to allow access to so VMs were allowed to have new devices passed through to them after creation, but I ended up finding quite a few bugs in the implementation in Cloud Hypervisor which had to be fixed first. The default rules were overly strict, resulting in VSOCK being non-functional if enabled[4], and in it failing to start at all on many aarch64 machines[5], and I also found that if landlock was requested, but the kernel didn't support it, Cloud Hypervisor would just start anyway, without landlock[6]. I've fixed all of these upstream now, and conveniently, a new Cloud Hypervisor release was scheduled for a couple of days afterwards, so these changes have already made their way into Spectrum. I haven't committed the Spectrum-side landlock stuff though, as I want to test it a bit more first. [4]: https://github.com/cloud-hypervisor/cloud-hypervisor/pull/7334 [5]: https://github.com/cloud-hypervisor/cloud-hypervisor/pull/7331 [6]: https://github.com/cloud-hypervisor/cloud-hypervisor/pull/7335 rust-vmm -------- The next step towards sound support in Spectrum is running vhost-device-sound on the host, but we can't currently do this robustly (and without polling), because vhost-device-sound doesn't support any form of socket activation or readiness notification. I've been working on adding support for the former, but it turned out not to be possible to implement with the current interface of the vhost-user-backend crate. In fact, I discovered, all the vhost-device devices create a new socket each time they're ready to accept a connection, meaning that, after the end of a connection, there's a window where a new connection can't be accepted. The right thing to do here is to reuse a socket between requests, and making that change will conveniently also make it straightforward to optionally pass in a socket provided by the service manager rather than creating it in the vhost-device program, so I changed the vhost-user-backend API[7], and am now waiting for a release, at which point I'll submit my changes to vhost-device-sound to update vhost-user-backend, fix the race between connections, and add support for using an externally provided socket. [7]: https://github.com/rust-vmm/vhost/pull/321 udev ---- Demi has been working on switching the host system from mdevd to systemd-udevd, with the motivation being that systemd-udevd comes with a big database of device quirks, which are likely to be beneficial to us in our aim of broad hardware support. This was originally done as part of her experiment with booting Spectrum with systemd[8], but has now been extracted into a standalone series[9], which is currently awaiting review. One of the complications of this, as I understand it, is that unlike every other service we run, we're not confident it's okay to start services that depend on udev as soon as the udev control socket is created, so we needed an adapter from systemd's readiness notification protocol to s6's. [8]: https://spectrum-os.org/lists/archives/spectrum-devel/20250904-systemd-v1-0… [9]: https://spectrum-os.org/lists/archives/spectrum-devel/20250913-udev-v1-0-ea… xdp-forwarder ------------- Yureka submitted a new version of her program for forwarding packets between two network interfaces[10], which will be part of our new networking stack. The idea here is that we can get the program reviewed and committed without having to wait for everything to be ready to switch to it. There were some more review comments from Demi and me, but nothing major, so I expect we'll see this part of the networking work finished soon. [10]: https://spectrum-os.org/lists/archives/spectrum-devel/20250906141228.235763… Ergonomics and other small improvements --------------------------------------- Demi has been submitting a lot of small fixes and improvements for the Spectrum development experience. Between conferences and the hospital I haven't had a chance to look at many of these yet, but I did apply a few of the most straightforward ones already. Of course, there have also been all the usual recurring things like server updates, Nixpkgs reviews and contributions, and so on, but this is getting far too long already, so I'm going to leave it here. Hopefully at the end of this week I'll be able to get back to regular updates of reasonable length…

1 0

[ANNOUNCE] cloud-hypervisor-48.0-spectrum0
by Alyssa Ross 14 Sep '25

14 Sep '25

I've updated Spectrum's Cloud Hypervisor patches (which add virtio-gpu support) to support Cloud Hypervisor 48.0. More information is available at <https://spectrum-os.org/software/cloud-hypervisor/>.

1 0