June 2025 - Spectrum Development

Re: [PATCH] Documentation: request patches not to be PGP signed
by Alyssa Ross 28 Jun '25

28 Jun '25

This patch has been committed as dc112f871ccf037c918b90d813b807259b4a3ba1, which can be viewed online at https://spectrum-os.org/git/spectrum/commit/?id=dc112f871ccf037c918b90d813b…. This is an automated message. Send comments/questions/requests to: Alyssa Ross <hi(a)alyssa.is>

1 0

[PATCH 0/8] Change serial getty instances dynamically
by Alyssa Ross 28 Jun '25

28 Jun '25

The host system is supposed to provide a shell on whatever serial devices are configured as the active kernel consoles. Previously, the system made the assumption that the active console would never change after boot, but in reality devices can be added and removed later than that. One situation in which this happened is when a serial device was configured whose driver was not built in to the kernel, but loaded as a module after the console was checked by rc.init. This series fixes that by creating and destroying instances of the serial-getty service whenever the active kernel consoles change, and adds a regression test. Since this is the second test that involves running a full Spectrum system in a VM, most of the series has ended up being refactoring the first such test, so that code can be shared with this new one. Yureka, I don't know whether this is the issue you were encountering when you tried debugging the Weston issue over serial, but maybe worth a try? I suspect not though, because if I recall correctly you weren't even getting kernel logs? This was the only issue I could find trying to reproduce the problem in QEMU. Alyssa Ross (8): host/rootfs: poll active consoles for serial getty release/checks/integration: init from networking release/checks/integration: use default Meson timeout release/checks/integration: extract config struct release/checks/integration: run from tmpdir release/checks/integration: name QEMU arg pointers release/checks/integration: extract library release/checks/integration: test late serial host/rootfs/Makefile | 1 + host/rootfs/default.nix | 8 +- .../service/serial-getty-generator/run | 43 +++ host/rootfs/etc/s6-linux-init/scripts/rc.init | 9 - release/checks/default.nix | 4 +- release/checks/integration/default.nix | 88 +++++ release/checks/integration/late-serial.c | 15 + release/checks/integration/lib.c | 221 ++++++++++++ release/checks/integration/lib.h | 20 ++ release/checks/integration/meson.build | 21 ++ release/checks/integration/meson_options.txt | 6 + release/checks/integration/networking.c | 138 +++++++ release/checks/networking/default.nix | 62 ---- release/checks/networking/test.c | 337 ------------------ release/checks/pkg-tests.nix | 2 +- 15 files changed, 560 insertions(+), 415 deletions(-) create mode 100755 host/rootfs/etc/s6-linux-init/run-image/service/serial-getty-generator/run create mode 100644 release/checks/integration/default.nix create mode 100644 release/checks/integration/late-serial.c create mode 100644 release/checks/integration/lib.c create mode 100644 release/checks/integration/lib.h create mode 100644 release/checks/integration/meson.build create mode 100644 release/checks/integration/meson_options.txt create mode 100644 release/checks/integration/networking.c delete mode 100644 release/checks/networking/default.nix delete mode 100644 release/checks/networking/test.c base-commit: 42887a0331fd058ee0930fd5209fc3265565c3a8 -- 2.49.0

2 14

Virtio-IOMMU interrupt remapping design
by Demi Marie Obenour 17 Jun '25

17 Jun '25

Virtio-IOMMU interrupt remapping turned out to be much harder than I realized. The main problem is that interrupt remapping is set up very early in boot. In fact, Linux calls the interrupt remapping probe function from the APIC initialization code: x86_64_probe_apic -> enable_IR_x2apic -> irq_remapping_prepare(). This is almost certainly much before PCI has been initialized. Also, the order in which devices will be initialized is not something Linux guarantees at all, which is a problem because interrupt remapping must be initialized before drivers start setting up interrupts. Otherwise, the interrupt remapping table won't include entries for already-existing interrupts, and things will either break badly, not get the benefit of interrupt remapping security-wise, or both. The reason I expect this doesn't cause problems for address translation is that the IOMMU probably starts in bypass mode by default, meaning that all DMA is permitted. If the IOMMU is only used by VFIO or IOMMUFD, it will not be needed until userspace starts up, which is after the IOMMU has been initialized. This isn't ideal, though, as it means that kernel drivers operate without DMA protection. Is a paravirtualized IOMMU with interrupt remapping something that makes sense? Absolutely! However, the IOMMU should be considered a platform device that must be initialized very early in boot. Using virtio-IOMMU with MMIO transport as the interface might be a reasonable option, but the IOMMU needs to be enumerated via ACPI, device tree, or kernel command line argument. This allows it to be brought up before anything capable of DMA is initialized. Is this the right path to go down? What do others think about this? -- Sincerely, Demi Marie Obenour (she/her/hers)

3 8

[RFC PATCH] Run PipeWire and WirePlumber in the VMs
by Demi Marie Obenour 17 Jun '25

17 Jun '25

WirePlumber is completely overkill as a session manager here, and ideally a trivial session manager would be used instead. Also, this code has seen absolutely no testing beyond "it builds", not least because there is no way to add a virtio-sound device to Cloud Hypervisor yet and I haven't tried using QEMU TCG. Signed-off-by: Demi Marie Obenour <demiobenour(a)gmail.com> --- img/app/default.nix | 5 ++++- img/app/etc/fstab | 5 +++-- .../etc/s6-rc/app/dependencies.d/wireplumber | 0 img/app/etc/s6-rc/directories/type | 1 + img/app/etc/s6-rc/directories/up | 11 ++++++++++ .../s6-rc/pipewire/dependencies.d/directories | 0 img/app/etc/s6-rc/pipewire/notification-fd | 1 + img/app/etc/s6-rc/pipewire/run | 20 +++++++++++++++++++ img/app/etc/s6-rc/pipewire/type | 1 + .../dependencies.d/directories | 0 img/app/etc/s6-rc/wayland-proxy-virtwl/run | 11 ---------- .../etc/s6-rc/wireplumber/dependencies.d/dbus | 0 .../s6-rc/wireplumber/dependencies.d/pipewire | 0 img/app/etc/s6-rc/wireplumber/run | 16 +++++++++++++++ img/app/etc/s6-rc/wireplumber/type | 1 + 15 files changed, 58 insertions(+), 14 deletions(-) create mode 100644 img/app/etc/s6-rc/app/dependencies.d/wireplumber create mode 100644 img/app/etc/s6-rc/directories/type create mode 100644 img/app/etc/s6-rc/directories/up create mode 100644 img/app/etc/s6-rc/pipewire/dependencies.d/directories create mode 100644 img/app/etc/s6-rc/pipewire/notification-fd create mode 100644 img/app/etc/s6-rc/pipewire/run create mode 100644 img/app/etc/s6-rc/pipewire/type create mode 100644 img/app/etc/s6-rc/wayland-proxy-virtwl/dependencies.d/directories create mode 100644 img/app/etc/s6-rc/wireplumber/dependencies.d/dbus create mode 100644 img/app/etc/s6-rc/wireplumber/dependencies.d/pipewire create mode 100644 img/app/etc/s6-rc/wireplumber/run create mode 100644 img/app/etc/s6-rc/wireplumber/type diff --git a/img/app/default.nix b/img/app/default.nix index a196205..32ddb67 100644 --- a/img/app/default.nix +++ b/img/app/default.nix @@ -8,7 +8,7 @@ pkgsStatic.callPackage ( { lib, stdenvNoCC, runCommand, writeClosure , erofs-utils, jq, s6-rc, util-linux , busybox, cacert, dejavu_fonts, execline, kmod, linux_latest, mdevd, s6 -, s6-linux-init, xdg-desktop-portal-spectrum, +, s6-linux-init, xdg-desktop-portal-spectrum }: let @@ -48,6 +48,9 @@ let pkgs.xwayland pkgs.xdg-desktop-portal pkgs.xdg-desktop-portal-gtk + # Depends on pulseaudio libs + pkgs.pipewire + pkgs.wireplumber ]; })).fhsenv; in diff --git a/img/app/etc/fstab b/img/app/etc/fstab index a95088b..40aa3bd 100644 --- a/img/app/etc/fstab +++ b/img/app/etc/fstab @@ -1,7 +1,8 @@ # SPDX-License-Identifier: CC0-1.0 # SPDX-FileCopyrightText: 2020-2022 Alyssa Ross <hi(a)alyssa.is> proc /proc proc defaults 0 0 -devpts /dev/pts devpts defaults,gid=4,mode=620 0 0 +devpts /dev/pts devpts defaults,gid=4,mode=620 0 0 tmpfs /dev/shm tmpfs defaults 0 0 sysfs /sys sysfs defaults 0 0 -tmpfs /tmp tmpfs defaults 0 0 +tmpfs /tmp tmpfs defaults,mode=1755 0 0 +tmpfs /run tmpfs defaults 0 0 diff --git a/img/app/etc/s6-rc/app/dependencies.d/wireplumber b/img/app/etc/s6-rc/app/dependencies.d/wireplumber new file mode 100644 index 0000000..e69de29 diff --git a/img/app/etc/s6-rc/directories/type b/img/app/etc/s6-rc/directories/type new file mode 100644 index 0000000..bdd22a1 --- /dev/null +++ b/img/app/etc/s6-rc/directories/type @@ -0,0 +1 @@ +oneshot diff --git a/img/app/etc/s6-rc/directories/up b/img/app/etc/s6-rc/directories/up new file mode 100644 index 0000000..2396fad --- /dev/null +++ b/img/app/etc/s6-rc/directories/up @@ -0,0 +1,11 @@ +#!/bin/execlineb -P +# SPDX-License-Identifier: EUPL-1.2+ +# SPDX-FileCopyrightText: 2023-2024 Alyssa Ross <hi(a)alyssa.is> +# +# Directory creation (if it's copyrightable): +# SPDX-License-Identifier: MIT +# SPDX-FileCopyrightText: 2022 Unikie + +if { mkdir -m 1755 /tmp/.X11-unix } +if { mkdir -m 0755 /run/user } +if { mkdir -m 0700 /run/user/0 } diff --git a/img/app/etc/s6-rc/pipewire/dependencies.d/directories b/img/app/etc/s6-rc/pipewire/dependencies.d/directories new file mode 100644 index 0000000..e69de29 diff --git a/img/app/etc/s6-rc/pipewire/notification-fd b/img/app/etc/s6-rc/pipewire/notification-fd new file mode 100644 index 0000000..7ed6ff8 --- /dev/null +++ b/img/app/etc/s6-rc/pipewire/notification-fd @@ -0,0 +1 @@ +5 diff --git a/img/app/etc/s6-rc/pipewire/run b/img/app/etc/s6-rc/pipewire/run new file mode 100644 index 0000000..c0b55a1 --- /dev/null +++ b/img/app/etc/s6-rc/pipewire/run @@ -0,0 +1,20 @@ +#!/bin/execlineb -P +# SPDX-License-Identifier: EUPL-1.2+ +# SPDX-FileCopyrightText: 2023-2024 Alyssa Ross <hi(a)alyssa.is> +# SPDX-FileCopyrightText: 2025 Demi Marie Obenour <demiobenour(a)gmail.com> + +s6-ipcserver-socketbinder -B /run/user/0/pipewire-0 +fdmove -c 3 0 + +s6-ipcserver-socketbinder -B /run/user/0/pipewire-0-manager +fdmove -c 4 0 + +redirfd -r 0 /dev/null + +# Notify readiness. +if { fdmove 1 5 echo } +fdclose 5 + +export LISTEN_FDS 2 +getpid LISTEN_PID +pipewire diff --git a/img/app/etc/s6-rc/pipewire/type b/img/app/etc/s6-rc/pipewire/type new file mode 100644 index 0000000..5883cff --- /dev/null +++ b/img/app/etc/s6-rc/pipewire/type @@ -0,0 +1 @@ +longrun diff --git a/img/app/etc/s6-rc/wayland-proxy-virtwl/dependencies.d/directories b/img/app/etc/s6-rc/wayland-proxy-virtwl/dependencies.d/directories new file mode 100644 index 0000000..e69de29 diff --git a/img/app/etc/s6-rc/wayland-proxy-virtwl/run b/img/app/etc/s6-rc/wayland-proxy-virtwl/run index 7b80343..c1e0e08 100755 --- a/img/app/etc/s6-rc/wayland-proxy-virtwl/run +++ b/img/app/etc/s6-rc/wayland-proxy-virtwl/run @@ -1,17 +1,6 @@ #!/bin/execlineb -P # SPDX-License-Identifier: EUPL-1.2+ # SPDX-FileCopyrightText: 2023-2024 Alyssa Ross <hi(a)alyssa.is> -# -# Directory creation (if it's copyrightable): -# SPDX-License-Identifier: MIT -# SPDX-FileCopyrightText: 2022 Unikie - -foreground { mkdir /tmp/.X11-unix } -foreground { mkdir /run/user } -foreground { - umask 077 - mkdir /run/user/0 -} s6-ipcserver-socketbinder -B /run/user/0/wayland-0 fdmove -c 3 0 diff --git a/img/app/etc/s6-rc/wireplumber/dependencies.d/dbus b/img/app/etc/s6-rc/wireplumber/dependencies.d/dbus new file mode 100644 index 0000000..e69de29 diff --git a/img/app/etc/s6-rc/wireplumber/dependencies.d/pipewire b/img/app/etc/s6-rc/wireplumber/dependencies.d/pipewire new file mode 100644 index 0000000..e69de29 diff --git a/img/app/etc/s6-rc/wireplumber/run b/img/app/etc/s6-rc/wireplumber/run new file mode 100644 index 0000000..6d6a549 --- /dev/null +++ b/img/app/etc/s6-rc/wireplumber/run @@ -0,0 +1,16 @@ +#!/bin/execlineb -P +# SPDX-License-Identifier: EUPL-1.2+ +# SPDX-FileCopyrightText: 2023-2024 Alyssa Ross <hi(a)alyssa.is> +# SPDX-FileCopyrightText: 2025 Demi Marie Obenour <demiobenour(a)gmail.com> +# +# Directory creation (if it's copyrightable): +# SPDX-License-Identifier: MIT +# SPDX-FileCopyrightText: 2022 Unikie + +foreground { mkdir /tmp/.X11-unix } +foreground { mkdir /run/user } +foreground { + umask 077 + mkdir /run/user/0 +} + diff --git a/img/app/etc/s6-rc/wireplumber/type b/img/app/etc/s6-rc/wireplumber/type new file mode 100644 index 0000000..5883cff --- /dev/null +++ b/img/app/etc/s6-rc/wireplumber/type @@ -0,0 +1 @@ +longrun -- Sincerely, Demi Marie Obenour (she/her/hers)

1 0

Virtio interrupt remapping
by Demi Marie Obenour 16 Jun '25

16 Jun '25

I’m working on virtio-IOMMU interrupt remapping for Spectrum OS [1], and am running into a problem. All of the current interrupt remapping drivers use __init code during initialization, and I’m not sure how to plumb the struct virtio_device * into the IOMMU initialization code. What is the proper way to do this, where “proper” means that it doesn’t do something disgusting like “stuff the virtio device in a global variable”? -- Sincerely, Demi Marie Obenour (she/her/hers)

3 4

Verified boot and filesystem choices
by Demi Marie Obenour 15 Jun '25

15 Jun '25

Bcachefs is not very stable right now, and BTRFS is not a good choice from a verified boot perspective. f2fs is what is used in Android and ext4 is used in Chromebooks, so they at least have the backing of Google's security team when it comes to vulnerabilities involving maliciously crafted filesystem images. BTRFS doesn't. The reason this matters for Spectrum is that verified boot aims to prevent system compromise from persisting across reboots, and an attacker who has compromised a Spectrum system can craft whatever image they want on the writable volume. Would it make sense to use f2fs or ext4? That means no reflinks and no snapshots, which would be annoying at least. Another option might be to use FUSE for the writable volume, with kernel filesystems only used for the (signed and dm-verity protected) root volume. This is the only option supported by Linux's upstream maintainers, who (with the notable exception of Kent Overstreet) appear to have no interest in hardening filesystems against maliciously crafted images. -- Sincerely, Demi Marie Obenour (she/her/hers)

2 4

[PATCH] vm: link /sbin to /usr/bin
by Alyssa Ross 09 Jun '25

09 Jun '25

This is necessary because the kernel tries to load modules by running /sbin/modprobe. This should fix Intel wireless cards in netvm. It's then important to disable Busybox init because /sbin/init takes precedence over our /etc/init. Signed-off-by: Alyssa Ross <hi(a)alyssa.is> --- img/app/Makefile | 2 +- img/app/default.nix | 1 + img/app/sbin | 1 + vm/sys/net/Makefile | 2 +- vm/sys/net/default.nix | 1 + vm/sys/net/sbin | 1 + 6 files changed, 6 insertions(+), 2 deletions(-) create mode 120000 img/app/sbin create mode 120000 vm/sys/net/sbin diff --git a/img/app/Makefile b/img/app/Makefile index 5ea023e..c827fd3 100644 --- a/img/app/Makefile +++ b/img/app/Makefile @@ -60,7 +60,7 @@ VM_FIFOS = etc/s6-linux-init/run-image/service/s6-linux-init-shutdownd/fifo # These are separate because they need to be included, but putting # them as make dependencies would confuse make. -VM_LINKS = bin etc/ssl/certs/ca-certificates.crt +VM_LINKS = bin etc/ssl/certs/ca-certificates.crt sbin VM_BUILD_FILES = build/etc/s6-rc diff --git a/img/app/default.nix b/img/app/default.nix index 6e423f6..a196205 100644 --- a/img/app/default.nix +++ b/img/app/default.nix @@ -24,6 +24,7 @@ let CONFIG_FEATURE_IP_ADDRESS y CONFIG_FEATURE_IP_LINK y CONFIG_FEATURE_IP_ROUTE y + CONFIG_INIT n CONFIG_IP y ''; }) diff --git a/img/app/sbin b/img/app/sbin new file mode 120000 index 0000000..1e881ed --- /dev/null +++ b/img/app/sbin @@ -0,0 +1 @@ +usr/bin \ No newline at end of file diff --git a/vm/sys/net/Makefile b/vm/sys/net/Makefile index 35da669..436ca6e 100644 --- a/vm/sys/net/Makefile +++ b/vm/sys/net/Makefile @@ -43,7 +43,7 @@ VM_DIRS = dev etc/s6-linux-init/env run proc sys var/lib/connman # These are separate because they need to be included, but putting # them as make dependencies would confuse make. -VM_LINKS = bin lib var/run +VM_LINKS = bin lib sbin var/run VM_BUILD_FILES = build/etc/s6-rc diff --git a/vm/sys/net/default.nix b/vm/sys/net/default.nix index 69f9564..b5873eb 100644 --- a/vm/sys/net/default.nix +++ b/vm/sys/net/default.nix @@ -22,6 +22,7 @@ let (busybox.override { extraConfig = '' CONFIG_DEPMOD n + CONFIG_INIT n CONFIG_INSMOD n CONFIG_LSMOD n CONFIG_MODINFO n diff --git a/vm/sys/net/sbin b/vm/sys/net/sbin new file mode 120000 index 0000000..1e881ed --- /dev/null +++ b/vm/sys/net/sbin @@ -0,0 +1 @@ +usr/bin \ No newline at end of file base-commit: b5dab0eea25e24531ac856985b8f79ef38c06366 -- 2.49.0

3 4

Camera virtualization in Spectrum
by Demi Marie Obenour 05 Jun '25

05 Jun '25

Alyssa and I discussed several designs for camera support in Spectrum. These included: - USB passthrough - virtio-media - v4l2loopback and a guest agent A custom PipeWire plugin in the guest is also an option. USB passthrough has serious security problems: the guest can perform actions like updating device firmware, and the guest also fully trusts the device. This means that it is only suitable for trusted guests operating on trusted devices, which is less than great. v4l2loopback is an ugly hack and neither Alyssa or I like it, so I don’t think it is going to be the approach taken. It has the advantage of being fairly easy to implement and not requiring any device emulation, which is why Qubes OS went this route, but Spectrum has infrastructure for implementing complex virtio devices in a reasonably secure way, thanks to using Rust rather than C for this. This leaves virtio-media and a fully custom solution based on PipeWire. During the discussion, the possibility of hardening virtio-media against a malicious device was considered. After the call, however, I found out that while hardening the kernel side is definitely possible, it is also insufficient. The reason is that virtio-media, as currently implemented, appears to be effectively V4L2 API passthrough, which would mean that the device can respond to V4L2 IOCTLs however it wants. Guest userspace will almost certainly treat V4L2 IOCTL outputs as trusted, so hardening the guest kernel would be of only limited value. Adding validation in the guest kernel driver would be an option, but it would add substantial complexity. Instead, I think it is necessary to add a media server. This server would expose a virtio-media output device to the VM with the camera, and would expose a virtio-media capture device to all VMs on the system, not just those authorized to receive video. This is because camera hotplug is not properly handled by at least Google Chrome, so I expect other applications to also mishandle it. Instead, a VM that does not have camera permission can be given a camera that always records black, as if the user had covered the camera. PipeWire is the only existing implementation of a media server I know of that allows custom media devices to be implemented out-of-process, so it is the best choice I know of. PipeWire is also considered the future by the entire Linux desktop community. One other factor that I did not consider at all during the discussion is the need to implement the XDG microphone and camera portals. These portals are based on PipeWire, and PipeWire cannot be directly used across VMs. This is partially because of security, but it is also because PipeWire relies on SCM_RIGHTS file descriptor passing, eventfds, and other Linux kernel APIs that do not work across the VM boundary. Therefore, it is necessary to either run PipeWire in the guest, or run a daemon in the guest that exposes the same interface PipeWire does. All portals supported in Spectrum that prompt the user require a Spectrum-specific implementation so that the prompt happens on the host. Are there any better options? This is *an* approach, which does not mean it is *the best* approach. -- Sincerely, Demi Marie Obenour (she/her/hers)

2 2

This Week in Spectrum, 2025-W22
by Alyssa Ross 02 Jun '25

02 Jun '25

This week I was once again mostly preoccupied with arranging continuing funding. If approved, this will be the biggest funding allocation Spectrum has ever had, but with that comes a corresponding increase in the work required to make it happen. I also spent some time fixing Nixpkgs build issues that would affect Spectrum in future. That doesn't mean there hasn't been any exciting Spectrum work this week, though. Demi, whose name you may recognize from her extensive past work on Qubes OS, submitted a patch to the VIRTIO specification to add interrupt remapping capabilities to virtio-iommu on x86_64. This is an important requirement for being able to move implementations of virtual devices into their own VMs, which will in turn enable running hardware drivers in VMs rather than on the host, and it has been a longstanding wish list item in Spectrum's upcoming projects list[1]. It's very exciting to have somebody as knowledgeable as Demi working towards Spectrum's goals. [1]: https://pad.lassul.us/s/mDfFPvHjA#Cloud-Hypervisor-support-interrupt-remapp… BTW: From next week onwards, I think I'll start sending these updates only to discuss@, not devel@, because I don't think it makes sense to duplicate them.

1 0