August 2025 - Spectrum Development

Arranging groups of services
by Demi Marie Obenour 05 Sep '25

05 Sep '25

I'm working on Spectrum OS (https://spectrum-os.org/) and am currently porting it from s6 (https://skarnet.org/software/s6-linux-init/) to systemd. Spectrum OS's host (which is what is being ported) is rather different from a normal system: - The root filesystem is completely read-only. There's no writable /var. I decided to put a tmpfs there for now. - There is no network access, so /etc/resolv.conf isn't needed. - The real work happens in VMs, each of which depends on a few services: - Cloud Hypervisor (https://www.cloudhypervisor.org) which runs the VM. - crosvm (https://crosvm.dev/book/) used for graphics. - virtiofsd (https://virtio-fs.gitlab.io) to provide a filesystem - Spectrum OS's own proxy for the XDG desktop portals - In the future, an instance of vhost-device-sound (https://github.com/rust-vmm/vhost-device/blob/main/vhost-device-sound/READM…) used for sound - A per-VM D-Bus daemon - An instance of xdg-desktop-portal If the Cloud Hypervisor instance is stopped or exits, the others should be stopped automatically, as they have no other use. Having BindsTo=, After=, PropagatesStopTo=, and PropagatesReloadTo= should handle most cases, but I don't know if that is sufficient if Cloud Hypervisor exits spontaneously (because the guest shut down) or crashes. Additionally, these services have different sandboxing needs. Cloud Hypervisor should only be able to connect to its own instance of the daemons that serve it, rather than to any instance. crosvm needs GPU and Wayland access and vhost-device-sound needs to connect to PipeWire. virtiofsd needs an id-mapped mount. I would also like to block abstract AF_UNIX socket access. Are there existing systemd features that can easily meet these needs? For the sockets I am thinking of placing them in RuntimeDirectory= and only giving the correct units access to those directories. Also, I would like to use `DynamicUser=` for everything where that is possible. -- Sincerely, Demi Marie Obenour (she/her/hers)

3 2

Generating file lists from a script
by Demi Marie Obenour 01 Sep '25

01 Sep '25

Maintaining the lists of files and directories in the makefiles is error-prone. Would it make sense to use a script to generate them from the git repository? The script would get it right every time, whereas I tend to get it wrong more times than not. -- Sincerely, Demi Marie Obenour (she/her/hers)

2 2

[DO_NOT_APPLY v2 0/3] xdp-forwarder
by Yureka Lilian 01 Sep '25

01 Sep '25

Same as v1, it doesn't make sense to apply this without the router. Changes since v1: - rebased - apply new uncrustify config - moved xdp-forwarder to tools/ - split integration into separate commit - use linuxHeaders instead of vmlinux.h - use original xdp-tutorial {parsing,rewrite}_helpers.h - inlined the load scripts into /etc/iface/mdev, using /usr/lib/xdp as fixed prefix for finding the XDP progs - removed the README, added a paragraph to architecture doc instead Yureka Lilian (3): add xdp-forwarder integrate xdp-forwarder into net-vm docs/architecture: add paragraph about networking Documentation/about/architecture.adoc | 20 ++ pkgs/default.nix | 1 + tools/xdp-forwarder/Makefile | 28 ++ tools/xdp-forwarder/default.nix | 24 ++ tools/xdp-forwarder/include/parsing_helpers.h | 273 ++++++++++++++++++ tools/xdp-forwarder/include/rewrite_helpers.h | 145 ++++++++++ tools/xdp-forwarder/prog_physical.c | 37 +++ tools/xdp-forwarder/prog_router.c | 43 +++ tools/xdp-forwarder/set_router_iface.c | 32 ++ vm/sys/net/Makefile | 8 +- vm/sys/net/default.nix | 19 +- vm/sys/net/etc/fstab | 2 + vm/sys/net/etc/mdev/iface | 27 +- vm/sys/net/etc/nftables.conf | 8 - vm/sys/net/etc/s6-rc/connman/dependencies | 4 - vm/sys/net/etc/s6-rc/connman/type | 1 - vm/sys/net/etc/s6-rc/connman/type.license | 2 - vm/sys/net/etc/s6-rc/nftables/type | 1 - vm/sys/net/etc/s6-rc/nftables/type.license | 2 - vm/sys/net/etc/s6-rc/nftables/up | 6 - 20 files changed, 623 insertions(+), 60 deletions(-) create mode 100644 tools/xdp-forwarder/Makefile create mode 100644 tools/xdp-forwarder/default.nix create mode 100644 tools/xdp-forwarder/include/parsing_helpers.h create mode 100644 tools/xdp-forwarder/include/rewrite_helpers.h create mode 100644 tools/xdp-forwarder/prog_physical.c create mode 100644 tools/xdp-forwarder/prog_router.c create mode 100644 tools/xdp-forwarder/set_router_iface.c delete mode 100644 vm/sys/net/etc/nftables.conf delete mode 100644 vm/sys/net/etc/s6-rc/connman/dependencies delete mode 100644 vm/sys/net/etc/s6-rc/connman/type delete mode 100644 vm/sys/net/etc/s6-rc/connman/type.license delete mode 100644 vm/sys/net/etc/s6-rc/nftables/type delete mode 100644 vm/sys/net/etc/s6-rc/nftables/type.license delete mode 100644 vm/sys/net/etc/s6-rc/nftables/up -- 2.50.1

3 6

[DO_NOT_APPLY 0/2] xdp-forwarder
by Yureka Lilian 01 Sep '25

01 Sep '25

Hi all, Since this worsens the networking situation unless combined with the userspace Router (which is to-be-done), I am formally sending this out for comments without expecting it to be applied for now. The net-vm's purpose is running the Linux drivers for any physical interfaces on the spectrum system. A net-vm (there could be multiple, one per IOMMU-group) will load the included XDP programs on the passed-through physical interfaces as well as the downstream virtio interface going into the router (recognized by its special MAC address) using mdev events. The net-vm needs to multiplex between the physical interfaces, as there might be several interfaces in the same IOMMU-group. For this, the XDP program loaded on the physical interfaces applies a VLAN tag corresponding to the interface id, and redirects the packets to the router interface (identified by the router_iface bpf map). In the other direction the XDP program loaded on the router interface removes one layer of VLAN tagging, and redirects the packets to the interface read from the VLAN tag. I have verified that when running a wpa_supplicant / iwd in the interface, a WiFi interface can correctly function as a physical interface with the XDP forwarder, assuming that the router sends packets with the correct source MAC address. Yureka Lilian (2): integrate xdp-forwarder into net-vm temporary changes for testing lib/nixpkgs.default.nix | 4 +- vm/sys/net/Makefile | 8 +- vm/sys/net/default.nix | 38 +++++-- vm/sys/net/etc/fstab | 1 + vm/sys/net/etc/mdev/iface | 23 +--- vm/sys/net/etc/nftables.conf | 8 -- vm/sys/net/etc/s6-rc/connman/dependencies | 4 - vm/sys/net/etc/s6-rc/connman/type | 1 - vm/sys/net/etc/s6-rc/connman/type.license | 2 - vm/sys/net/etc/s6-rc/nftables/type | 1 - vm/sys/net/etc/s6-rc/nftables/type.license | 2 - vm/sys/net/etc/s6-rc/nftables/up | 6 - vm/sys/net/xdp-forwarder/README.md | 9 ++ vm/sys/net/xdp-forwarder/default.nix | 35 ++++++ .../xdp-forwarder/include/parsing_helpers.h | 38 +++++++ .../xdp-forwarder/include/rewrite_helpers.h | 103 ++++++++++++++++++ vm/sys/net/xdp-forwarder/load_physical | 4 + vm/sys/net/xdp-forwarder/load_router | 6 + vm/sys/net/xdp-forwarder/prog_physical.c | 28 +++++ vm/sys/net/xdp-forwarder/prog_router.c | 34 ++++++ vm/sys/net/xdp-forwarder/set_router_iface.c | 31 ++++++ 21 files changed, 325 insertions(+), 61 deletions(-) delete mode 100644 vm/sys/net/etc/nftables.conf delete mode 100644 vm/sys/net/etc/s6-rc/connman/dependencies delete mode 100644 vm/sys/net/etc/s6-rc/connman/type delete mode 100644 vm/sys/net/etc/s6-rc/connman/type.license delete mode 100644 vm/sys/net/etc/s6-rc/nftables/type delete mode 100644 vm/sys/net/etc/s6-rc/nftables/type.license delete mode 100644 vm/sys/net/etc/s6-rc/nftables/up create mode 100644 vm/sys/net/xdp-forwarder/README.md create mode 100644 vm/sys/net/xdp-forwarder/default.nix create mode 100644 vm/sys/net/xdp-forwarder/include/parsing_helpers.h create mode 100644 vm/sys/net/xdp-forwarder/include/rewrite_helpers.h create mode 100755 vm/sys/net/xdp-forwarder/load_physical create mode 100755 vm/sys/net/xdp-forwarder/load_router create mode 100644 vm/sys/net/xdp-forwarder/prog_physical.c create mode 100644 vm/sys/net/xdp-forwarder/prog_router.c create mode 100644 vm/sys/net/xdp-forwarder/set_router_iface.c -- 2.50.1

3 9

[PATCH] host/rootfs: add /dev/fd and /dev/std*
by Demi Marie Obenour 31 Aug '25

31 Aug '25

This is the same as 14483e1a690c (img/app: add /dev/fd and /dev/std*), but for the host. Signed-off-by: Demi Marie Obenour <demiobenour(a)gmail.com> --- host/rootfs/etc/s6-linux-init/scripts/rc.init | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/host/rootfs/etc/s6-linux-init/scripts/rc.init b/host/rootfs/etc/s6-linux-init/scripts/rc.init index 674fd38cc76837c7be25a5ef060f0f4d4b786394..ec654dd0ff329e01a25c03bdd80ee839e679e4d0 100755 --- a/host/rootfs/etc/s6-linux-init/scripts/rc.init +++ b/host/rootfs/etc/s6-linux-init/scripts/rc.init @@ -2,6 +2,11 @@ # SPDX-License-Identifier: EUPL-1.2+ # SPDX-FileCopyrightText: 2020-2022, 2024 Alyssa Ross <hi(a)alyssa.is> +if { ln -s /proc/self/fd /dev } +if { ln -s /proc/self/fd/0 /dev/stdin } +if { ln -s /proc/self/fd/1 /dev/stdout } +if { ln -s /proc/self/fd/2 /dev/stderr } + if { s6-rc-init -c /etc/s6-rc /run/service } if { mount --make-shared /run } --- base-commit: 39baa378367d95fac6ce4d0140b25203b2ee9b53 change-id: 20250805-host-dev-fd-51d57d8341bb -- Sincerely, Demi Marie Obenour (she/her/hers)

2 3

'make clean' followed by 'make' in the same Nix shell doesn't work
by Demi Marie Obenour 31 Aug '25

31 Aug '25

The following command fails to rebuild the image: $ nix-shell --run 'make clean && make run-qemu' The following command succeeds: $ nix-shell --pure --run 'make clean' && nix-shell --pure --run 'make run-qemu' Interactive Nix shells exhibit the same problems, and adding --pure to the first invocation does not help. -- Sincerely, Demi Marie Obenour (she/her/hers)

2 5

Speeding up image builds?
by Demi Marie Obenour 31 Aug '25

31 Aug '25

I often find myself waiting for image builds to run. What are some steps I can take to make them faster? Generally, I find that image builds are I/O bound (CPU usage remains quite low), so one option would be to use a machine with better I/O. What is the best option for this? Is this something where thin provisioning (instead of thick provisioning) causes a large performance penalty? Or is the only solution to throw hardware at the problem? If the latter, what would better hardware be? -- Sincerely, Demi Marie Obenour (she/her/hers)

2 1

Moderate Cybersecurity Concern
by ball drinker 29 Aug '25

29 Aug '25

Salutations -- I write not as another spammer talking of riches and fame but as a concerned cybersecurity student aspiring to be a graduate. YOUR OPENSSH PRIVATE KEY WAS COMPROMISED IN A GOOGLE DORK IN CASE YOU WERE NOT AWARE! I figure that now that the important part was stated, I should share some proof of it. Attached will be a series of two (2) images. These were stumbled across while I was completing an assignment for my college course. I only wanted to share the information with you considering I figured it'd be pertinent for something like this.

2 1

Spectrum OS management daemon?
by Demi Marie Obenour 22 Aug '25

22 Aug '25

I'm wondering if Spectrum OS should have a management daemon that is responsible for a few tasks: - Starting VMs when requested by the UI. - Notifying the user of various situations that require their attention, such if a VM crashes. - Suspending VMs before the host suspends, and resuming them after the host resumes. I believe Qubes OS didn't do this initially, but that led to problems so they do do this now. - Acting on devices being plugged and unplugged from system VM. -- Sincerely, Demi Marie Obenour (she/her/hers)

2 3

Switching to udev?
by Demi Marie Obenour 20 Aug '25

20 Aug '25

I'm seriously wondering if Spectrum should switch from mdevd to udev. It's where all of the upstream device detection work happens, and it would avoid needing to statically configure sound devices for PipeWire. The current PipeWire patch statically creates devices. This works for VMs because VMs are statically configured and only have virtio devices. This was also the most challenging part of the effort, and I still had to guess some of the parameters. Supporting hotplug would require either a custom PipeWire module, a custom daemon that talked to PipeWire, keeping pw-cli running in interactive mode, or patches to PipeWire to make device lingering work. In the audio case, ALSA expects device paths of the form hw:card:device:subdevice. However, I don't believe card numbers are stable: they reflect the order in which the kernel enumerates devices, and that is not at all guaranteed. A VM that only has a virtio sound device will only ever have card 0, device 0, and subdevice 0, so hard-coding hw:0,0,0 is fine. This is not true if physical hardware is present. That requires a lot of device identification logic, and all of the upstream work on that is in udev or the systemd hardware database (which udev also uses). In the future, there may be other projects Spectrum wants to use that rely on udev for enumerating devices. Some of them might *only* support PipeWire and have no other means of configuration. Even if other means of configuration *exist*, I expect them to be very rarely used by upstream and thus poorly tested. tl;dr: you can get away without udev in VMs, but I expect problems with physical hardware, and I want to switch to udev as a preemptive measure to avoid those problems and to ease development. -- Sincerely, Demi Marie Obenour (she/her/hers)

2 6