[PATCH v6 0/2] Switch from mdevd to systemd-udevd in root filesystem
systemd-udevd provides support for significantly more hardware than mdevd does. Therefore, switch to it. Signed-off-by: Demi Marie Obenour <demiobenour@gmail.com> --- Changes in v6: - Avoid using GOTO and LABEL in udev rules. Changes in v5: - Use -D_GNU_SOURCE in build system instead of #define _GNU_SOURCE 1 in C files. - Do not check for EAGAIN or EWOULDBLOCK after poll() has reported FD as ready. - Do not use MSG_NONBLOCK. - Use static variable for iovec. - Network hotplug script is safe to call more than once, so do not check for it having already been called. - Do not try to prevent /run/user from being created with wrong mode when it is known to already exist. - Fix copyright texts. - Link to v4: https://spectrum-os.org/lists/archives/spectrum-devel/20251003-udev-v4-0-7d7... Changes in v4: - Use 'udevadm trigger --action=add' at coldplug time. - Ensure that the wrapper does not exit after the first message. - Include the hardware database. - sd-notify-adapter should not exit if it receives a message without READY=1. - Link to v3: https://spectrum-os.org/lists/archives/spectrum-devel/20250928-udev-v3-0-bb0... Changes in v3: - Try to avoid net-add running twice. - Use full systemd instead of systemdMinimal, as something else already pulls it into the image. - Hopefully fix integration tests. - Link to v2: https://spectrum-os.org/lists/archives/spectrum-devel/20250924-udev-v2-2-608... Changes in v2: - Use more optional arguments instead of positional parameters in sd-notify-wrapper. - Explain why explicit modprobe is needed. - Link to v1: https://spectrum-os.org/lists/archives/spectrum-devel/20250913-udev-v1-0-ead... --- Demi Marie Obenour (2): tools: Add adapter tool for services using sd_notify host/rootfs: Switch to systemd-udevd host/rootfs/Makefile | 2 +- host/rootfs/default.nix | 118 ++++++++------------- host/rootfs/file-list.mk | 33 +++--- host/rootfs/image/etc/mdev.conf | 7 -- host/rootfs/image/etc/mdev/listen | 11 -- host/rootfs/image/etc/mdev/wait | 14 --- host/rootfs/image/etc/s6-rc/card0/type.license | 2 - host/rootfs/image/etc/s6-rc/card0/up | 4 - host/rootfs/image/etc/s6-rc/kvm/timeout-up | 1 - host/rootfs/image/etc/s6-rc/kvm/timeout-up.license | 2 - host/rootfs/image/etc/s6-rc/kvm/type | 1 - host/rootfs/image/etc/s6-rc/kvm/type.license | 2 - host/rootfs/image/etc/s6-rc/kvm/up | 4 - host/rootfs/image/etc/s6-rc/mdevd-coldplug/type | 1 - host/rootfs/image/etc/s6-rc/mdevd-coldplug/up | 4 - .../image/etc/s6-rc/mdevd/notification-fd.license | 2 - host/rootfs/image/etc/s6-rc/mdevd/run | 5 - .../contents.d/systemd-udevd-coldplug} | 0 .../dependencies.d/systemd-udevd} | 0 .../s6-rc/{card0 => systemd-udevd-coldplug}/type | 0 .../type.license | 0 .../image/etc/s6-rc/systemd-udevd-coldplug/up | 3 + .../s6-rc/{mdevd => systemd-udevd}/notification-fd | 0 .../s6-rc/systemd-udevd/notification-fd.license | 2 + host/rootfs/image/etc/s6-rc/systemd-udevd/run | 14 +++ .../image/etc/s6-rc/{mdevd => systemd-udevd}/type | 0 .../s6-rc/{mdevd => systemd-udevd}/type.license | 0 .../contents.d/systemd-udevd-coldplug} | 0 .../contents.d/systemd-udevd-coldplug} | 0 .../weston/dependencies.d/systemd-udevd-coldplug | 0 host/rootfs/image/etc/s6-rc/weston/run | 6 +- .../image/etc/udev/rules.d/99-spectrum.rules | 5 + host/rootfs/image/usr/bin/run-vmm | 1 + host/rootfs/image/usr/bin/systemd-udevd | 1 + .../{etc/mdev/net/add => usr/libexec/net-add} | 0 tools/default.nix | 1 + tools/meson.build | 1 + tools/sd-notify-adapter/meson.build | 4 + tools/sd-notify-adapter/sd-notify-adapter.c | 107 +++++++++++++++++++ 39 files changed, 199 insertions(+), 159 deletions(-) --- base-commit: 22e216712322cdfb85094bbd27ff34c4366fad41 change-id: 20250911-udev-24cf4bf68fbe -- Sincerely, Demi Marie Obenour (she/her/hers)
This adapts programs using sd_notify for use with s6 readiness notification. stdin and stdout are hard-coded for simplicity. Signed-off-by: Demi Marie Obenour <demiobenour@gmail.com> --- systemd readiness notification has two strict advantages over the s6 version: 1. It allows reliable reloading. 2. It allows providing a status message that the service manager can show in status output. s6 would actually benefit from both of these features. --- Changes since v4: - Use CC0 for the trivial Meson file. - Use -D_GNU_SOURCE instead of #define _GNU_SOURCE 1. - Use a global variable for the iovec. - Use ISC license for the C code in case s6 wants to use it. - Remove unneeded MSG_DONTWAIT. - Remove unneeded check for EAGAIN and EWOULDBLOCK when poll() has already indicated that the file descriptor is ready. Changes since v1: - Hard-code file descriptors. - Run wrapper as background process. - Massively reduce code size. - Use // instead of /* */ for comments. - Check that the notification FD is a pipe and that the listening socket is a socket. - Rely on s6-ipc-socketbinder to create the listening socket. - Do not unlink the listening socket. Signed-off-by: Demi Marie Obenour <demiobenour@gmail.com> --- tools/default.nix | 1 + tools/meson.build | 1 + tools/sd-notify-adapter/meson.build | 4 ++ tools/sd-notify-adapter/sd-notify-adapter.c | 107 ++++++++++++++++++++++++++++ 4 files changed, 113 insertions(+) diff --git a/tools/default.nix b/tools/default.nix index 398f57038d3f9fdec47b72850755e289f225f339..36a9a8c84de9f83e31981d3d47fab27e98f6ae5a 100644 --- a/tools/default.nix +++ b/tools/default.nix @@ -76,6 +76,7 @@ stdenv.mkDerivation (finalAttrs: { ./lsvm ./start-vmm ./subprojects + ./sd-notify-adapter ] ++ lib.optionals driverSupport [ ./xdp-forwarder ])); diff --git a/tools/meson.build b/tools/meson.build index 186008dbc9dd2b63adbce7475c375fb0de5c2c6a..5d0ae81042fd3d77646594500f32cb1d48a6af0c 100644 --- a/tools/meson.build +++ b/tools/meson.build @@ -27,6 +27,7 @@ if get_option('host') subdir('lsvm') subdir('start-vmm') + subdir('sd-notify-adapter') endif if get_option('app') diff --git a/tools/sd-notify-adapter/meson.build b/tools/sd-notify-adapter/meson.build new file mode 100644 index 0000000000000000000000000000000000000000..906fbde673105fd65436b34802f1bcc74e9f823e --- /dev/null +++ b/tools/sd-notify-adapter/meson.build @@ -0,0 +1,4 @@ +# SPDX-License-Identifier: CC0-1.0 +# SPDX-FileCopyrightText: 2025 Demi Marie Obenour <demiobenour@gmail.com> + +executable('sd-notify-adapter', 'sd-notify-adapter.c', c_args : ['-D_GNU_SOURCE'], install: true) diff --git a/tools/sd-notify-adapter/sd-notify-adapter.c b/tools/sd-notify-adapter/sd-notify-adapter.c new file mode 100644 index 0000000000000000000000000000000000000000..734f018da605e4359d3fa6422a01effbd9e878ea --- /dev/null +++ b/tools/sd-notify-adapter/sd-notify-adapter.c @@ -0,0 +1,107 @@ +// SPDX-License-Identifier: ISC +// SPDX-FileCopyrightText: 2025 Demi Marie Obenour <demiobenour@gmail.com> + +#include <assert.h> +#include <errno.h> +#include <limits.h> +#include <signal.h> +#include <stdarg.h> +#include <stddef.h> +#include <stdint.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> + +#include <err.h> +#include <fcntl.h> +#include <poll.h> +#include <sys/socket.h> +#include <sys/stat.h> +#include <sys/un.h> +#include <unistd.h> + +#define ARRAY_SIZE(s) (sizeof(s)/sizeof(s[0])) + +enum { + socket_fd, + notification_fd, +}; + +struct iovec msg; + +#define READY "READY=1" +#define READY_SIZE (sizeof(READY) - 1) + +static void process_notification(void) +{ + ssize_t first_recv_size = recv(socket_fd, msg.iov_base, msg.iov_len, + MSG_TRUNC | MSG_PEEK); + if (first_recv_size == -1) { + if (errno == EINTR) + return; // signal caught + err(EXIT_FAILURE, "recv from notification socket"); + } + size_t size = (size_t)first_recv_size; + if (size == 0) + return; // avoid arithmetic on NULL pointer + if (size > msg.iov_len) { + msg.iov_base = realloc(msg.iov_base, size); + if (msg.iov_base == NULL) + err(EXIT_FAILURE, "allocation failure"); + msg.iov_len = size; + } + ssize_t second_recv_size = recv(socket_fd, msg.iov_base, msg.iov_len, + MSG_CMSG_CLOEXEC | MSG_TRUNC); + if (second_recv_size == -1) { + if (errno == EINTR) + return; + err(EXIT_FAILURE, "recv from notification socket"); + } + assert(first_recv_size == second_recv_size); + for (char *next, *cursor = msg.iov_base, *end = cursor + size; + cursor != NULL; cursor = (next == NULL ? NULL : next + 1)) { + next = memchr(cursor, '\n', (size_t)(end - cursor)); + size_t message_size = (size_t)((next == NULL ? end : next) - cursor); + if (message_size == READY_SIZE && + memcmp(cursor, READY, READY_SIZE) == 0) { + ssize_t write_size = write(notification_fd, "\n", 1); + if (write_size != 1) + err(EXIT_FAILURE, "writing to notification descriptor"); + exit(0); + } + } +} + +int main(int argc, char **) +{ + if (argc != 1) + errx(EXIT_FAILURE, "stdin is listening socket, stdout is notification pipe"); + for (;;) { + struct pollfd p[] = { + { + .fd = socket_fd, + .events = POLLIN, + .revents = 0, + }, + { + .fd = notification_fd, + .events = 0, + .revents = 0, + }, + }; + int r = poll(p, ARRAY_SIZE(p), -1); + if (r < 0) { + if (errno == EINTR) + continue; + err(EXIT_FAILURE, "poll"); + } + if (p[0].revents) { + if (p[0].revents & POLLERR) + errx(EXIT_FAILURE, "unexpected POLLERR"); + if (p[0].revents & POLLIN) + process_notification(); + } + if (p[1].revents) + errx(EXIT_FAILURE, "s6 closed its pipe before the child was ready"); + } +} -- 2.51.2
This patch has been committed as 91bd1d06314e907f6a4c59872967f520dad031bf, which can be viewed online at https://spectrum-os.org/git/spectrum/commit/?id=91bd1d06314e907f6a4c59872967.... This is an automated message. Send comments/questions/requests to: Alyssa Ross <hi@alyssa.is>
This replaces the mdevd uevent daemon with systemd-udevd, which supports much more hardware and is therefore to be preferred for a full desktop environment. Various Wayland compositors integrate with systemd-udevd, allowing them to discover devices as they appear rather than having to only use devices plugged in when the compositor is started. Additionally, systemd-udevd has quirks for various devices that are needed to support the wide variety of hardware end-users have. The dependency on /dev/dri/card0 being present is eliminated, and whatever devices the user has are now picked up by the compositor. New dependencies are added to ensure that udev coldplug has finished before any non-trivial services are started. systemd-udev-trigger.service runs 'udevadm trigger' and has Before=sysinit.target, so anything that is not an early boot service can assume 'udevadm trigger' has run. systemd-udevd unsets PATH before it runs programs. Therefore, manually set it to /usr/bin before invoking any scripts. systemd-udevd provides the 'udevadm wait' command to wait for a device to appear and be processed. This replaces all uses of /etc/mdev/wait, so this script is removed. /etc/mdev/listen and /run/wait only existed to wake up /etc/mdev/wait, so they are also removed. The command-line options to 'udevadm trigger' are taken from systemd-udev-trigger.service on my Arch machine. Using the wrong arguments leads to problems that take a very long time to debug. Signed-off-by: Demi Marie Obenour <demiobenour@gmail.com> --- host/rootfs/Makefile | 2 +- host/rootfs/default.nix | 118 ++++++++------------- host/rootfs/file-list.mk | 33 +++--- host/rootfs/image/etc/mdev.conf | 7 -- host/rootfs/image/etc/mdev/listen | 11 -- host/rootfs/image/etc/mdev/wait | 14 --- host/rootfs/image/etc/s6-rc/card0/type.license | 2 - host/rootfs/image/etc/s6-rc/card0/up | 4 - host/rootfs/image/etc/s6-rc/kvm/timeout-up | 1 - host/rootfs/image/etc/s6-rc/kvm/timeout-up.license | 2 - host/rootfs/image/etc/s6-rc/kvm/type | 1 - host/rootfs/image/etc/s6-rc/kvm/type.license | 2 - host/rootfs/image/etc/s6-rc/kvm/up | 4 - host/rootfs/image/etc/s6-rc/mdevd-coldplug/type | 1 - host/rootfs/image/etc/s6-rc/mdevd-coldplug/up | 4 - .../image/etc/s6-rc/mdevd/notification-fd.license | 2 - host/rootfs/image/etc/s6-rc/mdevd/run | 5 - .../contents.d/systemd-udevd-coldplug} | 0 .../dependencies.d/systemd-udevd} | 0 .../s6-rc/{card0 => systemd-udevd-coldplug}/type | 0 .../type.license | 0 .../image/etc/s6-rc/systemd-udevd-coldplug/up | 3 + .../s6-rc/{mdevd => systemd-udevd}/notification-fd | 0 .../s6-rc/systemd-udevd/notification-fd.license | 2 + host/rootfs/image/etc/s6-rc/systemd-udevd/run | 14 +++ .../image/etc/s6-rc/{mdevd => systemd-udevd}/type | 0 .../s6-rc/{mdevd => systemd-udevd}/type.license | 0 .../contents.d/systemd-udevd-coldplug} | 0 .../contents.d/systemd-udevd-coldplug} | 0 .../weston/dependencies.d/systemd-udevd-coldplug | 0 host/rootfs/image/etc/s6-rc/weston/run | 6 +- .../image/etc/udev/rules.d/99-spectrum.rules | 5 + host/rootfs/image/usr/bin/run-vmm | 1 + host/rootfs/image/usr/bin/systemd-udevd | 1 + .../{etc/mdev/net/add => usr/libexec/net-add} | 0 35 files changed, 86 insertions(+), 159 deletions(-) diff --git a/host/rootfs/Makefile b/host/rootfs/Makefile index aa45ca1d5c18d0dfb78d19267f263cc4222e8e84..00d125774bb7b98736d0928c69cb307740cee034 100644 --- a/host/rootfs/Makefile +++ b/host/rootfs/Makefile @@ -12,6 +12,7 @@ DIRS = \ dev \ etc/s6-linux-init/env \ etc/s6-linux-init/run-image/configs \ + etc/s6-linux-init/run-image/sd-notify-wrapper \ etc/s6-linux-init/run-image/service/dbus/instance \ etc/s6-linux-init/run-image/service/dbus/instances \ etc/s6-linux-init/run-image/service/dbus/template/data \ @@ -36,7 +37,6 @@ DIRS = \ etc/s6-linux-init/run-image/user \ etc/s6-linux-init/run-image/vm/by-id \ etc/s6-linux-init/run-image/vm/by-name \ - etc/s6-linux-init/run-image/wait \ ext \ proc \ run \ diff --git a/host/rootfs/default.nix b/host/rootfs/default.nix index 9632f52674521bafb4ca122f15fc30a52abaad18..0d79f7ca54ccc86eb0fa6e743f2011237d365f24 100644 --- a/host/rootfs/default.nix +++ b/host/rootfs/default.nix @@ -8,88 +8,48 @@ import ../../lib/call-package.nix ( }: pkgsStatic.callPackage ( -{ spectrum-host-tools -, lib, stdenvNoCC, nixos, runCommand, writeClosure, erofs-utils, s6-rc -, busybox, cloud-hypervisor, cryptsetup, dbus, execline, inkscape -, iproute2, inotify-tools, jq, kmod, mdevd, s6, s6-linux-init, socat -, util-linuxMinimal, virtiofsd, xorg, xdg-desktop-portal-spectrum-host +{ busybox, cloud-hypervisor, cryptsetup, dbus, erofs-utils, execline +, inkscape, inotify-tools, iproute2, jq, lib, mdevd, nixos +, runCommand, s6, s6-linux-init, s6-rc, socat, spectrum-host-tools +, stdenvNoCC, util-linuxMinimal, virtiofsd, writeClosure +, xdg-desktop-portal-spectrum-host, xorg }: - let - inherit (nixosAllHardware.config.hardware) firmware; inherit (lib) - concatMapStringsSep concatStrings escapeShellArgs fileset optionalAttrs + concatMapStringsSep concatStrings escapeShellArgs fileset mapAttrsToList systems trivial; - pkgsGui = pkgsMusl.extend ( - final: super: - (optionalAttrs (systems.equals pkgsMusl.stdenv.hostPlatform super.stdenv.hostPlatform) { - flatpak = super.flatpak.override { - withMalcontent = false; - }; - - libgudev = super.libgudev.overrideAttrs ({ ... }: { - # Tests use umockdev, which is not compatible with libudev-zero. - doCheck = false; - }); - - qt6 = super.qt6.overrideScope (_: prev: { - qttranslations = prev.qttranslations.override { - qttools = prev.qttools.override { - qtbase = prev.qtbase.override { - qttranslations = null; - systemdSupport = false; - }; - qtdeclarative = null; - }; - }; - - qtbase = prev.qtbase.override { - systemdSupport = false; - }; - }); - - systemd = super.systemd.overrideAttrs ({ meta ? { }, ... }: { + _final: super: + (lib.optionalAttrs (systems.equals pkgsMusl.stdenv.hostPlatform super.stdenv.hostPlatform) { + malcontent = super.malcontent.overrideAttrs ({ meta ? { }, ... }: { meta = meta // { platforms = [ ]; }; }); + })); +in +# Something already pulls in the full +# systemd, so might as well use it. +pkgsGui.callPackage ( +{ cosmic-files, crosvm, dejavu_fonts, foot, kmod, mesa +, systemd, westonLite, xdg-desktop-portal, xdg-desktop-portal-gtk +}: - upower = super.upower.override { - # Not ideal, but it's the best way to get rid of an installed - # test that needs umockdev. - withIntrospection = false; - }; - - udev = final.libudev-zero; - - weston = super.weston.overrideAttrs ({ mesonFlags ? [], ... }: { - mesonFlags = mesonFlags ++ [ - "-Dsystemd=false" - ]; - }); - - xdg-desktop-portal = (super.xdg-desktop-portal.override { - enableSystemd = false; - }).overrideAttrs ({ ... }: { - # Tests use umockdev. - doCheck = false; - }); - }) - ); - - foot = pkgsGui.foot.override { allowPgo = false; }; +let + inherit (nixosAllHardware.config.hardware) firmware; + no_pgo_foot = foot.override { allowPgo = false; }; packages = [ - cloud-hypervisor cryptsetup dbus execline inotify-tools iproute2 - jq kmod mdevd s6 s6-linux-init s6-rc socat spectrum-host-tools - virtiofsd xdg-desktop-portal-spectrum-host + cloud-hypervisor crosvm cryptsetup dbus execline inotify-tools + iproute2 jq mdevd s6 s6-linux-init s6-rc socat + spectrum-host-tools virtiofsd xdg-desktop-portal-spectrum-host (busybox.override { extraConfig = '' CONFIG_CHATTR n CONFIG_DEPMOD n CONFIG_FINDFS n + CONFIG_HALT n CONFIG_INIT n CONFIG_INSMOD n CONFIG_IP n @@ -100,10 +60,13 @@ let CONFIG_MODINFO n CONFIG_MODPROBE n CONFIG_MOUNT n + CONFIG_POWEROFF n + CONFIG_REBOOT n CONFIG_RMMOD n + CONFIG_SHUTDOWN n ''; }) - ] ++ (with pkgsGui; [ cosmic-files crosvm foot ]); + ]; nixosAllHardware = nixos ({ modulesPath, ... }: { imports = [ (modulesPath + "/profiles/all-hardware.nix") ]; @@ -113,14 +76,15 @@ let kernel = linux_latest; - appvm = callSpectrumPackage ../../img/app { inherit (foot) terminfo; }; - netvm = callSpectrumPackage ../../vm/sys/net { inherit (foot) terminfo; }; + appvm = callSpectrumPackage ../../img/app { inherit (no_pgo_foot) terminfo; }; + netvm = callSpectrumPackage ../../vm/sys/net { inherit (no_pgo_foot) terminfo; }; # Packages that should be fully linked into /usr, # (not just their bin/* files). usrPackages = [ - appvm kernel.modules firmware netvm - ] ++ (with pkgsGui; [ mesa dejavu_fonts westonLite ]); + appvm kernel.modules firmware kmod kmod.lib + netvm mesa dejavu_fonts westonLite + ]; appvms = { appvm-firefox = callSpectrumPackage ../../vm/app/firefox.nix {}; @@ -138,16 +102,16 @@ let # Weston doesn't support SVG icons. inkscape -w 20 -h 20 \ -o $out/usr/share/icons/hicolor/20x20/apps/com.system76.CosmicFiles.png \ - ${pkgsGui.cosmic-files}/share/icons/hicolor/24x24/apps/com.system76.CosmicFiles.svg + ${cosmic-files}/share/icons/hicolor/24x24/apps/com.system76.CosmicFiles.svg ln -st $out/usr/bin \ ${concatMapStringsSep " " (p: "${p}/bin/*") packages} \ - ${pkgsGui.xdg-desktop-portal}/libexec/xdg-document-portal \ - ${pkgsGui.xdg-desktop-portal-gtk}/libexec/xdg-desktop-portal-gtk + ${xdg-desktop-portal}/libexec/xdg-document-portal \ + ${xdg-desktop-portal-gtk}/libexec/xdg-desktop-portal-gtk ln -st $out/usr/share/dbus-1 \ ${dbus}/share/dbus-1/session.conf ln -st $out/usr/share/dbus-1/services \ - ${pkgsGui.xdg-desktop-portal-gtk}/share/dbus-1/services/org.freedesktop.impl.portal.desktop.gtk.service + ${xdg-desktop-portal-gtk}/share/dbus-1/services/org.freedesktop.impl.portal.desktop.gtk.service for pkg in ${escapeShellArgs usrPackages}; do lndir -ignorelinks -silent "$pkg" "$out/usr" @@ -161,6 +125,10 @@ let # programs we want. # https://lore.kernel.org/util-linux/87zgrl6ufb.fsf@alyssa.is/ ln -s ${util-linuxMinimal}/bin/{findfs,uuidgen,lsblk,mount} $out/usr/bin + + # TODO: this is another hack and it should be possible + # to build systemd without this. + ln -s -- ${lib.escapeShellArg systemd}/bin/udevadm "$out/usr/bin" ''; in @@ -197,7 +165,7 @@ stdenvNoCC.mkDerivation { dontFixup = true; passthru = { - inherit appvm firmware kernel nixosAllHardware packagesSysroot pkgsGui; + inherit appvm firmware kernel nixosAllHardware packagesSysroot; }; meta = with lib; { @@ -205,4 +173,4 @@ stdenvNoCC.mkDerivation { platforms = platforms.linux; }; } -) {}) (_: {}) +) {}) {}) (_: {}) diff --git a/host/rootfs/file-list.mk b/host/rootfs/file-list.mk index 3694dc472a013fff931847171145465329fb800f..9acaa1d90bed674814775becf89c1c847d0ce3e3 100644 --- a/host/rootfs/file-list.mk +++ b/host/rootfs/file-list.mk @@ -7,10 +7,6 @@ FILES = \ image/etc/group \ image/etc/init \ image/etc/login \ - image/etc/mdev.conf \ - image/etc/mdev/listen \ - image/etc/mdev/net/add \ - image/etc/mdev/wait \ image/etc/parse-devname \ image/etc/passwd \ image/etc/s6-linux-init/env/WAYLAND_DISPLAY \ @@ -46,6 +42,7 @@ FILES = \ image/etc/s6-linux-init/run-image/service/xdg-desktop-portal-spectrum-host/template/notification-fd \ image/etc/s6-linux-init/run-image/service/xdg-desktop-portal-spectrum-host/template/run \ image/etc/s6-linux-init/scripts/rc.init \ + image/etc/udev/rules.d/99-spectrum.rules \ image/etc/xdg/weston/autolaunch \ image/etc/xdg/weston/weston.ini \ image/usr/bin/assign-devices \ @@ -57,6 +54,7 @@ FILES = \ image/usr/bin/vm-start \ image/usr/bin/vm-stop \ image/usr/bin/xdg-open \ + image/usr/libexec/net-add \ image/usr/share/dbus-1/services/org.freedesktop.portal.Documents.service LINKS = \ @@ -64,24 +62,14 @@ LINKS = \ image/etc/s6-linux-init/run-image/opengl-driver \ image/etc/s6-linux-init/run-image/service/vmm/template/run \ image/lib \ - image/sbin + image/sbin \ + image/usr/bin/systemd-udevd S6_RC_FILES = \ - image/etc/s6-rc/card0/type \ - image/etc/s6-rc/card0/up \ image/etc/s6-rc/core/type \ image/etc/s6-rc/core/up \ - image/etc/s6-rc/kvm/timeout-up \ - image/etc/s6-rc/kvm/type \ - image/etc/s6-rc/kvm/up \ - image/etc/s6-rc/mdevd-coldplug/dependencies.d/mdevd \ - image/etc/s6-rc/mdevd-coldplug/type \ - image/etc/s6-rc/mdevd-coldplug/up \ - image/etc/s6-rc/mdevd/notification-fd \ - image/etc/s6-rc/mdevd/run \ - image/etc/s6-rc/mdevd/type \ - image/etc/s6-rc/ok-all/contents.d/mdevd-coldplug \ image/etc/s6-rc/ok-all/contents.d/sys-vmms \ + image/etc/s6-rc/ok-all/contents.d/systemd-udevd-coldplug \ image/etc/s6-rc/ok-all/contents.d/vm-env \ image/etc/s6-rc/ok-all/type \ image/etc/s6-rc/static-nodes/type \ @@ -89,14 +77,21 @@ S6_RC_FILES = \ image/etc/s6-rc/sys-vmms/dependencies.d/vmm-env \ image/etc/s6-rc/sys-vmms/type \ image/etc/s6-rc/sys-vmms/up \ + image/etc/s6-rc/systemd-udevd-coldplug/dependencies.d/systemd-udevd \ + image/etc/s6-rc/systemd-udevd-coldplug/type \ + image/etc/s6-rc/systemd-udevd-coldplug/up \ + image/etc/s6-rc/systemd-udevd/notification-fd \ + image/etc/s6-rc/systemd-udevd/run \ + image/etc/s6-rc/systemd-udevd/type \ image/etc/s6-rc/vm-env/contents.d/static-nodes \ + image/etc/s6-rc/vm-env/contents.d/systemd-udevd-coldplug \ image/etc/s6-rc/vm-env/contents.d/weston \ image/etc/s6-rc/vm-env/type \ image/etc/s6-rc/vmm-env/contents.d/core \ - image/etc/s6-rc/vmm-env/contents.d/kvm \ image/etc/s6-rc/vmm-env/contents.d/static-nodes \ + image/etc/s6-rc/vmm-env/contents.d/systemd-udevd-coldplug \ image/etc/s6-rc/vmm-env/type \ - image/etc/s6-rc/weston/dependencies.d/card0 \ + image/etc/s6-rc/weston/dependencies.d/systemd-udevd-coldplug \ image/etc/s6-rc/weston/notification-fd \ image/etc/s6-rc/weston/run \ image/etc/s6-rc/weston/type diff --git a/host/rootfs/image/etc/mdev.conf b/host/rootfs/image/etc/mdev.conf deleted file mode 100644 index bddcfdc44ec2a8b1aa95e84cb88fdde625c766d8..0000000000000000000000000000000000000000 --- a/host/rootfs/image/etc/mdev.conf +++ /dev/null @@ -1,7 +0,0 @@ -# SPDX-License-Identifier: EUPL-1.2+ -# SPDX-FileCopyrightText: 2021-2022, 2024 Alyssa Ross <hi@alyssa.is> - -$PCI_CLASS=^2....$ 0:0 660 +/etc/mdev/net/add --$MODALIAS=.* 0:0 660 +importas -Siu MODALIAS modprobe -q $MODALIAS -kvm 0:0 660 +background { /etc/mdev/listen kvm } -dri/card0 0:0 660 +background { /etc/mdev/listen card0 } diff --git a/host/rootfs/image/etc/mdev/listen b/host/rootfs/image/etc/mdev/listen deleted file mode 100755 index ab50ee8c5ed1139d1129bac56afa7263af150745..0000000000000000000000000000000000000000 --- a/host/rootfs/image/etc/mdev/listen +++ /dev/null @@ -1,11 +0,0 @@ -#!/bin/execlineb -S1 -# SPDX-License-Identifier: EUPL-1.2+ -# SPDX-FileCopyrightText: 2021 Alyssa Ross <hi@alyssa.is> - -foreground { - redirfd -w 2 /dev/null - mkfifo /run/wait/${1} -} - -redirfd -w 1 /run/wait/${1} -echo diff --git a/host/rootfs/image/etc/mdev/wait b/host/rootfs/image/etc/mdev/wait deleted file mode 100755 index 6bddb303d2671ce4e5b8581cd81235d7404916e7..0000000000000000000000000000000000000000 --- a/host/rootfs/image/etc/mdev/wait +++ /dev/null @@ -1,14 +0,0 @@ -#!/bin/execlineb -S1 -# SPDX-License-Identifier: EUPL-1.2+ -# SPDX-FileCopyrightText: 2021 Alyssa Ross <hi@alyssa.is> - -foreground { - redirfd -w 2 /dev/null - mkfifo /run/wait/${1} -} - -foreground { - redirfd -w 1 /dev/null - head -1 /run/wait/${1} -} -rm /run/wait/${1} diff --git a/host/rootfs/image/etc/s6-rc/card0/type.license b/host/rootfs/image/etc/s6-rc/card0/type.license deleted file mode 100644 index c49c11b66262c7edc57ac06a486c1166d867c31d..0000000000000000000000000000000000000000 --- a/host/rootfs/image/etc/s6-rc/card0/type.license +++ /dev/null @@ -1,2 +0,0 @@ -SPDX-License-Identifier: CC0-1.0 -SPDX-FileCopyrightText: 2021 Alyssa Ross <hi@alyssa.is> diff --git a/host/rootfs/image/etc/s6-rc/card0/up b/host/rootfs/image/etc/s6-rc/card0/up deleted file mode 100644 index 703562e5442aea45198350afe86a8f38c11ed072..0000000000000000000000000000000000000000 --- a/host/rootfs/image/etc/s6-rc/card0/up +++ /dev/null @@ -1,4 +0,0 @@ -# SPDX-License-Identifier: EUPL-1.2+ -# SPDX-FileCopyrightText: 2021 Alyssa Ross <hi@alyssa.is> - -/etc/mdev/wait card0 diff --git a/host/rootfs/image/etc/s6-rc/kvm/timeout-up b/host/rootfs/image/etc/s6-rc/kvm/timeout-up deleted file mode 100644 index c5da56ae490a8ab35074fdcb6644a0dbbd280e3b..0000000000000000000000000000000000000000 --- a/host/rootfs/image/etc/s6-rc/kvm/timeout-up +++ /dev/null @@ -1 +0,0 @@ -40000 diff --git a/host/rootfs/image/etc/s6-rc/kvm/timeout-up.license b/host/rootfs/image/etc/s6-rc/kvm/timeout-up.license deleted file mode 100644 index d705e974a864074490588104a24a9ea789141572..0000000000000000000000000000000000000000 --- a/host/rootfs/image/etc/s6-rc/kvm/timeout-up.license +++ /dev/null @@ -1,2 +0,0 @@ -SPDX-License-Identifier: CC0-1.0 -SPDX-FileCopyrightText: 2024 Alyssa Ross <hi@alyssa.is> diff --git a/host/rootfs/image/etc/s6-rc/kvm/type b/host/rootfs/image/etc/s6-rc/kvm/type deleted file mode 100644 index bdd22a1850ae6c03a414eeb8084998679a2cdf92..0000000000000000000000000000000000000000 --- a/host/rootfs/image/etc/s6-rc/kvm/type +++ /dev/null @@ -1 +0,0 @@ -oneshot diff --git a/host/rootfs/image/etc/s6-rc/kvm/type.license b/host/rootfs/image/etc/s6-rc/kvm/type.license deleted file mode 100644 index a941ca495a4211cf6659eda03b30f83c02985fe6..0000000000000000000000000000000000000000 --- a/host/rootfs/image/etc/s6-rc/kvm/type.license +++ /dev/null @@ -1,2 +0,0 @@ -SPDX-License-Identifier: CC0-1.0 -SPDX-FileCopyrightText: 2023 Alyssa Ross <hi@alyssa.is> diff --git a/host/rootfs/image/etc/s6-rc/kvm/up b/host/rootfs/image/etc/s6-rc/kvm/up deleted file mode 100644 index c02e3f90245e005b98b4de8245a1863fb49c1158..0000000000000000000000000000000000000000 --- a/host/rootfs/image/etc/s6-rc/kvm/up +++ /dev/null @@ -1,4 +0,0 @@ -# SPDX-License-Identifier: EUPL-1.2+ -# SPDX-FileCopyrightText: 2023 Alyssa Ross <hi@alyssa.is> - -/etc/mdev/wait kvm diff --git a/host/rootfs/image/etc/s6-rc/mdevd-coldplug/type b/host/rootfs/image/etc/s6-rc/mdevd-coldplug/type deleted file mode 100644 index bdd22a1850ae6c03a414eeb8084998679a2cdf92..0000000000000000000000000000000000000000 --- a/host/rootfs/image/etc/s6-rc/mdevd-coldplug/type +++ /dev/null @@ -1 +0,0 @@ -oneshot diff --git a/host/rootfs/image/etc/s6-rc/mdevd-coldplug/up b/host/rootfs/image/etc/s6-rc/mdevd-coldplug/up deleted file mode 100644 index 8698f7d7988a017786fb91a584eafbfb23b3165d..0000000000000000000000000000000000000000 --- a/host/rootfs/image/etc/s6-rc/mdevd-coldplug/up +++ /dev/null @@ -1,4 +0,0 @@ -# SPDX-License-Identifier: EUPL-1.2+ -# SPDX-FileCopyrightText: 2020-2021 Alyssa Ross <hi@alyssa.is> - -mdevd-coldplug diff --git a/host/rootfs/image/etc/s6-rc/mdevd/notification-fd.license b/host/rootfs/image/etc/s6-rc/mdevd/notification-fd.license deleted file mode 100644 index 2b3b032142b7286bd317cf0abaa44fba3a9b8941..0000000000000000000000000000000000000000 --- a/host/rootfs/image/etc/s6-rc/mdevd/notification-fd.license +++ /dev/null @@ -1,2 +0,0 @@ -SPDX-License-Identifier: CC0-1.0 -SPDX-FileCopyrightText: 2020 Alyssa Ross <hi@alyssa.is> diff --git a/host/rootfs/image/etc/s6-rc/mdevd/run b/host/rootfs/image/etc/s6-rc/mdevd/run deleted file mode 100644 index 55899bbe674426e4591e866a4d0617361ba34305..0000000000000000000000000000000000000000 --- a/host/rootfs/image/etc/s6-rc/mdevd/run +++ /dev/null @@ -1,5 +0,0 @@ -#!/bin/execlineb -P -# SPDX-License-Identifier: EUPL-1.2+ -# SPDX-FileCopyrightText: 2020-2022 Alyssa Ross <hi@alyssa.is> - -mdevd -D3 -O4 -b134217728 diff --git a/host/rootfs/image/etc/s6-rc/mdevd-coldplug/dependencies.d/mdevd b/host/rootfs/image/etc/s6-rc/ok-all/contents.d/systemd-udevd-coldplug similarity index 100% rename from host/rootfs/image/etc/s6-rc/mdevd-coldplug/dependencies.d/mdevd rename to host/rootfs/image/etc/s6-rc/ok-all/contents.d/systemd-udevd-coldplug diff --git a/host/rootfs/image/etc/s6-rc/ok-all/contents.d/mdevd-coldplug b/host/rootfs/image/etc/s6-rc/systemd-udevd-coldplug/dependencies.d/systemd-udevd similarity index 100% rename from host/rootfs/image/etc/s6-rc/ok-all/contents.d/mdevd-coldplug rename to host/rootfs/image/etc/s6-rc/systemd-udevd-coldplug/dependencies.d/systemd-udevd diff --git a/host/rootfs/image/etc/s6-rc/card0/type b/host/rootfs/image/etc/s6-rc/systemd-udevd-coldplug/type similarity index 100% rename from host/rootfs/image/etc/s6-rc/card0/type rename to host/rootfs/image/etc/s6-rc/systemd-udevd-coldplug/type diff --git a/host/rootfs/image/etc/s6-rc/mdevd-coldplug/type.license b/host/rootfs/image/etc/s6-rc/systemd-udevd-coldplug/type.license similarity index 100% rename from host/rootfs/image/etc/s6-rc/mdevd-coldplug/type.license rename to host/rootfs/image/etc/s6-rc/systemd-udevd-coldplug/type.license diff --git a/host/rootfs/image/etc/s6-rc/systemd-udevd-coldplug/up b/host/rootfs/image/etc/s6-rc/systemd-udevd-coldplug/up new file mode 100644 index 0000000000000000000000000000000000000000..655b11a267f586aca471d47e45483bbe65ad608a --- /dev/null +++ b/host/rootfs/image/etc/s6-rc/systemd-udevd-coldplug/up @@ -0,0 +1,3 @@ +# SPDX-License-Identifier: CC0-1.0 +# SPDX-FileCopyrightText: 2025 Demi Marie Obenour <demiobenour@gmail.com> +udevadm trigger --type=all --action=add --prioritized-subsystem=module,block,tpmrm,net,tty,input diff --git a/host/rootfs/image/etc/s6-rc/mdevd/notification-fd b/host/rootfs/image/etc/s6-rc/systemd-udevd/notification-fd similarity index 100% rename from host/rootfs/image/etc/s6-rc/mdevd/notification-fd rename to host/rootfs/image/etc/s6-rc/systemd-udevd/notification-fd diff --git a/host/rootfs/image/etc/s6-rc/systemd-udevd/notification-fd.license b/host/rootfs/image/etc/s6-rc/systemd-udevd/notification-fd.license new file mode 100644 index 0000000000000000000000000000000000000000..c4a0586a407fe14c3e0855749a7524ac3871dda4 --- /dev/null +++ b/host/rootfs/image/etc/s6-rc/systemd-udevd/notification-fd.license @@ -0,0 +1,2 @@ +SPDX-License-Identifier: CC0-1.0 +SPDX-FileCopyrightText: 2025 Demi Marie Obenour <demiobenour@gmail.com> diff --git a/host/rootfs/image/etc/s6-rc/systemd-udevd/run b/host/rootfs/image/etc/s6-rc/systemd-udevd/run new file mode 100644 index 0000000000000000000000000000000000000000..a3b74cffe5af191d099ce9314b29669652762092 --- /dev/null +++ b/host/rootfs/image/etc/s6-rc/systemd-udevd/run @@ -0,0 +1,14 @@ +#!/bin/execlineb -P +# SPDX-License-Identifier: EUPL-1.2+ +# SPDX-FileCopyrightText: 2025 Demi Marie Obenour <demiobenour@gmail.com> +s6-setlock /run/sd-notify-wrapper/systemd-udevd.lock +# Save stdin to restore it later. +s6-ipcserver-socketbinder -b0 -m -a 0600 /run/sd-notify-wrapper/systemd-udevd.sock +background { + fdmove 1 3 + sd-notify-adapter +} +fdclose 3 +redirfd -r 0 /dev/null +export NOTIFY_SOCKET /run/sd-notify-wrapper/systemd-udevd.sock +systemd-udevd diff --git a/host/rootfs/image/etc/s6-rc/mdevd/type b/host/rootfs/image/etc/s6-rc/systemd-udevd/type similarity index 100% rename from host/rootfs/image/etc/s6-rc/mdevd/type rename to host/rootfs/image/etc/s6-rc/systemd-udevd/type diff --git a/host/rootfs/image/etc/s6-rc/mdevd/type.license b/host/rootfs/image/etc/s6-rc/systemd-udevd/type.license similarity index 100% rename from host/rootfs/image/etc/s6-rc/mdevd/type.license rename to host/rootfs/image/etc/s6-rc/systemd-udevd/type.license diff --git a/host/rootfs/image/etc/s6-rc/vmm-env/contents.d/kvm b/host/rootfs/image/etc/s6-rc/vm-env/contents.d/systemd-udevd-coldplug similarity index 100% rename from host/rootfs/image/etc/s6-rc/vmm-env/contents.d/kvm rename to host/rootfs/image/etc/s6-rc/vm-env/contents.d/systemd-udevd-coldplug diff --git a/host/rootfs/image/etc/s6-rc/weston/dependencies.d/card0 b/host/rootfs/image/etc/s6-rc/vmm-env/contents.d/systemd-udevd-coldplug similarity index 100% rename from host/rootfs/image/etc/s6-rc/weston/dependencies.d/card0 rename to host/rootfs/image/etc/s6-rc/vmm-env/contents.d/systemd-udevd-coldplug diff --git a/host/rootfs/image/etc/s6-rc/weston/dependencies.d/systemd-udevd-coldplug b/host/rootfs/image/etc/s6-rc/weston/dependencies.d/systemd-udevd-coldplug new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/host/rootfs/image/etc/s6-rc/weston/run b/host/rootfs/image/etc/s6-rc/weston/run index 7dca0dab095569c9e7d49df9d245533a7265283e..c1bce8505944b68c75c1b87d1ae736ff655e0f07 100644 --- a/host/rootfs/image/etc/s6-rc/weston/run +++ b/host/rootfs/image/etc/s6-rc/weston/run @@ -4,10 +4,7 @@ unexport WAYLAND_DISPLAY -foreground { - umask 077 - mkdir /run/user/0 -} +if { mkdir -p -m 0700 /run/user/0 } unexport ? backtick USER { id -un } @@ -20,4 +17,5 @@ redirfd -r 0 /dev/tty1 importas -i home HOME cd $home +if { udevadm wait /dev/dri/card0 } weston diff --git a/host/rootfs/image/etc/udev/rules.d/99-spectrum.rules b/host/rootfs/image/etc/udev/rules.d/99-spectrum.rules new file mode 100644 index 0000000000000000000000000000000000000000..337bbe47dbbc6f3828722d8244f2689a39f3090f --- /dev/null +++ b/host/rootfs/image/etc/udev/rules.d/99-spectrum.rules @@ -0,0 +1,5 @@ +# SPDX-License-Identifier: EUPL-1.2+ +# SPDX-FileCopyrightText: 2025 Demi Marie Obenour <demiobenour@gmail.com> + +# systemd-udevd unsets PATH, so fix that. +ACTION!="remove", ENV{PCI_CLASS}=="2????", RUN+="/usr/bin/env PATH=/usr/bin /usr/libexec/net-add" diff --git a/host/rootfs/image/usr/bin/run-vmm b/host/rootfs/image/usr/bin/run-vmm index bcb6cdaf6646da6bb4970fe97f5ef03badbd66a6..5649674c51b29b08532fc2cd985a5b20a78c3764 100755 --- a/host/rootfs/image/usr/bin/run-vmm +++ b/host/rootfs/image/usr/bin/run-vmm @@ -53,4 +53,5 @@ unexport ! fdmove -c 3 0 redirfd -r 0 /dev/null +if { udevadm wait /dev/kvm } cloud-hypervisor --api-socket fd=3 diff --git a/host/rootfs/image/usr/bin/systemd-udevd b/host/rootfs/image/usr/bin/systemd-udevd new file mode 120000 index 0000000000000000000000000000000000000000..b7887eaf6dd3279116ba61d613bd467598089597 --- /dev/null +++ b/host/rootfs/image/usr/bin/systemd-udevd @@ -0,0 +1 @@ +udevadm \ No newline at end of file diff --git a/host/rootfs/image/etc/mdev/net/add b/host/rootfs/image/usr/libexec/net-add similarity index 100% rename from host/rootfs/image/etc/mdev/net/add rename to host/rootfs/image/usr/libexec/net-add -- 2.51.2
Demi Marie Obenour <demiobenour@gmail.com> writes:
@@ -113,14 +76,15 @@ let
kernel = linux_latest;
- appvm = callSpectrumPackage ../../img/app { inherit (foot) terminfo; }; - netvm = callSpectrumPackage ../../vm/sys/net { inherit (foot) terminfo; }; + appvm = callSpectrumPackage ../../img/app { inherit (no_pgo_foot) terminfo; }; + netvm = callSpectrumPackage ../../vm/sys/net { inherit (no_pgo_foot) terminfo; };
# Packages that should be fully linked into /usr, # (not just their bin/* files). usrPackages = [ - appvm kernel.modules firmware netvm - ] ++ (with pkgsGui; [ mesa dejavu_fonts westonLite ]); + appvm kernel.modules firmware kmod kmod.lib + netvm mesa dejavu_fonts westonLite + ];
appvms = { appvm-firefox = callSpectrumPackage ../../vm/app/firefox.nix {};
So based on what we'd discussed previously, I was ready to just apply this and pare down the Nix changes myself since I don't have to worry about rebuilds, etc., but in doing so I came across this. Do you know why kmod.lib is required? Is something impurely dlopening it? Seems very strange, but I did verify that it's required for Weston to start.
On 11/8/25 14:49, Alyssa Ross wrote:
Demi Marie Obenour <demiobenour@gmail.com> writes:
@@ -113,14 +76,15 @@ let
kernel = linux_latest;
- appvm = callSpectrumPackage ../../img/app { inherit (foot) terminfo; }; - netvm = callSpectrumPackage ../../vm/sys/net { inherit (foot) terminfo; }; + appvm = callSpectrumPackage ../../img/app { inherit (no_pgo_foot) terminfo; }; + netvm = callSpectrumPackage ../../vm/sys/net { inherit (no_pgo_foot) terminfo; };
# Packages that should be fully linked into /usr, # (not just their bin/* files). usrPackages = [ - appvm kernel.modules firmware netvm - ] ++ (with pkgsGui; [ mesa dejavu_fonts westonLite ]); + appvm kernel.modules firmware kmod kmod.lib + netvm mesa dejavu_fonts westonLite + ];
appvms = { appvm-firefox = callSpectrumPackage ../../vm/app/firefox.nix {};
So based on what we'd discussed previously, I was ready to just apply this and pare down the Nix changes myself since I don't have to worry about rebuilds, etc., but in doing so I came across this. Do you know why kmod.lib is required? Is something impurely dlopening it? Seems very strange, but I did verify that it's required for Weston to start.
systemd uses dlopen heavily. This allows dependencies to be optional at runtime. I remember strace showing that systemd-udevd dlopen's kmod.lib. In short, your diagnosis is correct. -- Sincerely, Demi Marie Obenour (she/her/hers)
Demi Marie Obenour <demiobenour@gmail.com> writes:
On 11/8/25 14:49, Alyssa Ross wrote:
Demi Marie Obenour <demiobenour@gmail.com> writes:
@@ -113,14 +76,15 @@ let
kernel = linux_latest;
- appvm = callSpectrumPackage ../../img/app { inherit (foot) terminfo; }; - netvm = callSpectrumPackage ../../vm/sys/net { inherit (foot) terminfo; }; + appvm = callSpectrumPackage ../../img/app { inherit (no_pgo_foot) terminfo; }; + netvm = callSpectrumPackage ../../vm/sys/net { inherit (no_pgo_foot) terminfo; };
# Packages that should be fully linked into /usr, # (not just their bin/* files). usrPackages = [ - appvm kernel.modules firmware netvm - ] ++ (with pkgsGui; [ mesa dejavu_fonts westonLite ]); + appvm kernel.modules firmware kmod kmod.lib + netvm mesa dejavu_fonts westonLite + ];
appvms = { appvm-firefox = callSpectrumPackage ../../vm/app/firefox.nix {};
So based on what we'd discussed previously, I was ready to just apply this and pare down the Nix changes myself since I don't have to worry about rebuilds, etc., but in doing so I came across this. Do you know why kmod.lib is required? Is something impurely dlopening it? Seems very strange, but I did verify that it's required for Weston to start.
systemd uses dlopen heavily. This allows dependencies to be optional at runtime. I remember strace showing that systemd-udevd dlopen's kmod.lib. In short, your diagnosis is correct.
Okay, that's a packaging bug in Nixpkgs then. (See e.g. [1].) I'll see what can be done. [1]: https://github.com/NixOS/nixpkgs/commit/2328731ad041735a2260698574ce6599591f...
Alyssa Ross <hi@alyssa.is> writes:
Demi Marie Obenour <demiobenour@gmail.com> writes:
On 11/8/25 14:49, Alyssa Ross wrote:
Demi Marie Obenour <demiobenour@gmail.com> writes:
@@ -113,14 +76,15 @@ let
kernel = linux_latest;
- appvm = callSpectrumPackage ../../img/app { inherit (foot) terminfo; }; - netvm = callSpectrumPackage ../../vm/sys/net { inherit (foot) terminfo; }; + appvm = callSpectrumPackage ../../img/app { inherit (no_pgo_foot) terminfo; }; + netvm = callSpectrumPackage ../../vm/sys/net { inherit (no_pgo_foot) terminfo; };
# Packages that should be fully linked into /usr, # (not just their bin/* files). usrPackages = [ - appvm kernel.modules firmware netvm - ] ++ (with pkgsGui; [ mesa dejavu_fonts westonLite ]); + appvm kernel.modules firmware kmod kmod.lib + netvm mesa dejavu_fonts westonLite + ];
appvms = { appvm-firefox = callSpectrumPackage ../../vm/app/firefox.nix {};
So based on what we'd discussed previously, I was ready to just apply this and pare down the Nix changes myself since I don't have to worry about rebuilds, etc., but in doing so I came across this. Do you know why kmod.lib is required? Is something impurely dlopening it? Seems very strange, but I did verify that it's required for Weston to start.
systemd uses dlopen heavily. This allows dependencies to be optional at runtime. I remember strace showing that systemd-udevd dlopen's kmod.lib. In short, your diagnosis is correct.
Okay, that's a packaging bug in Nixpkgs then. (See e.g. [1].) I'll see what can be done.
[1]: https://github.com/NixOS/nixpkgs/commit/2328731ad041735a2260698574ce6599591f...
I have looked into it a bit and am now even more confused. From what I can see, loading libkmod should be done by libsystemd-shared.so, which has kmod.lib in its RUNPATH.
On 11/8/25 17:14, Alyssa Ross wrote:
Alyssa Ross <hi@alyssa.is> writes:
Demi Marie Obenour <demiobenour@gmail.com> writes:
On 11/8/25 14:49, Alyssa Ross wrote:
Demi Marie Obenour <demiobenour@gmail.com> writes:
@@ -113,14 +76,15 @@ let
kernel = linux_latest;
- appvm = callSpectrumPackage ../../img/app { inherit (foot) terminfo; }; - netvm = callSpectrumPackage ../../vm/sys/net { inherit (foot) terminfo; }; + appvm = callSpectrumPackage ../../img/app { inherit (no_pgo_foot) terminfo; }; + netvm = callSpectrumPackage ../../vm/sys/net { inherit (no_pgo_foot) terminfo; };
# Packages that should be fully linked into /usr, # (not just their bin/* files). usrPackages = [ - appvm kernel.modules firmware netvm - ] ++ (with pkgsGui; [ mesa dejavu_fonts westonLite ]); + appvm kernel.modules firmware kmod kmod.lib + netvm mesa dejavu_fonts westonLite + ];
appvms = { appvm-firefox = callSpectrumPackage ../../vm/app/firefox.nix {};
So based on what we'd discussed previously, I was ready to just apply this and pare down the Nix changes myself since I don't have to worry about rebuilds, etc., but in doing so I came across this. Do you know why kmod.lib is required? Is something impurely dlopening it? Seems very strange, but I did verify that it's required for Weston to start.
systemd uses dlopen heavily. This allows dependencies to be optional at runtime. I remember strace showing that systemd-udevd dlopen's kmod.lib. In short, your diagnosis is correct.
Okay, that's a packaging bug in Nixpkgs then. (See e.g. [1].) I'll see what can be done.
[1]: https://github.com/NixOS/nixpkgs/commit/2328731ad041735a2260698574ce6599591f...
I have looked into it a bit and am now even more confused. From what I can see, loading libkmod should be done by libsystemd-shared.so, which has kmod.lib in its RUNPATH.
According to https://wiki.debian.org/RpathIssue, only the RUNPATH of the executable is honored. The RUNPATH of libraries is not honored. For this to work, libsystemd-shared.so needs to have kmod.lib in its RPATH, and udevadm *may* need to not have a RUNPATH. -- Sincerely, Demi Marie Obenour (she/her/hers)
Demi Marie Obenour <demiobenour@gmail.com> writes:
On 11/8/25 17:14, Alyssa Ross wrote:
Alyssa Ross <hi@alyssa.is> writes:
Demi Marie Obenour <demiobenour@gmail.com> writes:
On 11/8/25 14:49, Alyssa Ross wrote:
Demi Marie Obenour <demiobenour@gmail.com> writes:
@@ -113,14 +76,15 @@ let
kernel = linux_latest;
- appvm = callSpectrumPackage ../../img/app { inherit (foot) terminfo; }; - netvm = callSpectrumPackage ../../vm/sys/net { inherit (foot) terminfo; }; + appvm = callSpectrumPackage ../../img/app { inherit (no_pgo_foot) terminfo; }; + netvm = callSpectrumPackage ../../vm/sys/net { inherit (no_pgo_foot) terminfo; };
# Packages that should be fully linked into /usr, # (not just their bin/* files). usrPackages = [ - appvm kernel.modules firmware netvm - ] ++ (with pkgsGui; [ mesa dejavu_fonts westonLite ]); + appvm kernel.modules firmware kmod kmod.lib + netvm mesa dejavu_fonts westonLite + ];
appvms = { appvm-firefox = callSpectrumPackage ../../vm/app/firefox.nix {};
So based on what we'd discussed previously, I was ready to just apply this and pare down the Nix changes myself since I don't have to worry about rebuilds, etc., but in doing so I came across this. Do you know why kmod.lib is required? Is something impurely dlopening it? Seems very strange, but I did verify that it's required for Weston to start.
systemd uses dlopen heavily. This allows dependencies to be optional at runtime. I remember strace showing that systemd-udevd dlopen's kmod.lib. In short, your diagnosis is correct.
Okay, that's a packaging bug in Nixpkgs then. (See e.g. [1].) I'll see what can be done.
[1]: https://github.com/NixOS/nixpkgs/commit/2328731ad041735a2260698574ce6599591f...
I have looked into it a bit and am now even more confused. From what I can see, loading libkmod should be done by libsystemd-shared.so, which has kmod.lib in its RUNPATH.
According to https://wiki.debian.org/RpathIssue, only the RUNPATH of the executable is honored. The RUNPATH of libraries is not honored. For this to work, libsystemd-shared.so needs to have kmod.lib in its RPATH, and udevadm *may* need to not have a RUNPATH.
Right, so we might need to add --force-rpath to the patchelf invocation. I wonder why this doesn't affect every NixOS system, though. They certainly don't have /usr/lib/libkmod.so.2.
On 11/8/25 17:37, Alyssa Ross wrote:
Demi Marie Obenour <demiobenour@gmail.com> writes:
On 11/8/25 17:14, Alyssa Ross wrote:
Alyssa Ross <hi@alyssa.is> writes:
Demi Marie Obenour <demiobenour@gmail.com> writes:
On 11/8/25 14:49, Alyssa Ross wrote:
Demi Marie Obenour <demiobenour@gmail.com> writes:
> @@ -113,14 +76,15 @@ let > > kernel = linux_latest; > > - appvm = callSpectrumPackage ../../img/app { inherit (foot) terminfo; }; > - netvm = callSpectrumPackage ../../vm/sys/net { inherit (foot) terminfo; }; > + appvm = callSpectrumPackage ../../img/app { inherit (no_pgo_foot) terminfo; }; > + netvm = callSpectrumPackage ../../vm/sys/net { inherit (no_pgo_foot) terminfo; }; > > # Packages that should be fully linked into /usr, > # (not just their bin/* files). > usrPackages = [ > - appvm kernel.modules firmware netvm > - ] ++ (with pkgsGui; [ mesa dejavu_fonts westonLite ]); > + appvm kernel.modules firmware kmod kmod.lib > + netvm mesa dejavu_fonts westonLite > + ]; > > appvms = { > appvm-firefox = callSpectrumPackage ../../vm/app/firefox.nix {};
So based on what we'd discussed previously, I was ready to just apply this and pare down the Nix changes myself since I don't have to worry about rebuilds, etc., but in doing so I came across this. Do you know why kmod.lib is required? Is something impurely dlopening it? Seems very strange, but I did verify that it's required for Weston to start.
systemd uses dlopen heavily. This allows dependencies to be optional at runtime. I remember strace showing that systemd-udevd dlopen's kmod.lib. In short, your diagnosis is correct.
Okay, that's a packaging bug in Nixpkgs then. (See e.g. [1].) I'll see what can be done.
[1]: https://github.com/NixOS/nixpkgs/commit/2328731ad041735a2260698574ce6599591f...
I have looked into it a bit and am now even more confused. From what I can see, loading libkmod should be done by libsystemd-shared.so, which has kmod.lib in its RUNPATH.
According to https://wiki.debian.org/RpathIssue, only the RUNPATH of the executable is honored. The RUNPATH of libraries is not honored. For this to work, libsystemd-shared.so needs to have kmod.lib in its RPATH, and udevadm *may* need to not have a RUNPATH.
Right, so we might need to add --force-rpath to the patchelf invocation.
I wonder why this doesn't affect every NixOS system, though. They certainly don't have /usr/lib/libkmod.so.2.
Difference between musl and glibc is the first thing that comes to mind. -- Sincerely, Demi Marie Obenour (she/her/hers)
Demi Marie Obenour <demiobenour@gmail.com> writes:
On 11/8/25 17:37, Alyssa Ross wrote:
Demi Marie Obenour <demiobenour@gmail.com> writes:
On 11/8/25 17:14, Alyssa Ross wrote:
Alyssa Ross <hi@alyssa.is> writes:
Demi Marie Obenour <demiobenour@gmail.com> writes:
On 11/8/25 14:49, Alyssa Ross wrote: > Demi Marie Obenour <demiobenour@gmail.com> writes: > >> @@ -113,14 +76,15 @@ let >> >> kernel = linux_latest; >> >> - appvm = callSpectrumPackage ../../img/app { inherit (foot) terminfo; }; >> - netvm = callSpectrumPackage ../../vm/sys/net { inherit (foot) terminfo; }; >> + appvm = callSpectrumPackage ../../img/app { inherit (no_pgo_foot) terminfo; }; >> + netvm = callSpectrumPackage ../../vm/sys/net { inherit (no_pgo_foot) terminfo; }; >> >> # Packages that should be fully linked into /usr, >> # (not just their bin/* files). >> usrPackages = [ >> - appvm kernel.modules firmware netvm >> - ] ++ (with pkgsGui; [ mesa dejavu_fonts westonLite ]); >> + appvm kernel.modules firmware kmod kmod.lib >> + netvm mesa dejavu_fonts westonLite >> + ]; >> >> appvms = { >> appvm-firefox = callSpectrumPackage ../../vm/app/firefox.nix {}; > > So based on what we'd discussed previously, I was ready to just apply > this and pare down the Nix changes myself since I don't have to worry > about rebuilds, etc., but in doing so I came across this. Do you know > why kmod.lib is required? Is something impurely dlopening it? Seems > very strange, but I did verify that it's required for Weston to start.
systemd uses dlopen heavily. This allows dependencies to be optional at runtime. I remember strace showing that systemd-udevd dlopen's kmod.lib. In short, your diagnosis is correct.
Okay, that's a packaging bug in Nixpkgs then. (See e.g. [1].) I'll see what can be done.
[1]: https://github.com/NixOS/nixpkgs/commit/2328731ad041735a2260698574ce6599591f...
I have looked into it a bit and am now even more confused. From what I can see, loading libkmod should be done by libsystemd-shared.so, which has kmod.lib in its RUNPATH.
According to https://wiki.debian.org/RpathIssue, only the RUNPATH of the executable is honored. The RUNPATH of libraries is not honored. For this to work, libsystemd-shared.so needs to have kmod.lib in its RPATH, and udevadm *may* need to not have a RUNPATH.
Right, so we might need to add --force-rpath to the patchelf invocation.
I wonder why this doesn't affect every NixOS system, though. They certainly don't have /usr/lib/libkmod.so.2.
Difference between musl and glibc is the first thing that comes to mind.
Was my first thought too, but people (including Yureka, who can maybe confirm) have run musl NixOS systems, and I've never heard of this problem with them.
On 11/8/25 17:52, Alyssa Ross wrote:
Demi Marie Obenour <demiobenour@gmail.com> writes:
On 11/8/25 17:37, Alyssa Ross wrote:
Demi Marie Obenour <demiobenour@gmail.com> writes:
On 11/8/25 17:14, Alyssa Ross wrote:
Alyssa Ross <hi@alyssa.is> writes:
Demi Marie Obenour <demiobenour@gmail.com> writes:
> On 11/8/25 14:49, Alyssa Ross wrote: >> Demi Marie Obenour <demiobenour@gmail.com> writes: >> >>> @@ -113,14 +76,15 @@ let >>> >>> kernel = linux_latest; >>> >>> - appvm = callSpectrumPackage ../../img/app { inherit (foot) terminfo; }; >>> - netvm = callSpectrumPackage ../../vm/sys/net { inherit (foot) terminfo; }; >>> + appvm = callSpectrumPackage ../../img/app { inherit (no_pgo_foot) terminfo; }; >>> + netvm = callSpectrumPackage ../../vm/sys/net { inherit (no_pgo_foot) terminfo; }; >>> >>> # Packages that should be fully linked into /usr, >>> # (not just their bin/* files). >>> usrPackages = [ >>> - appvm kernel.modules firmware netvm >>> - ] ++ (with pkgsGui; [ mesa dejavu_fonts westonLite ]); >>> + appvm kernel.modules firmware kmod kmod.lib >>> + netvm mesa dejavu_fonts westonLite >>> + ]; >>> >>> appvms = { >>> appvm-firefox = callSpectrumPackage ../../vm/app/firefox.nix {}; >> >> So based on what we'd discussed previously, I was ready to just apply >> this and pare down the Nix changes myself since I don't have to worry >> about rebuilds, etc., but in doing so I came across this. Do you know >> why kmod.lib is required? Is something impurely dlopening it? Seems >> very strange, but I did verify that it's required for Weston to start. > > systemd uses dlopen heavily. This allows dependencies to be optional > at runtime. I remember strace showing that systemd-udevd dlopen's > kmod.lib. In short, your diagnosis is correct.
Okay, that's a packaging bug in Nixpkgs then. (See e.g. [1].) I'll see what can be done.
[1]: https://github.com/NixOS/nixpkgs/commit/2328731ad041735a2260698574ce6599591f...
I have looked into it a bit and am now even more confused. From what I can see, loading libkmod should be done by libsystemd-shared.so, which has kmod.lib in its RUNPATH.
According to https://wiki.debian.org/RpathIssue, only the RUNPATH of the executable is honored. The RUNPATH of libraries is not honored. For this to work, libsystemd-shared.so needs to have kmod.lib in its RPATH, and udevadm *may* need to not have a RUNPATH.
Right, so we might need to add --force-rpath to the patchelf invocation.
I wonder why this doesn't affect every NixOS system, though. They certainly don't have /usr/lib/libkmod.so.2.
Difference between musl and glibc is the first thing that comes to mind.
Was my first thought too, but people (including Yureka, who can maybe confirm) have run musl NixOS systems, and I've never heard of this problem with them.
For now I think it is just best to work around the bug. I will come back tomorrow with more info. -- Sincerely, Demi Marie Obenour (she/her/hers)
On 11/8/25 17:52, Alyssa Ross wrote:
Demi Marie Obenour <demiobenour@gmail.com> writes:
On 11/8/25 17:37, Alyssa Ross wrote:
Demi Marie Obenour <demiobenour@gmail.com> writes:
On 11/8/25 17:14, Alyssa Ross wrote:
Alyssa Ross <hi@alyssa.is> writes:
Demi Marie Obenour <demiobenour@gmail.com> writes:
> On 11/8/25 14:49, Alyssa Ross wrote: >> Demi Marie Obenour <demiobenour@gmail.com> writes: >> >>> @@ -113,14 +76,15 @@ let >>> >>> kernel = linux_latest; >>> >>> - appvm = callSpectrumPackage ../../img/app { inherit (foot) terminfo; }; >>> - netvm = callSpectrumPackage ../../vm/sys/net { inherit (foot) terminfo; }; >>> + appvm = callSpectrumPackage ../../img/app { inherit (no_pgo_foot) terminfo; }; >>> + netvm = callSpectrumPackage ../../vm/sys/net { inherit (no_pgo_foot) terminfo; }; >>> >>> # Packages that should be fully linked into /usr, >>> # (not just their bin/* files). >>> usrPackages = [ >>> - appvm kernel.modules firmware netvm >>> - ] ++ (with pkgsGui; [ mesa dejavu_fonts westonLite ]); >>> + appvm kernel.modules firmware kmod kmod.lib >>> + netvm mesa dejavu_fonts westonLite >>> + ]; >>> >>> appvms = { >>> appvm-firefox = callSpectrumPackage ../../vm/app/firefox.nix {}; >> >> So based on what we'd discussed previously, I was ready to just apply >> this and pare down the Nix changes myself since I don't have to worry >> about rebuilds, etc., but in doing so I came across this. Do you know >> why kmod.lib is required? Is something impurely dlopening it? Seems >> very strange, but I did verify that it's required for Weston to start. > > systemd uses dlopen heavily. This allows dependencies to be optional > at runtime. I remember strace showing that systemd-udevd dlopen's > kmod.lib. In short, your diagnosis is correct.
Okay, that's a packaging bug in Nixpkgs then. (See e.g. [1].) I'll see what can be done.
[1]: https://github.com/NixOS/nixpkgs/commit/2328731ad041735a2260698574ce6599591f...
I have looked into it a bit and am now even more confused. From what I can see, loading libkmod should be done by libsystemd-shared.so, which has kmod.lib in its RUNPATH.
According to https://wiki.debian.org/RpathIssue, only the RUNPATH of the executable is honored. The RUNPATH of libraries is not honored. For this to work, libsystemd-shared.so needs to have kmod.lib in its RPATH, and udevadm *may* need to not have a RUNPATH.
Right, so we might need to add --force-rpath to the patchelf invocation.
I wonder why this doesn't affect every NixOS system, though. They certainly don't have /usr/lib/libkmod.so.2.
Difference between musl and glibc is the first thing that comes to mind.
Was my first thought too, but people (including Yureka, who can maybe confirm) have run musl NixOS systems, and I've never heard of this problem with them.
I did the following tests: 1. Move kmod and kmod.libs from usrPackages to packages. Module loading breaks. 2. Switch from pkgsGui.systemd (musl) to pkgs.systemd (glibc). Module loading works again. My conclusion is that this problem is specific to musl. Would it be okay to merge this with the workaround, and revert it later when it is no longer needed? -- Sincerely, Demi Marie Obenour (she/her/hers)
Demi Marie Obenour <demiobenour@gmail.com> writes:
On 11/8/25 17:52, Alyssa Ross wrote:
Demi Marie Obenour <demiobenour@gmail.com> writes:
On 11/8/25 17:37, Alyssa Ross wrote:
Demi Marie Obenour <demiobenour@gmail.com> writes:
On 11/8/25 17:14, Alyssa Ross wrote:
Alyssa Ross <hi@alyssa.is> writes:
> Demi Marie Obenour <demiobenour@gmail.com> writes: > >> On 11/8/25 14:49, Alyssa Ross wrote: >>> Demi Marie Obenour <demiobenour@gmail.com> writes: >>> >>>> @@ -113,14 +76,15 @@ let >>>> >>>> kernel = linux_latest; >>>> >>>> - appvm = callSpectrumPackage ../../img/app { inherit (foot) terminfo; }; >>>> - netvm = callSpectrumPackage ../../vm/sys/net { inherit (foot) terminfo; }; >>>> + appvm = callSpectrumPackage ../../img/app { inherit (no_pgo_foot) terminfo; }; >>>> + netvm = callSpectrumPackage ../../vm/sys/net { inherit (no_pgo_foot) terminfo; }; >>>> >>>> # Packages that should be fully linked into /usr, >>>> # (not just their bin/* files). >>>> usrPackages = [ >>>> - appvm kernel.modules firmware netvm >>>> - ] ++ (with pkgsGui; [ mesa dejavu_fonts westonLite ]); >>>> + appvm kernel.modules firmware kmod kmod.lib >>>> + netvm mesa dejavu_fonts westonLite >>>> + ]; >>>> >>>> appvms = { >>>> appvm-firefox = callSpectrumPackage ../../vm/app/firefox.nix {}; >>> >>> So based on what we'd discussed previously, I was ready to just apply >>> this and pare down the Nix changes myself since I don't have to worry >>> about rebuilds, etc., but in doing so I came across this. Do you know >>> why kmod.lib is required? Is something impurely dlopening it? Seems >>> very strange, but I did verify that it's required for Weston to start. >> >> systemd uses dlopen heavily. This allows dependencies to be optional >> at runtime. I remember strace showing that systemd-udevd dlopen's >> kmod.lib. In short, your diagnosis is correct. > > Okay, that's a packaging bug in Nixpkgs then. (See e.g. [1].) I'll see > what can be done. > > [1]: https://github.com/NixOS/nixpkgs/commit/2328731ad041735a2260698574ce6599591f...
I have looked into it a bit and am now even more confused. From what I can see, loading libkmod should be done by libsystemd-shared.so, which has kmod.lib in its RUNPATH.
According to https://wiki.debian.org/RpathIssue, only the RUNPATH of the executable is honored. The RUNPATH of libraries is not honored. For this to work, libsystemd-shared.so needs to have kmod.lib in its RPATH, and udevadm *may* need to not have a RUNPATH.
Right, so we might need to add --force-rpath to the patchelf invocation.
I wonder why this doesn't affect every NixOS system, though. They certainly don't have /usr/lib/libkmod.so.2.
Difference between musl and glibc is the first thing that comes to mind.
Was my first thought too, but people (including Yureka, who can maybe confirm) have run musl NixOS systems, and I've never heard of this problem with them.
I did the following tests:
1. Move kmod and kmod.libs from usrPackages to packages. Module loading breaks.
2. Switch from pkgsGui.systemd (musl) to pkgs.systemd (glibc). Module loading works again.
My conclusion is that this problem is specific to musl.
Would it be okay to merge this with the workaround, and revert it later when it is no longer needed?
You're correct. I asked in #exotic:nixos.org if anybody has encountered anything like this, and I was linked to a musl patch for this very issue[1]. So given there's already been an upstream fixed proposed, I'm fine with adding it to the upstream bug tracking board and doing the workaround for now. [1]: https://inbox.vuxu.org/musl/20251017-dlopen-use-rpath-of-caller-dso-v1-1-46c...
On 11/8/25 16:37, Alyssa Ross wrote:
Demi Marie Obenour <demiobenour@gmail.com> writes:
On 11/8/25 14:49, Alyssa Ross wrote:
Demi Marie Obenour <demiobenour@gmail.com> writes:
@@ -113,14 +76,15 @@ let
kernel = linux_latest;
- appvm = callSpectrumPackage ../../img/app { inherit (foot) terminfo; }; - netvm = callSpectrumPackage ../../vm/sys/net { inherit (foot) terminfo; }; + appvm = callSpectrumPackage ../../img/app { inherit (no_pgo_foot) terminfo; }; + netvm = callSpectrumPackage ../../vm/sys/net { inherit (no_pgo_foot) terminfo; };
# Packages that should be fully linked into /usr, # (not just their bin/* files). usrPackages = [ - appvm kernel.modules firmware netvm - ] ++ (with pkgsGui; [ mesa dejavu_fonts westonLite ]); + appvm kernel.modules firmware kmod kmod.lib + netvm mesa dejavu_fonts westonLite + ];
appvms = { appvm-firefox = callSpectrumPackage ../../vm/app/firefox.nix {};
So based on what we'd discussed previously, I was ready to just apply this and pare down the Nix changes myself since I don't have to worry about rebuilds, etc., but in doing so I came across this. Do you know why kmod.lib is required? Is something impurely dlopening it? Seems very strange, but I did verify that it's required for Weston to start.
systemd uses dlopen heavily. This allows dependencies to be optional at runtime. I remember strace showing that systemd-udevd dlopen's kmod.lib. In short, your diagnosis is correct.
Okay, that's a packaging bug in Nixpkgs then. (See e.g. [1].) I'll see what can be done.
[1]: https://github.com/NixOS/nixpkgs/commit/2328731ad041735a2260698574ce6599591f...
Does it make sense to use the workaround for now until the upstream bug can be fixed? -- Sincerely, Demi Marie Obenour (she/her/hers)
This patch has been committed as ced08f7ebdadfb07d34ef6a0e6016e8e6d2a24b1, which can be viewed online at https://spectrum-os.org/git/spectrum/commit/?id=ced08f7ebdadfb07d34ef6a0e601.... This is an automated message. Send comments/questions/requests to: Alyssa Ross <hi@alyssa.is>
participants (3)
-
Alyssa Ross -
Alyssa Ross -
Demi Marie Obenour