Apparently the last time I wrote an update like this was 2021-08-22[1]. Recently I've been missing them a lot, because I don't think it's ever been as easy to keep up with the progress being made as when I wrote weekly updates. Hopefully this time I can avoid going into quite the same level of detail, so they're more sustainable to keep writing. [1]: https://spectrum-os.org/lists/archives/spectrum-discuss/20210822213036.6lx5h... So, here's what I did this week: Automated testing ----------------- For a long time I've had a branch on my computer that makes some significant changes to how networking in Spectrum works to try to resolve a longstanding race condition when starting VMs. I've found it difficult to work on, because it's hard for me to be confident when changing things around that I'm not going to subtly break something in a different way. It's been clear to me for a long time that the obvious answer to this is automated tests, but how they should work was a lot less clear. I'd like to avoid having to make a special testable version of the system, because I want to be testing the real thing. The aarch64 port I completed early this year has helped make the system just a bit more flexible, and a couple of weeks ago it clicked for me that perhaps it's time to make a serious attempt at this. So far, I have a program that starts Spectrum in QEMU. This is an unmodified Spectrum image, but the test uses systemd-stub's support for appending to the kernel command line via through SMBIOS Type 11 strings[2] to configure an extra serial console, which it then communicates with over a socket. Because this console was configured on the kernel command line, it has kernel logs printed to it, which could make trying to read command output from it messy. To work around that, the only interaction the test has with this console is to write a command into it to start a shell on a second serial console that isn't a destination for kernel logs, and then further interaction happens via that second console. (I tried lowering the kernel loglevel, but it seems some messages come through even when it's set to zero.) The test program then uses the Spectrum host shell to mount a filesystem on an attached disk image containing a custom VM configuration, import that VM, and run it. The VM just runs netcat to try to connect a server running on the host via the test program. If the host receives that connection, the test will pass, and once I have that working I can expand the test to make sure that it keeps working after restarting the different VMs involved. What's still to do is getting that server bit right — originally I planned to use QEMU's guestfwd[3] feature to forward connections to a specific IP address and port to an AF_UNIX socket, but that been broken for a long time[4], so instead I'll just use an AF_INET socket on the host. Since it has to be a non-loopback address, I also want the test to run itself in a network namespace so I'm not interfering with networking on the machine running it. Nothing that should be especially difficult. An early experiment with this led me to discover that Python's built-in socket library didn't expose the IP_FREEBIND socket option, so I sent a quick PR to do that[5], which was merged. [2]: https://www.freedesktop.org/software/systemd/man/latest/systemd-stub.html#SM... [3]: https://www.qemu.org/docs/master/system/invocation.html#hxtool-5 [4]: https://gitlab.com/qemu-project/qemu/-/issues/1835 [5]: https://github.com/python/cpython/pull/132998 User-mode networking for development ------------------------------------ When developing the built-in Spectrum VMs, I run them directly on my NixOS development system. For the router VM in particular, it's important to test that it still connects to the network. Until now, that's only been possible with QEMU, which has built-in slirp-based user-mode networking — other VMMs would require privileged operations on the development system to have working networking, and it's important to me that the Spectrum development environment does not require special privileges. While I was working on the test and investigating QEMU's guestfwd feature, I discovered that there's a new option for user-mode networking: passt[6][7]. passt runs outside the VMM process, and implements a vhost-user interface, meaning that it should be possible to use from any VMM. I've changed the Spectrum development environment to use it when running VMs in Cloud Hypervisor[8], and also to run the router VM in Cloud Hypervisor by default[9], which means it's now easier to test that VM during development in an environment much more similar to the one it will ultimately run in as part of the Spectrum system. I also tried using passt with crosvm, but it didn't work due to crosvm's slightly strange interpretation of the vhost-user protocol. I sent a couple of patches[10][11] to crosvm to fix the incompatibility, which haven't been accepted yet but have prompted things to move in the right direction. I also sent a patch to passt itself to make it easier to use with an inetd/UCSPI-style super-server like s6-ipcserver, but a day later it still hasn't appeared in their list archive so I think it's probably waiting for somebody to find it in the list spam filter. [6]: https://passt.top/ [7]: https://www.qemu.org/docs/master/system/devices/net.html#using-passt-as-the-... [8]: https://spectrum-os.org/git/spectrum/commit/?id=aac74f6165740a6b041a7205ec86... [9]: https://spectrum-os.org/git/spectrum/commit/?id=ca10038c080f396a6cf2649b726b... [10]: https://chromium-review.googlesource.com/c/crosvm/crosvm/+/6491650 [11]: https://chromium-review.googlesource.com/c/crosvm/crosvm/+/6491651 Nixpkgs ------- I contributed a few fixes for build regressions that would get in the way of updating Spectrum's pinned Nixpkgs[12][13][14], and I also added debug symbols to the passt package[15]. [12]: https://github.com/NixOS/nixpkgs/pull/401563 [13]: https://github.com/NixOS/nixpkgs/pull/402027 [14]: https://github.com/NixOS/nixpkgs/pull/402055 [15]: https://github.com/NixOS/nixpkgs/pull/401971 That's it for this week. I'm at just over a thousand words, which is more than I was aiming for but better than I remember a lot of the past weekly updates being. Hopefully I'll be back with another update next week. As a reminder, you can support my work by donating to Spectrum's development through GitHub Sponsors[16] or Liberapay[17]. This helps a lot, especially with ongoing maintenance (usually fixing upstream bugs etc.) that is difficult to fund via a grant. [16]: https://github.com/sponsors/alyssais/ [17]: https://liberapay.com/qyliss