- Spectrum Discuss - spectrum-os.org

virtio-gpu in cloud-hypervisor: which path do we take?
by Alyssa Ross 17 Aug '22

17 Aug '22

Recently I've been working on making it possible for us to use crosvm's implementation of virtio-gpu (which is necessary for multi-VM Wayland). The approach I was originally planning on was porting crosvm's vhost-user-gpu frontend to cloud-hypervisor. That would allow us to run the crosvm implementation of the device unmodified, with just a small amount of glue code in cloud-hypervisor. But then I discovered some things that made me decide to investigate other approaches: - crosvm does not implement the standard vhost-user-gpu protocol. It implements it its own special way. Perhaps ironically, the crosvm-specific way seems to be closer to how vhost-user works for other devices (like network and block), which should actually make it easier to port the frontend to cloud-hypervisor. But it also changes the potential to upstream that port to cloud-hypervisor from "a hard sell" to "not going to happen". So if I did that, it would commit us to carrying a cloud-hypervisor patch indefinitely. - There's an interesting new protocol called vfio-user, that would be really helpful to us in this situation. Whereas vhost-user requires the VMM to still have some basic per-device knowledge (the glue code I was planning to port), vfio-user operates at the PCI level, so the VMM only needs to know that the device is PCI. So if we could somehow provide a virtio-gpu device to cloud-hypervisor over vfio-user, cloud-hypervisor wouldn't need any GPU-specific code at all. Everything should just work without any changes to cloud-hypervisor, as it already implements a vfio-user client. - The next release of QEMU, 7.1.0, will include support for exporting any virtual device QEMU can provide over vfio-user. So with all this in mind, there are three ways we could try to proceed: 1. Port the crosvm-specific vhost-user-gpu frontend to cloud-hypervisor. 2. Make crosvm speak the standard version of vhost-user-gpu, then use QEMU to act as a bridge between the crosvm GPU device, and cloud-hypervisor, translating vhost-user to vfio-user, but not doing anything else. (So we're not using QEMU to run a VM, just to translate between these two protocols and handle the PCI stuff.) 3. Implement a vfio-user server in crosvm, so crosvm device backends can be used directly with cloud-hypervisor. 3 is the clear best option, because it doesn't require adding QEMU into the system, and it would be entirely upstreamable — I spoke to a crosvm developer about it in their Matrix channel and they said they'd be interested in patches for it. But it's also quite complicated to implement, and beyond my ability, at least if I want results any time soon. 2 would probably be even more complicated as it would require coordinating between crosvm and QEMU to get them to agree on how the protocol should work. So if we want this working soon, 1 is the only feasible option, at least if it's me doing the work. But it means committing to carrying a cloud-hypervisor patch until somebody comes along to implement 3. It gives me bad vibes because it goes against the upstream-first approach I try to take with Spectrum development, and once timely package updates are something we have to take more seriously, the patch no longer applying would block any sort of automatic updates. So I'm posting this to solicit thoughts on what to do here. The ideal scenario is that we are able to find somebody else (with more VMM implementation experience than me) who is able to do 3, and then I would be totally comfortable with doing 1 as a stopgap until that can happen. Otherwise, I can either keep trying to chip away at doing it myself, however long that takes, or we'd have to just accept the consequences of having the patch indefinitely, and hope that Google or somebody else also finds themself wishing crosvm had a vfio-user server and implements it themself. Thoughts welcome. :)

1 0

Unikie has hired me to work on Spectrum
by Alyssa Ross 09 Aug '22

09 Aug '22

If you've been paying close attention recently, you'll have seen patches coming from a few different Unikie[1] email accounts. In addition to contributing to Spectrum, Unikie has hired me to work on it. Unikie is interested in Spectrum for both desktop and embedded use. They're contracting for the TII Secure Systems Research Center[2], working on developing a Spectrum-based reference system. One thing they're currently working on is being able to run Spectrum on the i.MX8 development board, which means we'll hopefully see patches adding ARM and cross-compilation support to Spectrum in the near future. The first big thing I'll be working on for Unikie is finally integrating crosvm's support for graphical application VMs into Spectrum. While I'm working for Unikie, I plan to set aside any donations I receive for my Spectrum work, to be used for project expenses and funding Spectrum work from other people, rather than being used to pay for my general living expenses as has been the case up to now. And just to clarify: Spectrum has not been acquired or anything — I'm still leading the project, and Unikie has the same rights as any other contributor (copyright ownership of contributions, mostly). [1]: https://www.unikie.com/en/ [2]: https://www.tii.ae/secure-systems

3 2

Reportback from MCH 2022
by Alyssa Ross 09 Aug '22

09 Aug '22

I was recently at MCH 2022[1], one of the big European hacker camps. We had some really good conversations about Spectrum, and I thought I'd share my takeaways here: 1. We were praised for our recent documentation efforts, both in implementing Diátaxis[2] and Architecture Decision Records[3]. So big thanks to Ville for spearheading the latter. 2. We talked about the use case of having multiple user data partitions. This would allow very strict separation of security domains, and could also be helpful for data portability — you could have one user data partition in your desktop, and another on a portable disk, for example. And if, way down the line, we want to do really cool things like have live migration of VMs between systems, architecting for multiple user data partitions will be a big help with that too. This is one of those things where it's not difficult to do, as long as we plan for doing it that way from the start. But if we didn't do it that way from the start, and decided we wanted to add it later, I can see how we'd be in for a world of pain. So I think it's a sensible change to make. We're unlikely to regret making it, but reasonably likely to regret not having done it earlier if it becomes really important later on. 3. Something that can apparently be difficult for Qubes is having every VM have a unique, human-readable name in a global namespace. This means that, for example, disposable VMs have to try to generate a name that isn't already in use. This is especially relevant if we end up supporting multiple sources of VMs as described above. So in the short term, we should probably change VMs to be identified with UUIDs, and have human-readable names be a layer on top. Not having a human-readable unique names in a single global namespace will help with thinking about VMs in terms of capabilities. Since points 2 and 3 are architectural changes, I'll write them up and submit them as proper ADRs when I get the chance. [1]: https://mch2022.org/ [2]: https://diataxis.fr/ [3]: https://spectrum-os.org/doc/decisions/

1 0

Workflow preferences - case aarch64 support
by Ville Ilvonen 20 Jun '22

20 Jun '22

Hi all, Good news is that the work on aarch64 support has progressed to the point where the Spectrum host can be built natively and run on aarch64 HW. It's still a bunch of patches in various stages of WIP or review to spectrum or further up the stream. In short, to "nixpkgs-spectrum": 0001-Allow-static-building-of-dtc.patch 0002-Workaround-ld-script-segmentation-fault.patch /* may be not needed */ 0003-Add-imx8-specific-kernel.patch and to "spectrum": 0001-Add-imx8-kernel.patch Should anyone want to reproduce, please let me know and I'll try to find a solution to share them. Probably worth waiting for cross-compilation support unless you have powerful aarch64 at your disposal. In the meanwhile, we try to establish a mechanism to collaborate in branches of required upstream repos. It's fairly impractical to fork "nixpkgs-spectrum"-repo due to its' size, over 2GB, Which takes me to the discussion point - what's the workflow proposal on multiple people working on the same functionality in the same branch? If I'm not mistaken the email workflow creates a new branch for each author? Then someone would have to cherry-pick them to the same branch to integrate them in dev time hydra builds. Best, -Ville

2 1

New documentation site is live!
by Alyssa Ross 19 May '22

19 May '22

We finally have an online render of the new documentation I've been writing: https://spectrum-os.org/doc/ Let me know if anything doesn't look right. This documentation lives in the "Documentation" directory in the Spectrum repository. The online version will be automatically updated whenever changes are pushed. I'm not sure what to put on the index page. I think maybe it would make sense to just merge all the content that's currently on the rest of the website into the documentation, but I'm not sure yet. It's going to be really important for Spectrum to have solid documentation. I recently watched a talk about the Diátaxis system[1], and I think it would be a good model for us to follow. [1]: https://diataxis.fr/

1 0

Documentation and diagrams
by ville.ilvonen＠unikie.com 19 May '22

19 May '22

Hi, I asked thoughts on Spectrum documentation improvements some time back on #spectrum. I'd like to improve documentation in terms of architecture (views, description, diagrams) to support communication with people adopting Spectrum in our research/engineering project(s). Of course I can create presentations for such purposes but those are not supporting the project. There's nothing wrong with asciidoc but as of now it lacks support for diagrams. Another view to diagrams was how we want to create diagrams - writing dsl(diagram-specific-language)-to-diagrams or drawing. Based on our #spectrum discussion I was guided to asciidoc diagram extension - https://docs.asciidoctor.org/diagram-extension/latest/ - so I finally gave it a try last Friday with ditaa - an ascii-to-png-conversion. Long story short - I got it working by adding dependencies to jdk8 etc. but I'm not happy with the result for following reasons: Limitations: - ditaa (and other dsl-to-diagrams) have limited support in either diagram dsl comprehension or available editors. Namely, ascii-based box drawing programs exist (e.g. https://asciiflow.com/#/) but do not directly support ditaa. If ascii-drawings can be modified to ditaa, it's slow, error-prone and worst of all - importing version controlled ditaa-ascii-graph text files back to editors do not work as object imports so in worst case the objects need to be drawn again when one wants to modify the diagrams. - dsls have learning curves with different slopes. Some are easy, some take domain level understanding (like UML) in addition to dsl but it's there. - layout algorithms - many cases go quite ok with this but then there's points where the layout algorithms fail miserably with solutions beyond reasonable effort. - dsls are usually not readable in source format but beyond trivial diagrams always require validation visually Benefits: + Some asciidoc diagram-extension dsl:s (like plantuml, graphviz) work quite ok for both visualizing UML, graphs, dependencies, trees. + Focus on content, not presentation - kind of like LaTeX. Though this is limited by tools like ditaa which can get worst of both worlds. + dsls can be great for code level views. I've used tool where dsl has been combined into generating the diagrams from code or tree like structures. In fact, there's quite a few such tools to do this with nix-store dependency trees and have been useful in spectrum architecture comprehension. What I'd like to propose that in addition to aiming at text-to-diagrams we support diagram drawing tool(s) and agree on either png or svg with diagrams. With diagram drawing tools, we should store the diagrams in editable format in version control (not only png and svg). My 2 cents would go to https://github.com/jgraph/drawio but I'm fine with others as well - as long as they do not require commercial license. I'd like to hear thoughts/decisions on this before sending any patches to contribute to spectrum architecture documentation. Best regards, -Ville

6 11

Spectrum status update, April 2022
by Alyssa Ross 01 May '22

01 May '22

The big news this month is that Puck is now working with us as a Wayland developer[1]. We also welcomed a bunch of new people into the chat channel who were interested in building Spectrum to try it out for themselves, and contributing to it or related projects, which is absolutely fantastic. [1] https://spectrum-os.org/lists/archives/spectrum-discuss/20220416092718.isdz… Here's what I did in April: Binary Cache ════════════ Using only the cache.nixos.org binary cache, building Spectrum and all its many dependencies could take more than 24 hours on commodity hardware, which was a big turn-off to the people who've been trying to build Spectrum for themselves recently. To mitigate this, this month I put together a binary cache solution for Spectrum[2], and wrote documentation explaining how to use it[3]. Thanks again to Daniel Kuehn (lejonet) for providing the storage hosting for the binary cache. [2] https://spectrum-os.org/lists/archives/spectrum-devel/20220414141547.rafken… [3] https://spectrum-os.org/git/spectrum/tree/Documentation/binary-cache.adoc Hardware support ════════════════ This month, quite a few people were interested in testing Spectrum on their hardware. This surfaced quite a few issues that have now been resolved: • Avoid EFI chainloading in the combined image.[4] • Use a bigger kernel buffer for uevents.[5] As well as one issue still in flight: • Rebroadcast uevents not handled by mdevd.[6] And one issue still being investigated. Additionally, I started to look into what Aarch64 support in Spectrum would look like. I was surprised to discover that Nixpkgs' musl stdenv hadn't even bootstrapped on aarch64 for three years! So I commited some hacks to fix that[7], and put together a follow-up that will make it maintainable going forward[8]. [4] https://spectrum-os.org/git/spectrum/commit/?id=b1ff26eb27ea905357f8bf217d0… [5] https://spectrum-os.org/git/spectrum/commit/?id=0b3b366653030de498e73e51b1c… [6] https://spectrum-os.org/lists/archives/spectrum-devel/20220424080020.32216-… [7] https://github.com/NixOS/nixpkgs/pull/169764 [8] https://github.com/NixOS/nixpkgs/pull/169793 Documentation ═════════════ I spent a lot of this month working on documentation. Having good documentation is going to be really important for bringing on new contributors. I expect to spend a bunch of time on this next month, too. • New documentation: a UUID reference[9], and information about the binary cache[3]. • A proper documentation site, based on Just the Docs[10]. Unfortunately not quite available online yet, but you can see it with nix-shell –run 'jekyll serve' in the Documentation directory. [9] https://spectrum-os.org/git/spectrum/tree/Documentation/uuid-reference.adoc [10] https://just-the-docs.github.io/just-the-docs/ Miscellaneous ═════════════ • I did a short development stream.[11] • Updated Nixpkgs, reducing our patch count by half. • Switched to s6-linux-init from a hand-rolled script, eliminating a longstanding race condition during boot. • Adopted systemd-boot, which will make Spectrum more dual-boot friendly.[12]. • Dropped s6-portable-utils dependency[13], and OVMF development dependency[14]. • Various cleanups to licensing metadata. • Fixed building Spectrum on Ubuntu[15]. • Minor cleanups to VM startup[16][17][18][19]. This is preparatory work in advance of new VM features. [11] https://diode.zone/w/p2Tythup7zJTSAgiGbJ331 [12] https://spectrum-os.org/git/spectrum/commit/?id=b72a136e9ae265f61f5c6dbbd37… [13] https://spectrum-os.org/git/spectrum/commit/?id=04060e0a59c072315b7b5106af2… [14] https://spectrum-os.org/git/spectrum/commit/?id=969842dd0b86922383b3b5e0306… [15] https://spectrum-os.org/git/spectrum/commit/?id=a567476380303fe0ccb0c136ad7… [16] https://spectrum-os.org/git/spectrum/commit/?id=ab9d7bf6d7e7b4cdd77933d5799… [17] https://spectrum-os.org/git/spectrum/commit/?id=9226197f1e9b9f17caf501b2a80… [18] https://spectrum-os.org/git/spectrum/commit/?id=0543017145207d9ea41b6140722… [19] https://spectrum-os.org/git/spectrum/commit/?h=32b701064a15f0045bce2d811a94… Spectrum related upstream Nixpkgs commits ═════════════════════════════════════════ • s6-rc: 0.5.3.0 -> 0.5.3.1 (1ee8b37a851) s6: 2.11.0.1 -> 2.11.1.0 • (bd6d4a6e67f) s6-portable-utils: 2.2.3.4 -> 2.2.4.0 (a3924d70655) • utmps: 0.1.1.0 -> 0.1.2.0 (92b74337544) skalibs: 2.11.1.0 -> • 2.11.2.0 (5eafa6cc7a7) patchelf: clarify license (3a73bf2b6e3) • pkgsMusl.coreutils: fix build on aarch64 (661dfd83478) patchelf: use • 0.13.x on aarch64+musl (e22d0b49a95) patchelf_0_13: init at 0.13.1 • (3838a0a7e7e) cloud-hypervisor: 22.1 -> 23.0 (430c56976fe) busybox: • 1.34.1 -> 1.35.0 (7d83997eb49) crosvm: 99.14468.0.0-rc1 -> • 100.14526.0.0-rc1 (40f140ad872) crosvm.updateScript: generate • Cargo.lock (26d66fbfa9b) squashfs-tools-ng: 1.1.3 -> 1.1.4 • (fd3f39303b4) qemu: remove redundant copy of qemu-ga (665bb90fc3f) • rust: 1.59.0 -> 1.60.0 (6d49a350807) openssh: 8.9p1 -> 9.0p1 • (4cfdd982653) busybox: fix CVE-2022-28391 (ac60e92b15a) Pending Spectrum related Nixpkgs PRs ════════════════════════════════════ • release.nix: build Musl native bootstrap tools (#169793) Spectrum infra related upstream Nixpkgs commits ═══════════════════════════════════════════════ • nixos/mailman: ensure Postfix is started after Mailman (572131c6a94) • ipxe: 1.21.1 -> unstable-2022-04-06 (7d691eadc37) • nixos/postfix: pull setup into its own unit (fb2fa1b50fe) • nixos/mailman: remove obsolete setting (e7c301df52e) • nixos/mailman: improve empty webHosts error (f4a0bb8334d)

1 1

Puck is joining Spectrum as a paid Wayland developer
by Alyssa Ross 16 Apr '22

16 Apr '22

Since I started working on Spectrum, one of the things I've been most excited about has been the potential of Wayland and related technologies to provide a level of integration and cohesion that hasn't been seen before in a compartmentalized operating system. It's the main thing that people who are aware of Spectrum but are not closely following its development know the project for. But people who /have/ been following my work closely will have noticed that I haven't had much to say about it for a while. My original grant agreement covered some basic Wayland functionality. With that funding, I was able to learn about, run (to my knowledge, for the first time outside the context of Chromium OS development) and experiment with virtio wayland. I produced documentation and a portable demo, that helped give other developments like Thomas's wayland-proxy-virtwl[1][2] their start. We're on a solid track to have basic Wayland functionality for applications running in isolated VMs, including things like inter-application copy/paste, and possibly optional GPU acceleration. But to have a system that provides the level of security I want Spectrum to have, there's still a lot of work to do. New protocols need to be designed, discussed, and implemented, not just in Spectrum, but in the Wayland ecosystem in general, to make it possible to do things like identify which application a Wayland window belongs to, or manage access to the clipboard. None of this is so big it's infeasible for Spectrum as a project, but it's a big time investment not covered by my original grant agreement for Spectrum, and so I've been shying away from it in favour of work with a more immediate return on investment, like my recent work creating an installer and live image for Spectrum, or figuring out how VM configuration will work. Recently, though, the stars aligned in a way that mean we should be able to make a lot of progress on these bigger Wayland developments much quicker than I was otherwise expecting. After a couple of months of working to make it happen, I'm so excited to finally be able to announce that we have secured a modest amount of additional funding to enable Puck Meerburg to work on Spectrum with me, focusing entirely on Wayland work. She will be designing the protocols we need, producing reference implementations, discussing them with upstream with an aim to get them standardised, and so on. One of the most exciting things about the work we have planned for Puck is that none of it is Spectrum-specific. In keeping with the general development philosophy for Spectrum, we want to move the ecosystem forward for everyone, rather than developing Spectrum-specific hacks that would be useless if the project ever wasn't able to continue. Puck is the ideal person to be working on this. She was already our resident Wayland expert, being able to answer just about any question I had when doing Wayland work, and even providing some patches of her own[3]. She has experience with standards processes through her work on ActivityPub. She has an amazing talent for finding bugs — just look at her list of Nix discoveries[4], and experience doing security researcher — she recently discovered a "critical"-rated security issue in Mastodon[5]. And she's proven over and over her ability to jump into an unfamiliar problem space and diagnose an issue at an amazing speed. I'm confident that having Puck spending a lot of time working with Spectrum is going to be a huge boon to the project even outside of the Wayland work she's specifically funded to work on. A final note: the grant funding for both Puck and myself expires at the end of September. What happens after that point is yet to be determined. Given the level of donations I currently receive, unless there's a sudden drop in my donation income, I'll should still be able to spend at least most of my time working on Spectrum, regardless of what happens with other funding sources. As I've said before, donation income is also extremely important to provide a level of stability and flexibility that grant funding just can't — it's important to know that I'm not going to be broke because it's taking longer than expected to implement something, or there's something unanticipated I need to do first that I don't have funding for. It would be really helpful if we could get some of that stability and flexibility for Puck as well, especially looking towards what happens once our grant funding is up. So Puck is now also accepting donations, and if you'd like to help Spectrum push forward Wayland security, please consider sponsoring her. https://github.com/sponsors/puckipedia I'm so excited for where things go from here. [1]: https://github.com/talex5/wayland-proxy-virtwl/ [2]: https://roscidus.com/blog/blog/2021/03/07/qubes-lite-with-kvm-and-wayland/ [3]: https://spectrum-os.org/git/nixpkgs/tree/pkgs/os-specific/linux/chromium-os… [4]: https://github.com/NixOS/nix/issues?q=author%3Apuckipedia+is%3Aissue [5]: https://github.com/mastodon/mastodon/releases/tag/v3.4.6

1 0

Spectrum Status Update, March 2022
by Alyssa Ross 02 Apr '22

02 Apr '22

Hi all, I thought I'd try a different format of update. It's difficult to find the time for the big This Week in Spectrum updates I've tried to do before, but I'd like to provide some sort of account of what I've been doing. So, here's an overview of what I did in March. I'm happy to expand on any of it that sounds interesting — just hit Reply All and tell me what you'd like to hear more about! I'm also interested to hear what you think of this status update format. I'd like to get better about communicating what I'm working on, in a sustainable way. Let me know if you have any suggestions! Miscellaneous ------------- • Edited and published demo video[1] • Set up an IRC bot to post incoming mailing list messages • Switched from Busybox's modprobe to kmod • Removed unused dependencies • Various other cleanups and fixes • Started work towards CI for Spectrum • Prototyped a shared base image for application VMs [1]: https://diode.zone/w/dWAWHR38Zu3feRtDKjVEJb virtiofs investigation ---------------------- • Prototyped virtiofs VM filesystem access • Reported a bug: "Can't run unprivileged any more due to setgroups"[2] • Participated in discussion and testing of Musl port[3][4][5] [2]: https://gitlab.com/virtio-fs/virtiofsd/-/issues/36 [3]: https://github.com/slp/capng/pull/2#issuecomment-1059976861 [4]: https://github.com/slp/capng/pull/3 [5]: https://github.com/rust-lang/libc/pull/2713 Spectrum-related upstream Nixpkgs commits ----------------------------------------- • lvm2: don't use targetPlatform (05a6c124e65) • coreutils: add debug output (e30f0f31e8d) • pkgsMusl.systemd: fix build for 250.4 (39eee39fd92) • nghttp2: only run tests on GNU (8685cea963b) • python3.pkgs.importlib-metadata: fix cross (3c7b77e638b) • spidermonkey: use the same LLVM as rustc (3ff5f0eb764) • pkgsStatic.stdenv.cc.cc: put static libs in $lib (12c37aec377) • Revert "gcc: Always pass `--enable-shared` by default" (c6dd11ca39a) • libudev-zero: 1.0.0 -> 1.0.1 (c7b7ad77985) • linux_latest: 5.16.14 -> 5.17 (58ae11758e8) • crosvm: 81.12871.0.0-rc1 -> 99.14468.0.0-rc1 (6aefdafbed9) • shadow: 4.8.1 -> 4.8.11 (8d35d7e2bf1) • pkgsMusl.libnetfilter_conntrack: fix build (2cc5ec86571) • pkgsMusl.systemdMinimal: fix build (b8734c50e29) • linux.configfile: fix alts containing "/m" (fb079c3110d) • cloud-hypervisor: 21.0 -> 22.0 (36a211e1ee3) • edk2: 202108 -> 202202 (9222b68380e) • kmod: add dev and lib outputs (dc1303185f8) • systemd: update patchShebangs comment (a0bfc8e7c1f) • systemd: fix a whole bunch of typos (479b1cb510b) Pending Spectrum related Nixpkgs PRs ------------------------------------ • crosvm: add support for virgl_renderer{,_next} (#165128) • qemu: 6.2.0 -> 7.0.0 (#165291) Spectrum infra related upstream Nixpkgs commits ----------------------------------------------- • irccat: init at 0.4.8 (ce8cbe3c01f) • git: enable debug info (4345b27dedf) • cgit-pink: init at 1.3.0 (deab83e1167) • mailman-web: fix django version check removal (3512f5b7075) Demo video related upstream Nixpkgs commits ------------------------------------------- • ccsymbols: init at 2020-04-19 (cf7556eea5a)

1 0

PlagueOS
by Geblaat 23 Mar '22

23 Mar '22

I recently came across Spectrum. I myself use Qubes OS quite a lot, and I also find Spectrum really interesting. (Though I'm not very technical, so I'll have to wait for development to be in another stage for me to try it.) On the mailing list, I saw an article about FlexOS, which has some similar goals, though it seems to be in very early stages. I thought PlagueOS might be of interest to people here as well. Like Spectrum, it has better hardware compatibility and performance compared to Qubes OS. They use Void OS as hypervisor(looks like standard KVM compared to a more secure Rust alternative) and a lot of hardening. It already has an ISO image available and a graphical desktop environment, so it can already be used by people with not much technical skills. discuss(a)spectrum-os.org

2 2