- Spectrum Discuss - spectrum-os.org

This week in Spectrum, 2020-W18
by Alyssa Ross 03 May '20

03 May '20

A belated Happy International Workers' Day. :) I had some other stuff going on this week but I'm still happy with what I got done. Wayfire ------- I've been working on packaging Wayfire in Nixpkgs so we can use it in Spectrum. (See last week's update for why I've chosen Wayfire for this.) This has required quite a bit of work on Wayfire itself, because up until now Wayfire has assumed that every component, including plugins, will be installed into Wayfire's prefix. With Nix, a package's prefix is only writable by that package, so we need to make some changes to Wayfire to support each component having its own prefix. This is necessary even for our simple case because even a basic Wayfire setup has at least one plugin to provide the desktop shell. Everything is supposed to be a plugin, after all. And we'll want to add our own plugins at some point. To implement plugins in other search paths, I added two environment variables WAYFIRE_PLUGIN_PATH (for libraries) and WAYFIRE_PLUGIN_XML_PATH (for the XML files that define Wayfire plugin options). These are then searched in addition to Wayfire's prefix. Support for plugins coming from multiple prefixes required some small changes in several parts of the Wayfire ecosystem. Wayfire has so far proven to be a very friendly upstream. They understood the problem I was having, even though it's less of an issue on traditional distributions, and I always got quick answers on IRC when I needed to ask for advice on implementing the search paths. Here's the issue I filed describing the need for search paths: https://github.com/WayfireWM/wayfire/issues/493 And here are all the pull requests I made to implement them across the ecosystem: https://github.com/WayfireWM/wayfire/pull/496 https://github.com/WayfireWM/wf-config/pull/25 https://github.com/WayfireWM/wayfire/pull/497 https://github.com/WayfireWM/wf-shell/pull/52 https://github.com/WayfireWM/wf-shell/pull/53 https://github.com/WayfireWM/wcm/pull/17 https://github.com/WayfireWM/wf-shell/pull/54 https://github.com/WayfireWM/wcm/pull/18 With these changes, I can have Wayfire all nicely packaged up. We can even provide a nice Nix interface for configuring a Wayfire with some specific plugins: (wayfireApplications.withPlugins (plugins: with plugins; [ wf-shell ])).wayfire I made a pull request to add Wayfire, with the interface demonstrated above, to Nixpkgs: https://github.com/NixOS/nixpkgs/pull/86606 Unbeknownst to me, while I was working on this, another PR was also opened to add the latest version of Wayfire to Nixpkgs: https://github.com/NixOS/nixpkgs/pull/86569 This PR does a good job of making the basic Wayfire installation work, but it doesn't support extra plugins. Still, it's likely that if this had been available before I started my work, it would have been good enough for us for now. This sort of thing is always a risk, and even if I hadn't done the Wayfire work now I'd have had to come back later anyway to do plugins, so it's not too bad. The author of that PR and I have agreed to work together from here to get Wayfire into Nixpkgs, using my PR as a base because it has the plugins support. So, hopefully by the time I write my next status update all my Wayfire pull requests will have been merged, and Wayfire will be in Nixpkgs. Reviews on the Nixpkgs PR in particular are very much appreciated. :) From here, I'll start trying to modify Wayfire/wlroots to make it request host memory rather than allocating its own, as described in last week's status update. I had a quick look, and it appears that there aren't many places these allocations are happening, so hopefully that shouldn't require too much modification. Then, we should be able to run an application in one VM, displayed on a compositor in another. :)

1 0

This Week in Spectrum, 2020-W17
by Alyssa Ross 26 Apr '20

26 Apr '20

It's been a bit of a slow week, I'm afraid. spectrum-vm ----------- As mentioned last week, it was becoming extremely inconvenient to test out custom kernel/crosvm/rootfs builds by modifying a Nix expression every time. So I wrote spectrum-vm, a small program that takes command line arguments for kernel, crosvm, and rootfs paths to use, and falls back to pre-built defaults if any of those aren't specified. This makes turnaround time on testing different kernel builds much quicker, because they can be built incrementally outside of nix sandboxes. To make things more concrete, suppose I have built a kernel in /home/src/linux/arch/x86_64/boot/bzImage, and I want to test it with the Spectrum crosvm demo setup. I can run spectrum-vm -k /home/src/linux/arch/x86_64/boot/bzImage and I get a VM booted from that kernel, with no need to nix-build anything. By default, the launched VM will run Weston Terminal inside a Wayland compositor. But you can even replace this with a custom command: spectrum-vm -c 'echo "Hello, world!"' The implementation of that last bit is quite the hack. I'll leave figuring out how it works as an exercise to the reader. ;) spectrum-vm is available now in Spectrum's Nixpkgs as spectrumPackages.spectrum-vm, as of commit 817a1a9b0de30d8bbc2ad2e80a802d87c56e96ff. Implementing this has meant that the huge start-vm.nix file that used to exist on the crosvm-demo branch had to be split up into a bunch of discreet units. This is really great, because start-vm.nix was starting to grow enormous, and experimenting with changes was becoming A Problem. Now, that's all nice and modular under the new spectrumPackages namespace. And _that_ means that the crosvm-demo branch can finally go away, and Spectrum's Nixpkgs is no longer split across two branches. (The purpose of the separation was to avoid there being a horrible start-vm.nix file in the root on master.) So in addition to a lovely helpful tool, this has also resulted in much cleaner code. Yay! One feature I would still like to add is an option that maps to crosvm run's --shared-dir, to mount a directory into a VM over 9p. That would make getting code into a VM to test it extremely convenient. I haven't gotten around to it yet though. Guest memory sharing -------------------- Last week I shared a bunch of research I had done on sharing guest-allocated buffers with other VMs. I was hoping to avoid having to patch a Wayland compositor to request buffers from the host instead of allocating them itself, but I've since come to the conclusion that avoiding that won't be feasible, for now at least. Basically, the kernel primitives aren't there yet, and it would be silly for me, with almost zero kernel development experience, to attempt to write them. There are all sorts of horrible edge cases to think about, like what happens if the guest-allocated memory moves around. Compared to this, patching a Wayland compositor doesn't seem too bad. Having to go through a request/response over a vsock to the host will be a bit of a performance overhead compared to a smarter solution, but I think this kind of allocation happens rarely enough that it probably doesn't matter anyway. A nice middle ground would have been to use virtio-gpu resources rather than a custom message over a vsock. It would have been a much cleaner way to do it, and it might not even involve writing any kernel code, but the practical benefits of it would be extremely limited when contrasted with the time it would take me to implement that. I don't expect the approach I'm taking now of patching the Wayland compositor to necessarily be permanent. As I linked to last week, there are kernel efforts happening to add the necessary primitives to share guest allocations. At some point in the future we might be able to use them. And at basically any time we could switch to using virtio-gpu resources if somebody wanted to put in the work. Patching a compositor really isn't a big deal, though. It's unlikely Spectrum will support more than one compositor anyway, because we'll need to customise it to do things like Qubes-style window borders that are coloured based on security domain. And, even then, if we're using a wlroots-based compositor, it's probably only wlroots that needs patching, so in that case any other wlroots-based compositor would probably work. Speaking of compositors... Wayfire ------- If you know what Wayfire is, you're probably surprised to see it as an item in a Spectrum status update it. If you don't, Wayfire is a Wayland compositor notable for having a bunch of over-the-top Compiz-style graphical effects, like wobbly windows and the titular fire. I recommend checking out the demo videos on https://wayfire.org/ if you haven't seen them -- it's quite something. But I bring this up not because I think support for wobbly windows is imperative to Spectrum's chances of success, but because fundamentally Wayfire is just an extremely modular Wayland compositor. Virtually everything in Wayfire is implemented as a plugin. Switching between windows is a plugin, screen zoom is a plugin, and so on. This is really cool because in Spectrum we'll want to have, for example, coloured window decorations like in Qubes. With any other compositor, we'd have to hack that into the compositor internals somehow. With Wayfire, it can just be a plugin. Wayfire is also a pleasantly simple but featureful windowing experience. You can drag windows around, put two side by side by dragging them to screen edges, have multiple workspaces, etc. I think it will be straightforward enough to not be a problem for people coming to Spectrum regardless of their computing background, and powerful enough to not get in the way of Serious Computer Business. And anything missing could of course be implemented as a plugin. ;) So, I've been doing some work to package Wayfire in Nixpkgs. This is not all that easy, because Wayfire seems to assume that plugins will be installed into Wayfire's prefix. This isn't really possible with Nix unless you compile all the plugins along with Wayfire, so I'm working some patches for Wayfire to look up plugins in a search path specified as an environment variable, which on the Nix side we can then wrap in the idiomatic wayfireWithPackages style. My initial impressions of the Wayfire codebase are that it is relatively clean, and straightforward enough to understand. The core compositor is about 13000 SLoC, which is pleasantly small. That, unfortunately, is it for this week. I didn't get as much done as I'd hoped, but I did at least have loads of good thoughts about things to do later in Spectrum's development, that aren't quite concrete enough to put into words yet.

1 0

This Week in Spectrum
by Alyssa Ross 19 Apr '20

19 Apr '20

Wow, a week isn't very long at all. But still, there's definitely enough to write about, even if I spent most of the week just reading mailing list threads. More on that in a bit. Spectrum Live + nixpkgs ----------------------- I did another stream. This time, we looked at updating the Chromium OS packages in Spectrum. The script I used to update them had broken -- it was stuck on an old version, so first we had to fix the script, and then fix the package builds. A lot of the Chromium OS packages aren't actually necessary at the moment, but I made them a while ago when it looked like they would be. I couldn't get one particular such package, libchrome, to build any more at all after spending quite a few hours on it, so I ended up just dropping it, along with libbrillo, which depended on it. Some other package updates had some complexity, but we managed to work through them all on the stream. This stream was much more focused on what I was working on than the last one. The last one was more about people asking general questions and me answering them and demonstrating things. This time I stayed on the package updates for the whole stream. I figure people asked all their questions last time. But there were lots of questions about specific issues in the package updates, etc. It was nice to know that I had explained things sufficiently well that people could follow along with what I was doing. Viewer numbers were up too, which was nice. We were on 15 for most of the stream. The packaging work I mostly did on the stream, and then finished up afterwards is here: https://spectrum-os.org/git/nixpkgs/log/?h=2bd191e91f097ff2146099fae8b7c8bd… And a couple of small upstream Nixpkgs PRs: https://github.com/NixOS/nixpkgs/pull/85549 https://github.com/NixOS/nixpkgs/pull/85551 crosvm still needs to be updated in upstream Nixpkgs. If you want to help out, this might be a good thing to pick up! You'd need to copy the update.py change I made in Spectrum's Nixpkgs to the crosvm update.py in upstream Nixpkgs, and then copy the changes in Spectrum nixpkgs' crosvm to upstream Nixpkgs's crosvm. If not, I'll probably get to it in a few days. crosvm ------ Last week, I said that we might me able to use Virtio Wayland as a transport for crosvm device <-> crosvm communication across a VM boundary, but that I'd just had the idea and I might decide it couldn't work. Having researched it extensively this week, reading over the code for the virtio_wl kernel driver with puck and consuming a huge number of messages on assorted kernel-adjacent mailing lists, I think we can make it work, and it's our best path forward, but I'm still not sure how exactly. The main problem is that Virtio Wayland can only send shared memory from host to guest, not the other way around. So when a Wayland compositor in a guest hands the Wayland crosvm device, it's not going to be able to pass it over the VM boundary to the crosvm instance running a Wayland client. Sharing a guest memory buffer with the host, or another guest, turns out not to be a solved problem in general. As an example, here is a very long mailing list thread about how a generic way to do this could work, from November/December, which reached no firm conclusions: https://lore.kernel.org/linux-media/20191105105456.7xbhtistnbp272lj@sirius.… I expect that for most devices, e.g. the network driver, this won't be a problem. But we're definitely going to need this for Wayland. One option would be to modify a Wayland compositor to make it ask the host for some shared memory, rather than allocating its own shared memory buffer. That would work, but it's the last resort, because it would have to be a modification made to any Wayland compositor we might want to use. At a high level, what we want to do is have a crosvm device running inside a device VM tell crosvm that a certain address in guest memory is some shared memory, and have it handle that as it would shared memory it had allocated itself. crosvm does support sending dmabufs from guest to host, because of virtio-gpu. There's a relatively new kernel API called udmabuf that converts a memfd into a dmabuf, which sounds like it should solve exactly this problem. It has no documentation in the kernel, but I did manage to find a small amount of documentation in a patch that was for some reason never applied[1]. I then managed to write a small test program that at least called all the ioctls properly and ended up with a dmabuf. Actually testing this any further will involve a VM, and testing this sort of thing in a VM at the moment is a real pain, and I want to make a small tooling improvement around that before I proceed any further with testing udmabuf. There are also good reasons to suspect that the udmabuf idea won't work anyway. puck thinks that the only dmabufs Virtio Wayland will be able to transfer are those allocated by virtio-gpu. So, not dmabufs that we just made out of a memfd given to us by a Wayland compositor. If udmabuf ends up not working, I'll want to see whether we can use virtio-gpu to allocate dmabufs. The problem with this is that I don't know how we'll bridge compositor shared memory to virtio-gpu dmabufs. It'll presumably involve writing kernel code, which I've been pleased to avoid so far. I can at least probably use the udmabuf implementation as a starting point. The third least bad option would be to modify the virtio_wl kernel driver so it supports shared memory originating from the guest. This would be very nice to use from userspace, but I just don't think there's any way I'll be able to write the required kernel code. And the fourth least bad option (the worst one) would be to modify the compositor as mentioned above. spectrum-vm ----------- The current way of running a Spectrum test VM is to build and run a file called start-vm.nix that lives in the root of a non-default branch of Spectrum's nixpkgs called crosvm-demo. Any time you want to try a custom kernel, rootfs, or crosvm, you need to modify this file. This is extremely annoying. Also, as time goes on, start-vm.nix is getting longer and longer, and more and more difficult to understand. So I think it's finally time to do something better. The something better is that we introduce derivations for a Spectrum rootfs, guest kernel, and crosvm, and then provide a program, spectrum-vm, that by default runs a VM with those three components. It would also have options for overriding those components, so if you built a custom kernel you could just drop that one in instead of the default one, for example. And finally, it would allow changing the flags passed to crosvm. I started working on this this morning, and I'm quite excited about it. It's not going to take very long at all to do, but it's going to be such a big improvement. It will also mean I no longer have a separate crosvm-demo branch, to merge Spectrum nixpkgs master into every time I update it. So I'm going to spend the next couple of days on this, and then get back into investigating udmabuf with a nice new tool. So, that's what happened this week. I feel like I just spent most of the week reading mailing lists and not making much progress, but I've managed to write quite a lot (again) so I suppose I've accomplished something! Now, I've been extremely immersed in this stuff all week, and I suspect that some of this update, especially the crosvm stuff, will not be clear to somebody who has not been similarly immersed. And, it's getting dangerously close to midnight UTC, my self-imposed This Week in Spectrum deadline. So I'm going to send this as is, even though it's probably confusing. But! I do want you to understand what's going on. So please feel free to reply and point out anything I didn't explain very well and ask me for a better explanation. Just be prepared for me not to know the answer, since I'm still trying to figure out a lot of this stuff! As a nice side effect, maybe we'll get a mailing list discussion going. :) [1]: https://lore.kernel.org/lkml/20180911134216.9760-14-kraxel@redhat.com/

1 0

This week in Spectrum
by Alyssa Ross 12 Apr '20

12 Apr '20

Before we begin, **if you are subscribed only to the discuss mailing list**, I'd like to hear from you whether you'd like future weekly development updates to come to you or not, because I'm not really sure whether this sort of update belongs on the devel mailing list, or the discuss one. So for this initial edition, I'm sending to both, but I probably won't do that going forward. So, as I wrote in the recent status update on my blog, I'm going to try weekly(?) mailing updates instead of lengthy blog posts every few months. I was going to start last week, but I ended up taking most of the week off because I'd been working every day for a long time and was starting to find it difficult to concentrate. But I'm back now, and this week has been a good one. Spectrum Live ------------- (This was actually last week, but I felt this deserved a mention.) A bit more than a week ago now, I did a live stream where I showed myself working on a fairly simple change to crosvm. Viewer numbers stabilised at around 11, and we had a lively IRC conversation, which I was pleasantly surprised by. I ended up spending most of the stream not working on that change, but instead explaining how various things in the system will work, and helping getting a couple of people started running the Spectrum code. By the end of the stream two people had succeeded in this, and that means two people who are set up to work on Spectrum. One was even starting work on trying to build a smaller kernel for Spectrum VMs! Overall I think this was extremely successful, and I'll try to do it again soon. I'll announce the stream a bit ahead of time in #spectrum on Freenode. crosvm ------ Having previously split crosvm into two separate programs -- one to run a virtual device, and one to run a device VM -- I further split it into three programs. This might sound a bit silly, but here's the idea -- one program is the crosvm that runs an application VM, the second will become part of the crosvm that runs a device VM, and the third will be the device implementation itself, that will run inside the device VM. As a reminder, the overall idea here is to put virtual devices into their own VMs, rather than being run directly on the host. I also pulled in the latest changes from Google. Of particular note is commit 146450b, which updates the msg_socket library to support dynamically sized messages. If you're paying attention, you'll recall that I just wrote my own serde-based replacement for msg_socket because I needed to do exactly this. That work might now have been superseded, but I'm not sure yet. msg_socket2 uses SOCK_STREAM sockets (which can be used between VMs or VM<->guest), while msg_socket uses SOCK_SEQPACKET, which can't. I haven't checked whether msg_socket's new implementation could work without SOCK_SEQPACKET-style preserved message boundaries -- if it can't, msg_socket2 will stick around. msg_socket2 also properly handles file descriptor lifetimes using the borrow checker. This means that safe Rust code can't send closed or non-existent file descriptors over msg_socket2. msg_socket does not do this, because it just represents file descriptors as integers with no lifetime information. Whatever happens to msg_socket2, at some point I'll want to make msg_socket do file descriptor lifetimes properly. In general, it's not clear from existing documentation how to properly handle file descriptor lifetimes in Rust, and so I plan a blog post on this topic soon. The next problem is to figure out communication between the device VM and the device crosvm. The protocols currently in use rely on being able to share memory using file descriptors, which is not possible between a VM and the host. To avoid using file descriptors, I would have to change those protocols, possibly on a per-device basis, which would be a lot of work, and might have to be done again whenever crosvm adds a new device type. But, I think I might have a better, if surprising solution. Despite its name, the Virtio Wayland protocol has almost nothing to do with Wayland. Instead, it's a fairly generic socket-like protocol that allows sending "virtual file descriptors" of certain specific types between a VM and its host. So I think I should be able to send this sort of communication over "Virtio Wayland", without requiring any changes to the communications being channeled. I just had this idea, so it's entirely possible it won't work out, but I think it's a neat unexpected reuse of a cool technology. Infrastructure -------------- The web interface for the Spectrum mailing lists has been upgraded from Django 1.11 (which is now EOL) to Django 2.2 (current LTS; Nixpkgs default). I sent this upstream as Nixpkgs PR #84896. Additionally, spectrum-os.org. now has an SPF header, which I thought was already there but apparently wasn't. This should help with mailing list deliverability. That's it for this week. Feel free to reply with any comments or insights, either on the content or the format. I'll try to be back with another development update next week.

2 1

Status Update, March 2020
by Alyssa Ross 29 Mar '20

29 Mar '20

I just published this blog post, which is mostly about what I've been doing for Spectrum so far this year: https://alyssa.is/working-on-crosvm/

1 0

GUI for installing compartmentalized apps?
by /#!/JoePea 12 Jan '20

12 Jan '20

Is this something planned for Spectrum down the line? #!/JoePea

2 3

Private GitLab for development?
by /#!/JoePea 12 Jan '20

12 Jan '20

This project is interesting! Has a private instance of GitLab (or similar) been considered, to promote contribution by online visibility and easier collaboration (discussions in issues, pull requests, etc, similar GitHub). I find email to be inefficient. #!/JoePea

2 3

Status Update, November 2019
by Alyssa Ross 03 Nov '19

03 Nov '19

I wrote a lot about current goings-on with Spectrum here: <https://alyssa.is/back-from-nixcon-2019/>

1 0