- Spectrum Development - spectrum-os.org

[PR nixpkgs] crosvm: 79.12607.0.0-rc4 -> 81.12871.0.0-rc1
by Alyssa Ross 21 Apr '20

21 Apr '20

Extracted crosvm update from recent chromiumOSPackages update in Spectrum for upstreaming. https://github.com/NixOS/nixpkgs/pull/85731

1 0

This Week in Spectrum
by Alyssa Ross 19 Apr '20

19 Apr '20

Wow, a week isn't very long at all. But still, there's definitely enough to write about, even if I spent most of the week just reading mailing list threads. More on that in a bit. Spectrum Live + nixpkgs ----------------------- I did another stream. This time, we looked at updating the Chromium OS packages in Spectrum. The script I used to update them had broken -- it was stuck on an old version, so first we had to fix the script, and then fix the package builds. A lot of the Chromium OS packages aren't actually necessary at the moment, but I made them a while ago when it looked like they would be. I couldn't get one particular such package, libchrome, to build any more at all after spending quite a few hours on it, so I ended up just dropping it, along with libbrillo, which depended on it. Some other package updates had some complexity, but we managed to work through them all on the stream. This stream was much more focused on what I was working on than the last one. The last one was more about people asking general questions and me answering them and demonstrating things. This time I stayed on the package updates for the whole stream. I figure people asked all their questions last time. But there were lots of questions about specific issues in the package updates, etc. It was nice to know that I had explained things sufficiently well that people could follow along with what I was doing. Viewer numbers were up too, which was nice. We were on 15 for most of the stream. The packaging work I mostly did on the stream, and then finished up afterwards is here: https://spectrum-os.org/git/nixpkgs/log/?h=2bd191e91f097ff2146099fae8b7c8bd… And a couple of small upstream Nixpkgs PRs: https://github.com/NixOS/nixpkgs/pull/85549 https://github.com/NixOS/nixpkgs/pull/85551 crosvm still needs to be updated in upstream Nixpkgs. If you want to help out, this might be a good thing to pick up! You'd need to copy the update.py change I made in Spectrum's Nixpkgs to the crosvm update.py in upstream Nixpkgs, and then copy the changes in Spectrum nixpkgs' crosvm to upstream Nixpkgs's crosvm. If not, I'll probably get to it in a few days. crosvm ------ Last week, I said that we might me able to use Virtio Wayland as a transport for crosvm device <-> crosvm communication across a VM boundary, but that I'd just had the idea and I might decide it couldn't work. Having researched it extensively this week, reading over the code for the virtio_wl kernel driver with puck and consuming a huge number of messages on assorted kernel-adjacent mailing lists, I think we can make it work, and it's our best path forward, but I'm still not sure how exactly. The main problem is that Virtio Wayland can only send shared memory from host to guest, not the other way around. So when a Wayland compositor in a guest hands the Wayland crosvm device, it's not going to be able to pass it over the VM boundary to the crosvm instance running a Wayland client. Sharing a guest memory buffer with the host, or another guest, turns out not to be a solved problem in general. As an example, here is a very long mailing list thread about how a generic way to do this could work, from November/December, which reached no firm conclusions: https://lore.kernel.org/linux-media/20191105105456.7xbhtistnbp272lj@sirius.… I expect that for most devices, e.g. the network driver, this won't be a problem. But we're definitely going to need this for Wayland. One option would be to modify a Wayland compositor to make it ask the host for some shared memory, rather than allocating its own shared memory buffer. That would work, but it's the last resort, because it would have to be a modification made to any Wayland compositor we might want to use. At a high level, what we want to do is have a crosvm device running inside a device VM tell crosvm that a certain address in guest memory is some shared memory, and have it handle that as it would shared memory it had allocated itself. crosvm does support sending dmabufs from guest to host, because of virtio-gpu. There's a relatively new kernel API called udmabuf that converts a memfd into a dmabuf, which sounds like it should solve exactly this problem. It has no documentation in the kernel, but I did manage to find a small amount of documentation in a patch that was for some reason never applied[1]. I then managed to write a small test program that at least called all the ioctls properly and ended up with a dmabuf. Actually testing this any further will involve a VM, and testing this sort of thing in a VM at the moment is a real pain, and I want to make a small tooling improvement around that before I proceed any further with testing udmabuf. There are also good reasons to suspect that the udmabuf idea won't work anyway. puck thinks that the only dmabufs Virtio Wayland will be able to transfer are those allocated by virtio-gpu. So, not dmabufs that we just made out of a memfd given to us by a Wayland compositor. If udmabuf ends up not working, I'll want to see whether we can use virtio-gpu to allocate dmabufs. The problem with this is that I don't know how we'll bridge compositor shared memory to virtio-gpu dmabufs. It'll presumably involve writing kernel code, which I've been pleased to avoid so far. I can at least probably use the udmabuf implementation as a starting point. The third least bad option would be to modify the virtio_wl kernel driver so it supports shared memory originating from the guest. This would be very nice to use from userspace, but I just don't think there's any way I'll be able to write the required kernel code. And the fourth least bad option (the worst one) would be to modify the compositor as mentioned above. spectrum-vm ----------- The current way of running a Spectrum test VM is to build and run a file called start-vm.nix that lives in the root of a non-default branch of Spectrum's nixpkgs called crosvm-demo. Any time you want to try a custom kernel, rootfs, or crosvm, you need to modify this file. This is extremely annoying. Also, as time goes on, start-vm.nix is getting longer and longer, and more and more difficult to understand. So I think it's finally time to do something better. The something better is that we introduce derivations for a Spectrum rootfs, guest kernel, and crosvm, and then provide a program, spectrum-vm, that by default runs a VM with those three components. It would also have options for overriding those components, so if you built a custom kernel you could just drop that one in instead of the default one, for example. And finally, it would allow changing the flags passed to crosvm. I started working on this this morning, and I'm quite excited about it. It's not going to take very long at all to do, but it's going to be such a big improvement. It will also mean I no longer have a separate crosvm-demo branch, to merge Spectrum nixpkgs master into every time I update it. So I'm going to spend the next couple of days on this, and then get back into investigating udmabuf with a nice new tool. So, that's what happened this week. I feel like I just spent most of the week reading mailing lists and not making much progress, but I've managed to write quite a lot (again) so I suppose I've accomplished something! Now, I've been extremely immersed in this stuff all week, and I suspect that some of this update, especially the crosvm stuff, will not be clear to somebody who has not been similarly immersed. And, it's getting dangerously close to midnight UTC, my self-imposed This Week in Spectrum deadline. So I'm going to send this as is, even though it's probably confusing. But! I do want you to understand what's going on. So please feel free to reply and point out anything I didn't explain very well and ask me for a better explanation. Just be prepared for me not to know the answer, since I'm still trying to figure out a lot of this stuff! As a nice side effect, maybe we'll get a mailing list discussion going. :) [1]: https://lore.kernel.org/lkml/20180911134216.9760-14-kraxel@redhat.com/

1 0

This week in Spectrum
by Alyssa Ross 12 Apr '20

12 Apr '20

Before we begin, **if you are subscribed only to the discuss mailing list**, I'd like to hear from you whether you'd like future weekly development updates to come to you or not, because I'm not really sure whether this sort of update belongs on the devel mailing list, or the discuss one. So for this initial edition, I'm sending to both, but I probably won't do that going forward. So, as I wrote in the recent status update on my blog, I'm going to try weekly(?) mailing updates instead of lengthy blog posts every few months. I was going to start last week, but I ended up taking most of the week off because I'd been working every day for a long time and was starting to find it difficult to concentrate. But I'm back now, and this week has been a good one. Spectrum Live ------------- (This was actually last week, but I felt this deserved a mention.) A bit more than a week ago now, I did a live stream where I showed myself working on a fairly simple change to crosvm. Viewer numbers stabilised at around 11, and we had a lively IRC conversation, which I was pleasantly surprised by. I ended up spending most of the stream not working on that change, but instead explaining how various things in the system will work, and helping getting a couple of people started running the Spectrum code. By the end of the stream two people had succeeded in this, and that means two people who are set up to work on Spectrum. One was even starting work on trying to build a smaller kernel for Spectrum VMs! Overall I think this was extremely successful, and I'll try to do it again soon. I'll announce the stream a bit ahead of time in #spectrum on Freenode. crosvm ------ Having previously split crosvm into two separate programs -- one to run a virtual device, and one to run a device VM -- I further split it into three programs. This might sound a bit silly, but here's the idea -- one program is the crosvm that runs an application VM, the second will become part of the crosvm that runs a device VM, and the third will be the device implementation itself, that will run inside the device VM. As a reminder, the overall idea here is to put virtual devices into their own VMs, rather than being run directly on the host. I also pulled in the latest changes from Google. Of particular note is commit 146450b, which updates the msg_socket library to support dynamically sized messages. If you're paying attention, you'll recall that I just wrote my own serde-based replacement for msg_socket because I needed to do exactly this. That work might now have been superseded, but I'm not sure yet. msg_socket2 uses SOCK_STREAM sockets (which can be used between VMs or VM<->guest), while msg_socket uses SOCK_SEQPACKET, which can't. I haven't checked whether msg_socket's new implementation could work without SOCK_SEQPACKET-style preserved message boundaries -- if it can't, msg_socket2 will stick around. msg_socket2 also properly handles file descriptor lifetimes using the borrow checker. This means that safe Rust code can't send closed or non-existent file descriptors over msg_socket2. msg_socket does not do this, because it just represents file descriptors as integers with no lifetime information. Whatever happens to msg_socket2, at some point I'll want to make msg_socket do file descriptor lifetimes properly. In general, it's not clear from existing documentation how to properly handle file descriptor lifetimes in Rust, and so I plan a blog post on this topic soon. The next problem is to figure out communication between the device VM and the device crosvm. The protocols currently in use rely on being able to share memory using file descriptors, which is not possible between a VM and the host. To avoid using file descriptors, I would have to change those protocols, possibly on a per-device basis, which would be a lot of work, and might have to be done again whenever crosvm adds a new device type. But, I think I might have a better, if surprising solution. Despite its name, the Virtio Wayland protocol has almost nothing to do with Wayland. Instead, it's a fairly generic socket-like protocol that allows sending "virtual file descriptors" of certain specific types between a VM and its host. So I think I should be able to send this sort of communication over "Virtio Wayland", without requiring any changes to the communications being channeled. I just had this idea, so it's entirely possible it won't work out, but I think it's a neat unexpected reuse of a cool technology. Infrastructure -------------- The web interface for the Spectrum mailing lists has been upgraded from Django 1.11 (which is now EOL) to Django 2.2 (current LTS; Nixpkgs default). I sent this upstream as Nixpkgs PR #84896. Additionally, spectrum-os.org. now has an SPF header, which I thought was already there but apparently wasn't. This should help with mailing list deliverability. That's it for this week. Feel free to reply with any comments or insights, either on the content or the format. I'll try to be back with another development update next week.

2 1

[PATCH] start-vm: Remove rngd (unecessary since the introduction of khwrngd)
by Nicolas Braud-Santoni 29 Dec '19

29 Dec '19

khwrngd provides the same functionality as rngd here, but is integrated in the kernel, and started by default by the hwrng driver as soon as the device is initialized. It was added in Linux 3.17 by the following commit: commit be4000bc4644d027c519b6361f5ae3bbfc52c347 Author: Torsten Duwe <duwe(a)lst.de> Date: Sat Jun 14 23:46:03 2014 -0400 hwrng: create filler thread This can be viewed as the in-kernel equivalent of hwrngd; like FUSE it is a good thing to have a mechanism in user land, but for some reasons (simplicity, secrecy, integrity, speed) it may be better to have it in kernel space. This patch creates a thread once a hwrng registers, and uses the previously established add_hwgenerator_randomness() to feed its data to the input pool as long as needed. A derating factor is used to bias the entropy estimation and to disable this mechanism entirely when set to zero. --- start-vm.nix | 8 -------- 1 file changed, 8 deletions(-) diff --git a/start-vm.nix b/start-vm.nix index 442f83a8f45..66fb358cb39 100644 --- a/start-vm.nix +++ b/start-vm.nix @@ -45,14 +45,6 @@ let s6-linux-init-hpr -fr ''; } // services.".s6-svscan" or {}; - - rngd = { - run = writeScript "rngd-run" '' - #! ${execline}/bin/execlineb -P - ${rng-tools}/bin/rngd -f -x pkcs11,rdrand - ''; - } // services.rngd or {}; - } // services; in -- 2.24.0

2 1

[PATCH] start-vm.nix: implement 3-stage init
by Alyssa Ross 09 Dec '19

09 Dec '19

This uses s6-svscan as stage 2. Right now all that means is reaping zombies -- there are no services for it to supervise. The application being run in the VM (sommelier hello-wayland in this case) is just run as a direct child of init. I felt it didn't belong as a service because it shouldn't be restarted when it terminates. I don't use s6-linux-init for a couple of reasons: - A lot of the reason to use s6-linux-init comes from wanting to do clever things to seamlessly redirect logs from the console to the file system at some point during stage1. For VMs, I think we're better just logging to the console so logs can be managed in one place on the host. - s6-linux-init is a pain with Nix, because s6-linux-init-maker needs to be run as root, and it generates stuff like pipes that can't be stored in the Nix store. coreutils is only used for cp-ing stage3 to the right place. When services are implemented, this can be removed and the VMs will have one less dependency. To shutdown, we use reboot instead of poweroff because that's what actually gets crosvm to quit. reboot(RB_POWER_OFF) seems to be interpreted by the kernel as halt for some reason I have failed to determine. It prints the halt message and everything. reboot(RB_HALT_SYSTEM) does exactly the same thing. Maybe it's just something to do with running in a VM. --- I've pushed this to crosvm-demo already, but I'm posting it here for feedback and suggestions, especially from anybody with experience with inits or s6. start-vm.nix | 54 +++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 43 insertions(+), 11 deletions(-) diff --git a/start-vm.nix b/start-vm.nix index b8d20007521..1345a5620c6 100644 --- a/start-vm.nix +++ b/start-vm.nix @@ -14,25 +14,57 @@ let DEVTMPFS_MOUNT = yes; }; }; - init = writeShellScript "init" '' - set -xe - ${coreutils}/bin/mkdir /dev/pts /dev/shm - ${utillinux}/bin/mount -t tmpfs none /tmp - ${utillinux}/bin/mount -t devpts -o gid=4,mode=620 none /dev/pts - ${utillinux}/bin/mount -t tmpfs none /dev/shm - export XDG_RUNTIME_DIR=/tmp - ${sommelier}/bin/sommelier ${hello-wayland}/bin/hello-wayland + stage1 = writeScript "stage1" '' + #! ${execline}/bin/execlineb -P + importas -i PATH PATH + export PATH ${lib.makeBinPath + [ s6-linux-init s6-portable-utils s6-linux-utils s6 execline coreutils ]} + ${s6}/bin/s6-setsid -qb -- + umask 022 + if { s6-mount -t tmpfs -o mode=0755 tmpfs /run } + if { s6-mkdir -p /run/service/.s6-svscan } + if { cp ${stage3} /run/service/.s6-svscan/finish } + emptyenv -p + + background { + s6-setsid -- + if { s6-mkdir -p /run/user/0 /dev/pts /dev/shm } + if { s6-mount -t devpts -o gid=4,mode=620 none /dev/pts } + if { s6-mount -t tmpfs none /dev/shm } + if { s6-mount -t proc none /proc } + export XDG_RUNTIME_DIR /run/user/0 + foreground { ${sommelier}/bin/sommelier ${hello-wayland}/bin/hello-wayland } + importas -i ? ? + if { echo STATUS: $? } + s6-svscanctl -6 /run/service + } + + unexport ! + cd /run/service + s6-svscan + ''; + + stage3 = writeScript "stage3" '' + #! ${execline}/bin/execlineb -S0 + foreground { s6-nuke -th } + s6-sleep -m -- 2000 + foreground { s6-nuke -k } + wait { } + s6-linux-init-hpr -fr ''; rootfs = runCommand "rootfs" {} '' - mkdir dev bin tmp + mkdir bin dev proc run tmp ln -s ${dash}/bin/sh bin - (cat ${writeReferencesToFile init}; printf "%s\n" bin dev tmp) | xargs tar -cP --hard-dereference | ${squashfs-tools-ng}/bin/tar2sqfs $out + + (cat ${writeReferencesToFile stage1}; printf "%s\n" bin dev proc run tmp) \ + | xargs tar -cP --owner root:0 --group root:0 --hard-dereference \ + | ${squashfs-tools-ng}/bin/tar2sqfs $out ''; in writeShellScript "crosvm" '' set -x - exec ${crosvm}/bin/crosvm run --wayland-sock=$XDG_RUNTIME_DIR/$WAYLAND_DISPLAY -p init=${init} --root=${rootfs} ${kernel}/bzImage + exec ${crosvm}/bin/crosvm run --wayland-sock=$XDG_RUNTIME_DIR/$WAYLAND_DISPLAY -p init=${stage1} --root=${rootfs} ${kernel}/bzImage '' -- 2.23.0

1 0

[spectrum-devel] Proposed design sketch
by Michael Raskin 21 Oct '19

21 Oct '19

Hello So I tried to look over what happens in my system, and then apply some hardening to it. The following is what I got. ///// Design sketch of a SpectrumOS prototype User starts an application with some choice of VM base image/capabilities, store paths available inside the VM, the store paths to add to environment, state directories readable/writeable in VM, external capabilities connected to the VM, internal devices visible to the applications. Single-use state directory may be designated to provide a VM with fully-owned secondary-storage-backed tmp-like space. Layout of the state directories visible to the VM can be adjusted. As a special case support there is support for generating VM-specific configuration files such as /etc/passwd (these should not be prebuilt due to per-launch UID generation). A unique UID is assigned to the VM. This UID is added to the ACLs so it can read the state directories assigned and all their subdirectories (files are world-readable by default). It also gets access to write to the writeable directories and possibly to some of the files in these directories UIDs/ACLs are tracked. The user can assign a UID to an alias for further reuse (this reduces the need for updating ACLs all the time). One-time UIDs are reused as rarely as possible. When a VM with a single-use UID quits, its UID is removed from the ACLs of the granted files. There is also a periodic enumeration of files in order to clean up long-obsolete ACLs in came of unclean shutdown or similar events. A temporary directory is created. Everything that should be visible inside the VM is bind-mounted there. This includes devices. An isolated environment is set up for the VM to run. VM is started serving the state directories and store paths to the inside via virtfs/virtio-fs. Inside VM the extra device node are removed from /dev/. Requested store paths are added to PATH and possibly to other variables like LD_LIBRARY_PATH or QT_PLUGIN_PATH as requested. Read-only state directories are bind-mounted over themselves read-only. The specified program is started normally under numerically the same UID as assigned to the VM (to avoid the need to use UID mapping). There are some predefined wrappers to launch a program requiring, for example, a D-Bus session in a VM-local D-Bus session. A special type of wrappers is restricted-environment setup, to reduce the ability of an program to attack the surrounding VM. (VM might eventually run a different OS kernel type than the host) Special kinds of resources: Expected to be handled by the VM: DRI passthrough. Sound passthrough (wrapper to launch full isolated PulseAudio). GUI is handled via virtio-wayland. Local sockets (such as ssh-agent): hopefully can be handled by the file sharing solution. Network: VM connected by socket to a virtual switch leading to a firewall VM, VM itself runs in no-network environment. Cross-VM network links do not correspond to any visible files inside the VM-surrounding isolated environment. A firewall VM per ruleset (can be shared for multiple Firefox instances with the same security profile). Firewall VMs connect via a virtual switch to the network-handling VM; it may have addressless filtering bridge VMs on both sides. External storage: considered a special case of state storage; might not support ACLs (so access control is purely by ro/rw bind mounts outside and inside the VM). Nontrivial external hardware: USB/PCI pass through?

1 0

[spectrum-devel] [PATCH www] design: state subdirectories, not block devices
by Alyssa Ross 18 Oct '19

18 Oct '19

Conversation on IRC has convinced me that this is the right thing to do after all: <https://logs.spectrum-os.org/spectrum/2019-10-15#1571117332-1571144389;> --- Jean-Phillipe, I'd be curious to hear your thoughts on the above discussion, since you recommended block devices to me when we talked. design.html | 29 ++++++++++++++--------------- 1 file changed, 14 insertions(+), 15 deletions(-) diff --git a/design.html b/design.html index 4b96a41..dc14cfe 100644 --- a/design.html +++ b/design.html @@ -43,23 +43,22 @@ one per application. Each virtual machine will be generated by a <a href="https://nixos.org/nix/">Nix</a> derivation, and will have a completely immutable root file system. Persistent storage will be -provided by virtual block devices, that arbitrary paths on the system -can be mapped to from the host. There may be other writable mount -points inside the virtual machine, but these will not persist between -reboots of the VM. Using Nix to generate virtual machines allows them -to be reproducibly built, rolled back, edited, and migrated as source -code, rather than large, opaque virtual machine images. +provided by mounting subdirectories of the global state directory into +virtual machines. There may be other writable mount points inside the +virtual machine, but these will not persist between reboots of the VM. +Using Nix to generate virtual machines allows them to be reproducibly +built, rolled back, edited, and migrated as source code, rather than +large, opaque virtual machine images. <p> -Virtual block devices will also be defined in Nix, and block devices -and applications will be <var>m</var>:<var>n</var>. Some virtual -machines may have no persistent storage, or even write access to a -disk, at all. In other cases, it might be desirable for multiple -applications to be able to access the same device, such as a local -mail store being shared by two mail clients. Other resources and -permissions, such as network cards and USB controllers, will similarly -be defined in Nix. There are three logical sections for the Nix -configuration -- applications, which are just packages, resources +State directories and applications will be <var>m</var>:<var>n</var>. +Some virtual machines may have no persistent storage, or even write +access to a disk, at all. In other cases, it might be desirable for +multiple applications to be able to access the same device, such as a +local mail store being shared by two mail clients. Other resources +and permissions, such as network cards and USB controllers, will +similarly be defined in Nix. There are three logical sections for the +Nix configuration -- applications, which are just packages, resources (virtual or physical devices), and <i>application instances</i>, which are mappings between applications and accessible resources. This structure allows users to have multiple instances of the same -- 2.23.0

2 7

[spectrum-devel] [PATCH] nlnet: software bill of materials
by Alyssa Ross 17 Oct '19

17 Oct '19

This changes the URI for the application too, but that was never published anywhere so I think it's fine. --- Anything I've missed? .../application.html | 0 nlnet-pet-2019-03/bom.txt | 8 ++++++++ 2 files changed, 8 insertions(+) rename nlnet-pet-2019-03.html => nlnet-pet-2019-03/application.html (100%) create mode 100644 nlnet-pet-2019-03/bom.txt diff --git a/nlnet-pet-2019-03.html b/nlnet-pet-2019-03/application.html similarity index 100% rename from nlnet-pet-2019-03.html rename to nlnet-pet-2019-03/application.html diff --git a/nlnet-pet-2019-03/bom.txt b/nlnet-pet-2019-03/bom.txt new file mode 100644 index 0000000..5ba36a2 --- /dev/null +++ b/nlnet-pet-2019-03/bom.txt @@ -0,0 +1,8 @@ +Spectrum -- Software Bill of Materials +====================================== + +- Linux (notably KVM): <https://kernel.org/> +- Nix: <https://nixos.org/nix/> +- Nixpkgs: <https://nixos.org/nixpkgs/> +- crosvm: <https://chromium.googlesource.com/chromiumos/platform/crosvm/> +- A Wayland compositer (undecided which one) -- 2.23.0

1 1