On Wed, 6 Jan 2021 at 07:01, Alyssa Ross <hi@alyssa.is> wrote:
Hi! Thanks for getting in touch. :)
To be honest, I'm surprised you got as far as you did -- like Michael says, I'm currently working towards a proof of concept, so none of what you've tried out so far is really meant for use outside of that proof of concept.
[...]
nix-build . -A spectrumPackages && ./result-3/bin/spectrum-vm
(I'm fairly new to Nix, so not sure if this is the right way to do things)
Pretty close -- spectrumPackages is an attribute set containing lots of derivations, which is why you end up with lots of numbered result-* symlinks. If you do -A spectrumPackages.spectrum-vm it'll just give you a single result symlink pointing to that, and you won't need to go hunting for the right one. :)
That's better, thanks! [...]
- I tried adding `--shared-dir /tmp/ff:ff:type=9p` to share a host directory. Then `mount -t 9p -o trans=virtio,version=9p2000.L ff /tmp` in the VM seemed to work, but `ls /tmp` crashed the VM.
Yeah, this is a known issue. I have a patch[1] for it but didn't add it to the package since I mostly have been working with my own source builds of crosvm.
[1]: https://spectrum-os.org/git/crosvm/commit/?id=1e318da5b57c12f67bed3b528100db...
Ah, I didn't realise it was using seccomp too. I'm not sure how to compile specific versions of crosvm. I tried with: srcs = lib.genAttrs [ "src/third_party/adhd" "src/aosp/external/minijail" ] getSrc // { "src/platform/crosvm" = /home/.../crosvm; }; and blanked out the hash as it requested, but then: error: failed to sync Caused by: failed to load pkg lockfile Caused by: failed to resolve patches for `https://github.com/rust-lang/crates.io-index` Caused by: failed to load source for dependency `libvda` Caused by: Unable to update /build/src/platform2/arc/vm/libvda/rust Caused by: failed to read `/build/src/platform2/arc/vm/libvda/rust/Cargo.toml` Looks like this happens since 57df6a0ab23c3b2ba233b9aa5886ecf47ba3f91f (added a dependency?). Commit 460406d10bbfaa890d56d616b4610813da63a312 just before that gets further, but: error: the lock file /build/src/platform/crosvm/Cargo.lock needs to be updated but --frozen was passed to prevent this How do you build it? (sorry for these basic Nix/Rust questions) However, I could get 9p to work by running the previous version with --seccomp-log-failures. With that, I can read and write files from the console, but I can't chown things and so can't write from the terminal window, which is running as a user. I guess it needs uidmap set, but I'm not sure how to make that work.
- I tried using `-d /dev/mapper/disk` to share an LVM partition, but `mount -t ext4 /dev/vdb /tmp` refused to mount it.
Never tried that, so I don't know anything about it I'm afraid.
OK, I'll keep trying stuff. I have discovered that if I add the squashfs file as another device (--root "$rootfs" -d "$rootfs") then it shows an error but mounts it anyway! # ls /tmp # mount /dev/vdb /tmp [ 15.288873] /dev/vdb: Can't open blockdev # ls /tmp bin dev etc nix proc run sbin sys tmp Sadly, this didn't work with my ext4 partition.
- I tried enabling networking with `--host_ip 10.0.0.1`, etc, but it said it couldn't create a tap device. I guess it needs more privileges.
Yeah, crosvm needs to be CAP_NET_ADMIN for that (which is difficult to do with Nix). You can make a TAP device yourself iproute2 and use --tap-fd to tell crosvm to use it, or you can use the mktuntap program I wrote (with a privelege drop after running mktuntap), like this:
sudo mktuntap -pvB 3 \ sudo -u $USER -C 4 result/bin/spectrum-vm -- --tap-fd 3
OK, I tried like this: exec sudo "$mktuntap" -pvB 3 \ sudo -u "$USER" -C 4 \ "$crosvm" run \ -p init=/sbin/init \ -p "spectrumcmd=$(printf %s "$command" | base64 -w0)" \ --tap-fd 3 \ --seccomp-log-failures \ --root "$rootfs" \ --host_ip 10.0.0.1 \ --netmask 255.0.0.0 \ --mac c0:ff:ee:c0:ff:ee \ -m 4096 \ "$@" \ "$kernel" I got "sudo: you are not permitted to use the -C option", which I fixed by editing the sudoers file. Then it fails with: [ERROR:src/main.rs:1351] The architecture failed to build the vm: error creating devices: failed to set up virtio networking: failed to open tap device: failed to create tap interface: Operation not permitted (os error 1) Strace shows: openat(AT_FDCWD, "/dev/net/tun", O_RDWR|O_NONBLOCK|O_CLOEXEC) = 31 ioctl(31, TUNSETIFF, 0x7ffee7ede238) = -1 EPERM (Operation not permitted) Maybe it's just because my crosvm is too old?
Ideally, I'd like to run a VM with each of my old Qubes filesystems, to get back to where I was with my Qubes setup, before investigating new spectrum stuff (e.g. one app per VM). Do you have any advice on this? I see these lists are a bit quiet - I hope someone is still working on this because it sounds great :-)
Like Michael said, there's a lot I need to do before it's really ready to use like this, but I am working on it (or at least I will be again once my anti-burnout break ends).
Great! I'm on a break myself at the moment, which is why I have some time to try all this out.
Once I am, I hope to be more active on the lists again. I used to post weekly status updates, and would like to get into doing that again once I'm back because they were a great way to keep people up to date with the project and for me to have a record of what I'd been doing. Reading some of the old status updates should give you a bit of a feel for where things are, although things are a bit further along than they were when I wrote the last one because I put the status updates on hold to try to chase a funding milestone.
I've read some of them - they're very helpful!
Hope that's all clear -- please ask more questions if you have them, although if it's anything particularly in the weeds I might wait until I'm back from my break to answer. :)
I have many questions :-) But don't feel pressured to answer them; I need to figure out how to make this all work myself anyway, and it's just a bonus if you've already done the work for me... -- talex5 (GitHub/Twitter) http://roscidus.com/blog/ GPG: 5DD5 8D70 899C 454A 966D 6A51 7513 3C8F 94F6 E0CC