On 6 Dec 2022, at 13:42, Alyssa Ross <alyssa.ross@unikie.com> wrote:

On Mon, Dec 05, 2022 at 12:42:35AM +0200, Vadim Likholetov wrote:

...

I've made an Firefox appVM for wayland using my waypipe patches. To make this appVM I have had to refactor Spectrum OS appvm infrastructure. The main idea of refactoring is enabling appvm to have a user with normal priviledges, not superuser. Running everything from root is not the best idea for secure OS :-)

In this case, the application running as root is the only thing running in the VM, so it doesn't _really_ matter, but it is still a good idea to fix, as some applications will refuse to run as root.

I don’t agree there — there are many reasons application should not have unnecessary privileges, even assuming the case where someone attacks the application using kernel/processor vulnerabilities to get out of the virtualisation fencing is more fantastic than real :) Also in it is never “the only" application in the VM — we always have those binutils, init stuff and other supplemental software — except the situation we’re running something like bare kernel. So if we have really talking about *secure* OS — drop the <s>gun</s> priviledges ! :-)

(cloud-hypervisor also runs on the /host/ as root, and that's something that we'll definitely want to fix, but that's more complicated as just statically assigning c-h processes to uids won't work when we want to dynamically create and destroy VMs. We'll likely want something akin to systemd's DynamicUser=.)

We can ran C-H as non-root on the host — I don’t see any huge problems there, it’s designed not to be run as root. But the idea to assign an unique UID to each VM is secure enough :-)

...

So now the .nix file for appvm has two sections, one that is executed as root and one as user. Here is the sample of this definitions:

{ config ? import ../../../nix/eval-config.nix {} }:

import ../make-vm.nix { inherit config; } { providers.net = [ "netvm" ];

run = config.pkgs.pkgsStatic.callPackage ( { writeScript }: writeScript "run-root-shell" '' #!/bin/execlineb -P /bin/sh '' ) { };

run-as-user = config.pkgs.pkgsStatic.callPackage ( { writeScript, lynx }: writeScript "run-lynx" '' #!/bin/execlineb -P ${lynx}/bin/lynx https://www.google.com/url?q=https://spectrum-os.org&source=gmail-imap&ust=1... '' ) { };

}

I'm not too sure about this part — it seems like quite a lot of complexity in the app VM implementation, when dropping privileges (unless there's something I haven't considered?) should be as simple as putting "s6-applyuidgid -u 1000 -g 1000" in the VM run script at the point where privileges can be dropped.

I think it’s more flexibility rather than complexity. Also because of immature nature of the Spectrum OS it is good to have some extra tools for developers to debug the stuff and to have extra controls. I can think of refactoring this on the /img/vm level — I’m using su and not s6-tools, if you’re interested in this patchset in general.

...

Cloud-hypervisor has virtual hardware limitations -- it supports only one console device and only one serial device. SpectrumOS is using serial device for kernel logs of appVM and console device as a console. To have access both to root-executed part and to user-executed part of the VM payload, I installed a tmux on console. Now, when you're running vm-console command you get access to the tmux and have the ability to switch between root and user consoles, that can be useful during debugging VM payload.

To run Firefox appVM use vm-start-way command: vm-start-way appvm-firefox :)

Would it work with virtio-gpu? I'm still not convinced on Waypipe — where the previous discussion left off, we were talking about VMs over the network. That would be an interesting thing to look at (and it would be really cool if we could make it work!), but doing it would take a lot more than just network-transparent Wayland proxying, so if that's the main thing we'd get out of Waypipe, I think it would only make sense to add Waypipe support as part of that bigger work. (And this point in time, when how VMs work at all in Spectrum is a bit in flux, is probably not the best time to start trying to massively expand their scope!)

I think it’s the wrong question :-) Both implementations of wayland passthru ( virtio-gpu and waypipe) are ugly right now but they both allow to move the development forward. But someday I believe they’ll become nice. I really believe! :-) The thing is definitely needed in the mainline is abstraction layer for wayland apps — to move all proxying and other system or hardware dependant details out of VM’s descriptions — we define that is an wayland appVM ( like it’s defined it’s networking appVM) and the mechanisms of how the WAYLAND_DISPLAY gets to the is covered by this abstraction layer.

...

i beleive that as soon as spectrumOS features will cover basic user needs it's popularity and community will grow and this will make positive impact on SpectrumOS itself. Using appvm-firefox prototype you may build another wayland-enabled appVMs.

Yeah, having a Firefox example VM would be really great for demonstrating how Spectrum works and what it can do, and I'm pleased that we're even getting to a point where it would work! Firefox is a big application that does a lot of stuff, so it would also be useful for testing all sorts of other features, like audio or XDG desktop portals. I'd be very happy to accept a Firefox VM that used virtio-gpu. :)

On Tue, Dec 06, 2022 at 05:57:19PM +0200, Vadim Likholetov wrote:

I’ve done this — I have patched cloud-hypervisor to have three console devices — serial, console and user-console, but than decided that this patches will never go to C-H mainline so got back with more traditional approach with tmux :)

I'd be pretty optimistic about the chances of a patch that just made it so you could provide multiple consoles the same way you can provide multiple block devices. Like "--console pty file=/path/to/console.out" on the command like to make two consoles, one going to a pty and the other to a file. In Spectrum, the way I'd see such an approach working is that by default there'd be a single console that gave you a shell inside the VM, and if you needed other consoles for other reasons (e.g. if you had an application running on the console, like Lynx) you'd configure another console in the VM configuration. But as we move more towards graphical applications, it will probably become rare to need a secondary console for application interaction like that. To get a non-root shell from the root console though, I think it should be enough to just: # s6-applyuidgid -u 1000 -g 1000 sh (Adding tmux to the VM in development to be able to hop between multiple shell sessions would be a reasonable thing to do, of course.)

...

On 6 Dec 2022, at 17:20, Alyssa Ross <alyssa.ross@unikie.com> wrote:

On Mon, Dec 05, 2022 at 12:42:35AM +0200, Vadim Likholetov wrote:

...

Cloud-hypervisor has virtual hardware limitations -- it supports only one console device and only one serial device. SpectrumOS is using serial device for kernel logs of appVM and console device as a console. To have access both to root-executed part and to user-executed part of the VM payload, I installed a tmux on console. Now, when you're running vm-console command you get access to the tmux and have the ability to switch between root and user consoles, that can be useful during debugging VM payload.

I wonder what it would take to make cloud-hypervisor support multiple virtio-console devices… I suspect it wouldn't be too hard, since it already supports multiples of every other virtio device just fine…

On Tue, Dec 06, 2022 at 10:12:49PM +0200, Vadim Likholetov wrote:

I’ll try to explain — running as user is not just dropping the priviledges, it is about preparing the environment - making home directory, fixing permissions, allocating pty-s in case of interactive sessions, setting environment variables. Many system daemons like dbus, pipewire , etc has their system-level and user-level parts and we should manage this separately — and it is our nearest future if we want some complex user environments to run on Spectrum, and possibly the packages for this daemons will extend this environment transparent to the user.

So it’s s another form of encapsulation like we discussed about making a separate layer for wayland.

Ah, thanks for the explanation. I understand now! You're quite right about running user daemons like PipeWire. I think to manage those, we'd want to run a user-level s6(-rc) instance, so that the services are supervised, and can therefore be restarted if they crash, and dependencies can be taken care of, and logs can be managed however that should work. Let's try adding PipeWire, D-Bus etc. daemons straight to the img/app image? It's intended to be the default image for running applications, so anything that an application is likely to expect to be running should be provided by that image by default. And then later on, when it comes up, we can figure out how to support adding extra services, or whether we should support removing default services from the image, etc. How does that sound?

I’ve published the cloud-hypervisor patches for extra user console device in my repository under -userconsole branch. I’ll test it (or anyone can) for side effects and then we can discuss if it worth to be used in Spectrum or pushed upstream of C-H. Here is the URL — https://github.com/vadika/cloud-hypervisor/tree/userconsole

...

On 6 Dec 2022, at 19:10, Alyssa Ross <alyssa.ross@unikie.com> wrote:

On Tue, Dec 06, 2022 at 05:57:19PM +0200, Vadim Likholetov wrote:

...

I’ve done this — I have patched cloud-hypervisor to have three console devices — serial, console and user-console, but than decided that this patches will never go to C-H mainline so got back with more traditional approach with tmux :)

I'd be pretty optimistic about the chances of a patch that just made it so you could provide multiple consoles the same way you can provide multiple block devices. Like "--console pty file=/path/to/console.out" on the command like to make two consoles, one going to a pty and the other to a file.

In Spectrum, the way I'd see such an approach working is that by default there'd be a single console that gave you a shell inside the VM, and if you needed other consoles for other reasons (e.g. if you had an application running on the console, like Lynx) you'd configure another console in the VM configuration. But as we move more towards graphical applications, it will probably become rare to need a secondary console for application interaction like that.

To get a non-root shell from the root console though, I think it should be enough to just:

# s6-applyuidgid -u 1000 -g 1000 sh

(Adding tmux to the VM in development to be able to hop between multiple shell sessions would be a reasonable thing to do, of course.)

...

...

On 6 Dec 2022, at 17:20, Alyssa Ross <alyssa.ross@unikie.com> wrote:

On Mon, Dec 05, 2022 at 12:42:35AM +0200, Vadim Likholetov wrote:

...

Cloud-hypervisor has virtual hardware limitations -- it supports only one console device and only one serial device. SpectrumOS is using serial device for kernel logs of appVM and console device as a console. To have access both to root-executed part and to user-executed part of the VM payload, I installed a tmux on console. Now, when you're running vm-console command you get access to the tmux and have the ability to switch between root and user consoles, that can be useful during debugging VM payload.

I wonder what it would take to make cloud-hypervisor support multiple virtio-console devices… I suspect it wouldn't be too hard, since it already supports multiples of every other virtio device just fine…

On 6 Dec 2022, at 19:10, Alyssa Ross <alyssa.ross@unikie.com> wrote:

(Adding tmux to the VM in development to be able to hop between multiple shell sessions would be a reasonable thing to do, of course.)

It may sound like religious point of view, but I strongly believe that developers is the number one priority to support with Spectrum OS now — as easier and friendlier the OS will be to the developer as faster it will be delivered to the common users. It, from my point of view, includes two main aspects — ease of development giving the developer proper “bricks” (well encapsulated OS services) and examples, that coveres main use cases/use patterns of the operating system.

...

It may sound like religious point of view, but I strongly believe that developers is the number one priority to support with Spectrum OS now — as easier and friendlier the OS will be to the developer as faster it will be delivered to the common users. It, from my point of view, includes two main aspects — ease of development giving the developer proper “bricks” (well encapsulated OS services) and examples, that coveres main use cases/use patterns of the operating system.

Yes, that's a compelling argument. I wonder if it would make sense to add some special configuration option for development, that would enable extra development tools and other goodies? (I often find myself adding strace…)

(I think they probably shouldn't be present by default, because then it could be difficult to be sure nothing was depending on them in future when it came time to remove them. But something that any given developer could set once and then always have developer stuff available, and toggle off when they wanted to test a production build, could make things a lot nicer.)

Well, one could have VM generator functions, so that you can easily generate a development-like VM or a reasonable-setup like VM or minimal VM (without D-Bus etc.), and so the developers can just use the dev version and always get the good stuff included?

And BTW, thanks so much for spending all this time engaging with this stuff! The project will be much better off for your efforts. :)

...

Well, one could have VM generator functions, so that you can easily generate a development-like VM or a reasonable-setup like VM or minimal VM (without D-Bus etc.), and so the developers can just use the dev version and always get the good stuff included?

Yes, that's true.

Just as a quick refresher, currently if you want to make a Spectrum VM, these are your options:

- Lay out the VM configuration directory manually in the user data partition, and bring your own kernel/rootfs.

- As above, but use the rootfs and kernel in /usr/img/appvm. These aim to provide a "standard" environment that can be shared between application VMs, so it comes with mesa, and in future virtio-gpu set up, as well as D-Bus, PipeWire, etc. It mounts a second block device that contains your application, as well as any dependencies not included in the base image, and then runs your application from there. This way, big stuff like mesa only has to exist once on disk.

- Create an application VM that uses /usr/img/appvm with Nix, using vm-lib/make-vm.nix.

Yes, sorry, I have still not tried all this (I am going to avoid Wayland as the main UI as long as practical, and I am not going to commit enough time to Spectrum-adjacent things to make it worth your time to ask how would I set up things for Spectrum-like isolation of single things into Weston-inside-Xorg or whatever on host).

So what we're missing is a way to create, with Nix, a VM that isn't based on the /usr/img/appvm "template". The reason we don't currently have such a function is that I don't think it's clear yet how it should work — what should be customisable, what should be part of the base image vs. not, etc.? For a while, how make-vm.nix should work was similarly unclear, which is why it only recently came into existence. After I'd manually created my third or so application VM, it became clear what could be part of the base image, and what had to be customisable. Currently, we only have one VM in the tree that doesn't use the /usr/img/appvm template, netvm. Once we have more, I think it'll be easier to evaluate how a more minimal VM builder function should work.

Hm, I would say that I have a good experience solving such a problem with a Nixpkgs-overlay-like structure, i.e. a makeExtensible with all possible tunables defined as attributes, and then more complex data structures built out of them, as well as packages for the things.

On 12/6/22 19:10, Alyssa Ross wrote:

On Tue, Dec 06, 2022 at 05:57:19PM +0200, Vadim Likholetov wrote:

...

I’ve done this — I have patched cloud-hypervisor to have three console devices — serial, console and user-console, but than decided that this patches will never go to C-H mainline so got back with more traditional approach with tmux :)

I'd be pretty optimistic about the chances of a patch that just made it so you could provide multiple consoles the same way you can provide multiple block devices. Like "--console pty file=/path/to/console.out" on the command like to make two consoles, one going to a pty and the other to a file.

Vadim, please go ahead and create a PR to C-H upstream. There's good chance your work would get accepted. Let's see their comments and and feedback. -Ville