- Spectrum Development - spectrum-os.org

[PATCH www] participating.html: Freenode -> Libera
by Alyssa Ross 30 May '21

30 May '21

The Matrix bridge is "in beta", which I think is good enough. --- participating.html | 44 ++++++++------------------------------------ 1 file changed, 8 insertions(+), 36 deletions(-) diff --git a/participating.html b/participating.html index 0e8ba65..7b47e8a 100644 --- a/participating.html +++ b/participating.html @@ -68,42 +68,14 @@ development</a>. <p> There is a single IRC -channel, <a href="ircs://chat.freenode.net:6997/spectrum">#spectrum</a> -on Freenode, for all kinds of Spectrum discussions. If you're new to -IRC, you might find -Freenode's <a href="https://webchat.freenode.net/#spectrum">web -chat</a> the easiest way to get started. The channel -is <strong>logged</strong>, and the logs -are <a href="https://logs.spectrum-os.org/spectrum/">published</a> on -the web. - -<p> -<strong>We will be moving -to <a href="https://libera.chat/">Libera.Chat</a> as soon as -a <a href="https://github.com/matrix-org/matrix-appservice-irc/issues/1324">Matrix -bridge</a> is available.</strong> - -<h3>Note for Matrix users</h3> - -<p> -There appears to be an issue with Matrix's IRC bridged room that prevents -Matrix users from joining the channel with the normal alias -#freenode_#spectrum:matrix.org. There are the following workarounds: - -<ul> - <li>Join the room via it's internal name: - <kbd>/join !ZsIiqZmCTSWHpTQWaw:matrix.org</kbd>. - <li>Join the room via - <a href="https://matrix.to/#/!ZsIiqZmCTSWHpTQWaw:matrix.org">matrix.to link</a>. - <li>Ask the bridges appservice to invite you to the room: - <kbd>/msg @appservice-irc:matrix.org !join #spectrum</kbd> and accept the invite. -</ul> - -<p> -If you experience this issue, kindly report it to the people who run -the Matrix IRC bridge. If you are able to join the channel without -the workaround, please report that in #spectrum so the website can be -updated. +channel, <a href="ircs://irc.libera.chat:6697/spectrum">#spectrum</a> +on <a href="https://libera.chat/">Libera.Chat</a>, for all kinds of +Spectrum discussions. You can also join +through <a href="https://matrix.org/">Matrix</a> +as <a href="https://matrix.to/#/#spectrum:libera.chat">#spectrum:libera.chat</a>. +If you're new to IRC, you might Matrix the easiest way to get started. +The channel is <strong>logged</strong>, but the logs are not available +online yet. <h2 id="mailing-lists">Mailing lists</h2> -- 2.31.1

2 3

[PATCH crosvm v2] crosvm: support setting guest MAC from tap-fd
by Alyssa Ross 30 May '21

30 May '21

This adds a mac= option to crosvm's --tap-fd option. The virtio-net driver in the guest will read the desired MAC from virtio configuration space. See the documentation for VIRTIO_NET_F_MAC in the Virtio spec[1]. [1]: https://docs.oasis-open.org/virtio/virtio/v1.1/virtio-v1.1.html Thanks-to: Puck Meerburg <puck(a)puckipedia.com> --- v1: https://spectrum-os.org/lists/archives/spectrum-devel/20210409222026.13886-… This is a new version of the patch, updated for a much more recent crosvm. I'm posting it again because I had to change it enough that I feel it could use a re-review. The biggest change is that I gave up on a NetOptions struct, since upstream has added a new option, and it seems that their approach is going to be just passing ever more function arguments around. No use fighting that for my single addition. Also, change from RawFd to crosvm's RawDescriptor. It would be nice if we could use SafeDescriptor earlier here (which represents fd ownership), but I don't think that's really easily doable. devices/src/virtio/net.rs | 12 ++++++-- src/crosvm.rs | 8 +++-- src/linux.rs | 12 ++++---- src/main.rs | 63 ++++++++++++++++++++++++++++++--------- 4 files changed, 71 insertions(+), 24 deletions(-) diff --git a/devices/src/virtio/net.rs b/devices/src/virtio/net.rs index 92368c440..4d0ea1560 100644 --- a/devices/src/virtio/net.rs +++ b/devices/src/virtio/net.rs @@ -419,6 +419,7 @@ where } pub struct Net<T: TapT> { + mac_address: Option<MacAddress>, queue_sizes: Box<[u16]>, workers_kill_evt: Vec<Event>, kill_evts: Vec<Event>, @@ -439,6 +440,7 @@ where ip_addr: Ipv4Addr, netmask: Ipv4Addr, mac_addr: MacAddress, + guest_mac_addr: Option<MacAddress>, vq_pairs: u16, ) -> Result<Net<T>, NetError> { let multi_queue = if vq_pairs > 1 { true } else { false }; @@ -450,12 +452,12 @@ where tap.enable().map_err(NetError::TapEnable)?; - Net::from(base_features, tap, vq_pairs) + Net::with_tap(base_features, tap, vq_pairs, guest_mac_addr) } /// Creates a new virtio network device from a tap device that has already been /// configured. - pub fn from(base_features: u64, tap: T, vq_pairs: u16) -> Result<Net<T>, NetError> { + pub fn with_tap(base_features: u64, tap: T, vq_pairs: u16, mac_address: Option<MacAddress>) -> Result<Net<T>, NetError> { let taps = tap.into_mq_taps(vq_pairs).map_err(NetError::TapOpen)?; // This would also validate a tap created by Self::new(), but that's a good thing as it @@ -488,7 +490,12 @@ where workers_kill_evt.push(worker_kill_evt); } + if mac_address.is_some() { + avail_features |= 1 << virtio_net::VIRTIO_NET_F_MAC; + } + Ok(Net { + mac_address, queue_sizes: vec![QUEUE_SIZE; (vq_pairs * 2 + 1) as usize].into_boxed_slice(), workers_kill_evt, kill_evts, @@ -503,6 +510,7 @@ where let vq_pairs = self.queue_sizes.len() as u16 / 2; VirtioNetConfig { + mac: self.mac_address.map(|m| m.octets()).unwrap_or_else(Default::default), max_vq_pairs: Le16::from(vq_pairs), // Other field has meaningful value when the corresponding feature // is enabled, but all these features aren't supported now. diff --git a/src/crosvm.rs b/src/crosvm.rs index 04e267d6c..1dde7d769 100644 --- a/src/crosvm.rs +++ b/src/crosvm.rs @@ -171,6 +171,10 @@ impl Default for SharedDir { } } +pub struct TapFdOptions { + pub mac: Option<net_util::MacAddress>, +} + /// Aggregate of all configurable options for a running VM. pub struct Config { pub vcpu_count: Option<usize>, @@ -194,7 +198,7 @@ pub struct Config { pub mac_address: Option<net_util::MacAddress>, pub net_vq_pairs: Option<u16>, pub vhost_net: bool, - pub tap_fd: Vec<RawFd>, + pub tap_fd: BTreeMap<RawFd, TapFdOptions>, pub cid: Option<u64>, pub wayland_socket_paths: BTreeMap<String, PathBuf>, pub wayland_dmabuf: bool, @@ -253,7 +257,7 @@ impl Default for Config { mac_address: None, net_vq_pairs: None, vhost_net: false, - tap_fd: Vec::new(), + tap_fd: BTreeMap::new(), cid: None, #[cfg(feature = "gpu")] gpu_parameters: None, diff --git a/src/linux.rs b/src/linux.rs index f452fef3f..42c07df4f 100644 --- a/src/linux.rs +++ b/src/linux.rs @@ -75,7 +75,7 @@ use vm_memory::{GuestAddress, GuestMemory}; #[cfg(all(target_arch = "x86_64", feature = "gdb"))] use crate::gdb::{gdb_thread, GdbStub}; -use crate::{Config, DiskOption, Executable, SharedDir, SharedDirKind, TouchDeviceOption}; +use crate::{Config, DiskOption, Executable, SharedDir, SharedDirKind, TapFdOptions, TouchDeviceOption}; use arch::{ self, LinuxArch, RunnableLinuxVm, SerialHardware, SerialParameters, VcpuAffinity, VirtioDeviceStub, VmComponents, VmImage, @@ -675,7 +675,7 @@ fn create_balloon_device(cfg: &Config, socket: BalloonControlResponseSocket) -> }) } -fn create_tap_net_device(cfg: &Config, tap_fd: RawDescriptor) -> DeviceResult { +fn create_tap_net_device(cfg: &Config, tap_fd: RawDescriptor, options: &TapFdOptions) -> DeviceResult { // Safe because we ensure that we get a unique handle to the fd. let tap = unsafe { Tap::from_raw_descriptor( @@ -691,7 +691,7 @@ fn create_tap_net_device(cfg: &Config, tap_fd: RawDescriptor) -> DeviceResult { vq_pairs = 1; } let features = virtio::base_features(cfg.protected_vm); - let dev = virtio::Net::from(features, tap, vq_pairs).map_err(Error::NetDeviceNew)?; + let dev = virtio::Net::with_tap(features, tap, vq_pairs, options.mac).map_err(Error::NetDeviceNew)?; Ok(VirtioDeviceStub { dev: Box::new(dev), @@ -725,7 +725,7 @@ fn create_net_device( .map_err(Error::VhostNetDeviceNew)?; Box::new(dev) as Box<dyn VirtioDevice> } else { - let dev = virtio::Net::<Tap>::new(features, host_ip, netmask, mac_address, vq_pairs) + let dev = virtio::Net::<Tap>::new(features, host_ip, netmask, mac_address, None, vq_pairs) .map_err(Error::NetDeviceNew)?; Box::new(dev) as Box<dyn VirtioDevice> }; @@ -1311,8 +1311,8 @@ fn create_virtio_devices( devs.push(create_balloon_device(cfg, balloon_device_socket)?); // We checked above that if the IP is defined, then the netmask is, too. - for tap_fd in &cfg.tap_fd { - devs.push(create_tap_net_device(cfg, *tap_fd)?); + for (tap_fd, options) in &cfg.tap_fd { + devs.push(create_tap_net_device(cfg, *tap_fd, options)?); } if let (Some(host_ip), Some(netmask), Some(mac_address)) = diff --git a/src/main.rs b/src/main.rs index 5d02af02f..f8bc0d14e 100644 --- a/src/main.rs +++ b/src/main.rs @@ -28,7 +28,7 @@ use base::{ }; use crosvm::{ argument::{self, print_help, set_arguments, Argument}, - platform, BindMount, Config, DiskOption, Executable, GidMap, SharedDir, TouchDeviceOption, + platform, BindMount, Config, DiskOption, Executable, GidMap, SharedDir, TapFdOptions, TouchDeviceOption, DISK_ID_LEN, }; #[cfg(feature = "gpu")] @@ -1319,17 +1319,52 @@ fn set_argument(cfg: &mut Config, name: &str, value: Option<&str>) -> argument:: } "vhost-net" => cfg.vhost_net = true, "tap-fd" => { - cfg.tap_fd.push( - value - .unwrap() - .parse() - .map_err(|_| argument::Error::InvalidValue { - value: value.unwrap().to_owned(), - expected: String::from( - "this value for `tap-fd` must be an unsigned integer", - ), - })?, - ); + let mut components = value.unwrap().split(','); + + let fd: RawDescriptor = components + .next() + .and_then(|x| x.parse().ok()) + .ok_or_else(|| argument::Error::InvalidValue { + value: value.unwrap().to_owned(), + expected: String::from("this value for `tap-fd` must be an unsigned integer"), + })?; + + let mut mac = None; + for c in components { + let mut kv = c.splitn(2, '='); + let (kind, value) = match (kv.next(), kv.next()) { + (Some(kind), Some(value)) => (kind, value), + _ => { + return Err(argument::Error::InvalidValue { + value: c.to_owned(), + expected: String::from("option must be of the form `kind=value`"), + }) + } + }; + match kind { + "mac" => { + mac = Some(value.parse().map_err(|_| argument::Error::InvalidValue { + value: value.to_owned(), + expected: String::from( + "`mac` needs to be in the form \"XX:XX:XX:XX:XX:XX\"", + ), + })?) + } + _ => { + return Err(argument::Error::InvalidValue { + value: kind.to_owned(), + expected: String::from("unrecognized option"), + }) + } + } + } + if cfg.tap_fd.contains_key(&fd) { + return Err(argument::Error::TooManyArguments(format!( + "TAP FD already used: '{}'", + name + ))); + } + cfg.tap_fd.insert(fd, TapFdOptions { mac }); } #[cfg(feature = "gpu")] "gpu" => { @@ -1644,8 +1679,8 @@ writeback=BOOL - Indicates whether the VM can use writeback caching (default: fa Argument::value("plugin-gid-map-file", "PATH", "Path to the file listing supplemental GIDs that should be mapped in plugin jail. Can be given more than once."), Argument::flag("vhost-net", "Use vhost for networking."), Argument::value("tap-fd", - "fd", - "File descriptor for configured tap device. A different virtual network card will be added each time this argument is given."), + "FD[,mac=MAC]", + "File descriptor for configured tap device. A different virtual network card will be added each time this argument is given. MAC is the MAC address that will be set in the guest."), #[cfg(feature = "gpu")] Argument::flag_or_value("gpu", "[width=INT,height=INT]", base-commit: f35d2c43ff19520855cffee761dc8899c5a439a1 -- 2.31.1

2 5

[PATCH www] participating.html: add note about leaving Freenode
by Alyssa Ross 26 May '21

26 May '21

--- participating.html | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/participating.html b/participating.html index d04c2e5..aed91ab 100644 --- a/participating.html +++ b/participating.html @@ -77,6 +77,12 @@ is <strong>logged</strong>, and the logs are <a href="https://logs.spectrum-os.org/spectrum/">published</a> on the web. +<p> +<strong>We will be moving +to <a href="https://libera.chat/">Libera.Chat</a> as soon as +a <a href="https://github.com/matrix-org/matrix-appservice-irc/issues/1324">Matrix +bridge</a> is available. + <h3>Note for Matrix users</h3> <p> -- 2.31.0

2 1

[PATCH nixpkgs] spectrumPackages: use link-local router addresses
by Alyssa Ross 14 May '21

14 May '21

There's no need for every interface on a router to have it's own unique IP. Using link-local addresses, we can assign every interface on a single router the same IP, and save the effort of adding/subtracting one all the time to infer the address of one side of a link from the other. This means that routers and clients are both only interested in what the IP of the client should be, since that's the only thing that is different between different links. So we change our MAC format to communicate the client IP to both sides. If we just made the whole NIC portion of the MAC the last three octets of an IP address like we did before, then both ends of the link would have the same MAC, so we still need to set a bit to encode whether a MAC is at the router side of the link or the client side. I've decided to simplify the implementation by using the most significant bit of the NIC part for that, and just declaring the rest of that octet to be reserved. This way, no calculation is required in the guests -- they just need to look at the last two octets. And we can still identify 65536 links in this scheme, which should be enough for anybody, right? --- I'd like to move the networking to be IPv6. That's a bit more complicated, but shouldn't get in the way of smaller improvements like this. .../linux/spectrum/testhost/default.nix | 25 ++++++++----------- .../linux/spectrum/vm/app/default.nix | 15 ++++------- .../linux/spectrum/vm/net/default.nix | 13 +++++----- 3 files changed, 23 insertions(+), 30 deletions(-) diff --git a/pkgs/os-specific/linux/spectrum/testhost/default.nix b/pkgs/os-specific/linux/spectrum/testhost/default.nix index 21c585f1490..de62f0add67 100644 --- a/pkgs/os-specific/linux/spectrum/testhost/default.nix +++ b/pkgs/os-specific/linux/spectrum/testhost/default.nix @@ -45,23 +45,20 @@ let if { ip link set br0 up } # Calculate the MACs for our TAP and the router's TAP. - backtick -in router_nic_dec { - expr ${toString sys-vms.app.vmID} * 2 + 64 * 256 * 256 - } - backtick -in client_nic_dec { - expr ${toString sys-vms.app.vmID} * 2 + 64 * 256 * 256 + 1 - } - multisubstitute { - importas -iu router_nic_dec router_nic_dec - importas -iu client_nic_dec client_nic_dec - } + # MAC address format, by octet: + # + # 0-3 Static OUI for Spectrum + # 4 Most significant bit is used to differentiate + # routers from clients. Other bits are reserved. + # 5-6 Last two octets of client's IP (in 100.64.0.0/16). + # backtick -i router_mac { - pipeline { printf %x $router_nic_dec } - sed s/^\$..\$\$..\$\$..\$$/0A:B3:EC:\\1:\\2:\\3/ + pipeline { printf %.4x ${toString sys-vms.app.vmID} } + sed s/^\$..\$\$..\$$/0A:B3:EC:80:\\1:\\2/ } backtick -i client_mac { - pipeline { printf %x $client_nic_dec } - sed s/^\$..\$\$..\$\$..\$$/0A:B3:EC:\\1:\\2:\\3/ + pipeline { printf %.4x ${toString sys-vms.app.vmID} } + sed s/^\$..\$\$..\$$/0A:B3:EC:00:\\1:\\2/ } multisubstitute { importas -iu router_mac router_mac diff --git a/pkgs/os-specific/linux/spectrum/vm/app/default.nix b/pkgs/os-specific/linux/spectrum/vm/app/default.nix index 65dbb51f5e1..f8ff480932c 100644 --- a/pkgs/os-specific/linux/spectrum/vm/app/default.nix +++ b/pkgs/os-specific/linux/spectrum/vm/app/default.nix @@ -28,20 +28,15 @@ runCommand "vm-app" rec { up = writeText "net-up" '' backtick -i LOCAL_IP { pipeline { ip -j link show eth0 } - pipeline { jq -r ".[0].address | split(\":\") | .[3:6] | \"0x\" + .[]" } - xargs printf "100.%d.%d.%d" + pipeline { jq -r ".[0].address | split(\":\") | .[4:6] | \"0x\" + .[]" } + xargs printf "100.64.%d.%d" } importas -iu LOCAL_IP LOCAL_IP - backtick -i REMOTE_IP { - jq -jn --arg localip $LOCAL_IP - "$localip | split(\".\") | .[3] |= tonumber - 1 | join(\".\")" - } - importas -iu REMOTE_IP REMOTE_IP - - if { ip address add ''${LOCAL_IP}/31 dev eth0 } + if { ip address add ''${LOCAL_IP}/32 dev eth0 } if { ip link set eth0 up } - ip route add default via $REMOTE_IP + if { ip route add 169.254.0.1 dev eth0 } + ip route add default via 169.254.0.1 dev eth0 ''; }; diff --git a/pkgs/os-specific/linux/spectrum/vm/net/default.nix b/pkgs/os-specific/linux/spectrum/vm/net/default.nix index 1deb7031caf..5921b62fcf7 100644 --- a/pkgs/os-specific/linux/spectrum/vm/net/default.nix +++ b/pkgs/os-specific/linux/spectrum/vm/net/default.nix @@ -54,15 +54,16 @@ runCommand "vm-net" rec { # Our IP is encoded in the NIC-specific portion of the # interface's MAC address. - backtick -i LOCAL_IP { + backtick -i CLIENT_IP { pipeline { ip -j link show $INTERFACE } - pipeline { jq -r ".[0].address | split(\":\") | .[3:6] | \"0x\" + .[]" } - xargs printf "100.%d.%d.%d" + pipeline { jq -r ".[0].address | split(\":\") | .[4:6] | \"0x\" + .[]" } + xargs printf "100.64.%d.%d" } - importas -iu LOCAL_IP LOCAL_IP + importas -iu CLIENT_IP CLIENT_IP - if { ip address add ''${LOCAL_IP}/31 dev $INTERFACE } - ip link set $INTERFACE up + if { ip address add 169.254.0.1/32 dev $INTERFACE } + if { ip link set $INTERFACE up } + ip route add $CLIENT_IP dev $INTERFACE } { -- 2.31.1

2 1

[DEMO] virtio-vhost-user between QEMU and crosvm
by Alyssa Ross 13 May '21

13 May '21

Introduction ------------ Virtio-vhost-user[1] is a promising virtualisation technology that allows virtual devices that are exposed to VMs to themselves be implemented in VMs. Let's break down its name a bit to understand how it works: * Virtio[2] is a standard driver interface for virtualisation. Interfaces are available for all sorts of types of virtual devices, e.g. virtio-net, virtio-blk, and virtio-scsi. Typically, virtio devices are implemented by a virtual machine monitor (VMM). * Vhost[3] is kernelspace implementation of virtio virtual devices, created for their performance benefit. Instead of implementing the virtual devices itself, the VMM talks to the kernel implementation of them using a special ioctl protocol. * Vhost-user[4] allows another process to implement the vhost protocol, instead of the kernel, by using a UNIX socket instead of ioctls on a special character device. This doesn't provide the raw performance of vhost, but it serves a different purpose -- it allows virtual devices to be implemented by external programs, in a standardised way so they're portable between VMMs. Virtio-vhost-user allows the program implementing the virtual device to run in a VM of its own, by having the VMM for that VM create the vhost-user socket, and transferring messages over it to its guest using virtio. This is exciting for Spectrum, because it would mean that the host system doesn't have to interact with physical hardware directly beyond the PCI level, and can instead pass it through to a VM, which is responsible for implementing the virtual device backed by that physical hardware, which can be exposed to other VMs. Last year I spent a while looking into virtio-vhost-user[5][6][7]. It's a long way from being ready to use, and it seems to be maturing very slowly. It might be useful to us eventually for driver isolation, or something else might come along. My conclusion from my research was that we should decide later, once the ecosystem has had a chance to develop. But I wanted something to come out of the research I did anyway, and so I've prepared a demonstration. [1]: https://wiki.qemu.org/Features/VirtioVhostUser [2]: https://docs.oasis-open.org/virtio/virtio/v1.1/virtio-v1.1.html [3]: https://blog.vmsplice.net/2011/09/qemu-internals-vhost-architecture.html [4]: https://qemu.readthedocs.io/en/latest/interop/vhost-user.html [5]: https://spectrum-os.org/lists/archives/spectrum-devel/87pn8rezqn.fsf@alyssa… [6]: https://spectrum-os.org/lists/archives/spectrum-devel/87blk1pwph.fsf@alyssa… [7]: https://spectrum-os.org/lists/archives/spectrum-devel/87wo2glkg0.fsf@alyssa… What the demo does ------------------ Using any sort of non-host-based virtual device implementation is going to have to start with taking the virtual device implementation out of the VMM running an application VM, and vhost-user the clear solution to this. Vhost-user isn't supported by crosvm -- its focus is on doing all the virtualisation required by Chromium OS, so there's no need for it to allow other programs to provide virtual device implementations. So another part of the research I did was to try to port the vhost-user implementation from cloud-hypervisor to crosvm, which I was able to do successfully[8]. This has implications beyond vhost-user, and even beyond crosvm, because it demonstrates that it's practical to port features between rust-vmm[9] VMMs, which means we don't have to worry about finding one that provides every feature we need (which is just as well, because there isn't one). So here I demonstrate not just a "standard" virtio-vhost-user setup (to the extent that such a thing can exist at this early stage), but also that my patched crosvm with vhost-user support is capable of interoperating with the experimental virtio-vhost-user implementation for QEMU, because both are speaking the standardised vhost-user protocol. The demo sets up two VMs. One is run with my patched crosvm, and expects a virtual ethernet device to be provided by a vhost-user socket. When it boots, it brings up its network interface, tries to run a DHCP client, and then exits. The other VM is run with Nikos Dragazis and Stefan Hajnoczi's experimental virtio-vhost-user implementation in QEMU[10]. It gets a standard virtual ethernet device (backed by a TAP device on the host), and the virtio-vhost-user device hooked up to the socket that crosvm will be connecting to. Inside the VM, a userspace networking stack (DPDK, again modified to support virtio-vhost-user by Nikos Dragazis[11]) implements the device side of virtio-vhost-user, and forwards packets sent by crosvm's guest to the virtual ethernet device backed by the host TAP. +-----------------------------------------------------------------------------+ | | | +------------------------+ +------------------------+ | | | | | | | | | +----------------+ | | +----------------+ | | | | | | | | | | | | | +-----+ | | +--------+ | | | | | | | | | TAP +------+---+---+ DPDK +---+---+------+---+ | | | | +-----+ | | +--------+ | | | | | | | | | | | | | | | | | | | | Linux | | | | Linux | | | | | +----------------+ | | +----------------+ | | | | | | | | | | QEMU | | crosvm | | | +------------------------+ +------------------------+ | | | | Linux | +-----------------------------------------------------------------------------+ A complicating factor is that the virtio-vhost-user implementation for DPDK only supports outgoing traffic[12]. So packets coming from crosvm will be relayed to the TAP, but not the other way around. This means that we can't just use ping inside the crosvm VM to verify that the connection is working. Instead, we have to tcpdump on the host and verify that the packets the DHCP client inside the crosvm VM is sending are arriving on the TAP. For this to be useful for our intended purpose of isolating drivers for physical devices, we'd pass through the device here rather than using a TAP. It would otherwise work exactly the same, but it's more difficult to test it's working correctly. (I have tested it though -- for the first version of this I got working last year, I verified it worked by checking the logs of my local network's DHCP server.) [8]: https://spectrum-os.org/lists/archives/spectrum-devel/20210512170812.192540… [9]: https://github.com/rust-vmm [10]: https://github.com/ndragazis/qemu/tree/virtio-vhost-user [11]: https://github.com/ndragazis/dpdk-next-virtio/tree/virtio-vhost-user [12]: https://github.com/ndragazis/dpdk-next-virtio/blob/2d60e63/drivers/virtio_v… Running the demo ---------------- First, create a TAP device for QEMU to use: # ip tuntap add qemutap mode tap # ip link set qemutap up Start tcpdump, so we can see if packets arrive on the TAP: # tcpdump -i qemutap Start the QEMU VM: $ $(nix-build -A qemuVm /path/to/demo.nix) When you see "Press enter to exit", DPDK is ready to receive a virtio-vhost-user connection. Start the crosvm VM: $ $(nix-build -A crosvmVm /path/to/demo.nix) Once that VM boots, you should see some "BOOTP/DHCP" lines in the tcpdump output. This demonstrates that traffic from the crosvm guest has been relayed over virtio-vhost-user to DPDK, and then to the TAP on the host over virtio-net. You'll want to press enter to shut down the QEMU VM now, because DPDK pegs a CPU core (for reasons[*] unrelated to virtio-vhost-user that are out of scope here). Then you can remove the TAP device: # ip link delete qemutap Nix expression for the demo --------------------------- # SPDX-License-Identifier: MIT OR Apache-2.0 # SPDX-FileCopyrightText: 2021 Alyssa Ross <hi(a)alyssa.is> let pinned = builtins.fetchTarball { url = "https://github.com/NixOS/nixpkgs/tarball/b14062b75c4e8ef4dd4110282f7105be87…"; sha256 = "1hzs0w6pcwwbzl2gkqyk46yrzizzm03mph4kggws02a6vlwphsib"; }; in { pkgs ? import pinned {} }: with pkgs; rec { linux = pkgs.linux.override { structuredExtraConfig = with lib.kernel; { "9P_FS" = yes; NET_9P = yes; NET_9P_VIRTIO = yes; PACKET = yes; VFIO = yes; VFIO_NOIOMMU = yes; VFIO_PCI = yes; VIRTIO_NET = yes; VIRTIO_PCI = yes; }; }; dpdk = stdenv.mkDerivation { name = "dpdk-virtio-vhost-user"; src = fetchFromGitHub { owner = "ndragazis"; repo = "dpdk-next-virtio"; rev = "0a46582dc1d02c0dc5069347ffff1a64239385f2"; sha256 = "169cxdps9k764jj420q44262x3291h2jcqsbrh7038hqjczjkgif"; }; buildInputs = [ numactl ]; configurePhase = '' runHook preConfigure make $makeFlags defconfig runHook postConfigure ''; enableParallelBuilding = true; RTE_KERNELDIR = "${linux.dev}/lib/modules/${linux.modDirVersion}/build"; NIX_CFLAGS_COMPILE = [ "-Wno-error=implicit-fallthrough" "-Wno-error=incompatible-pointer-types" ]; makeFlags = [ "RTE_OUTPUT=$(out)/lib" "kerneldir=$(out)/lib/modules/${linux.modDirVersion}/build" "prefix=$(out)" ]; inherit (pkgs.dpdk) meta; }; # DPDK is huge! We just need one program from it. testpmd = runCommandNoCC "testpmd" {} '' mkdir -p $out/bin cp ${dpdk}/bin/testpmd $out/bin ''; # qemu has changed build system since the virtio-vhost-user branch # was last updated, so it's simpler to just make a new derivation # and inherit the bits that are the same than to override the # existing one. qemu = stdenv.mkDerivation { name = "qemu-virtio-vhost-user"; src = fetchFromGitHub { owner = "ndragazis"; repo = "qemu"; rev = "f9ab08c0c8cfc58036ed95b895f9780397448071"; sha256 = "0p6v4i7gj70d6x7s28x3i3x9z8vlswcbbqdwfbhlx87bbnxjrn3b"; fetchSubmodules = true; }; enableParallelBuilding = true; nativeBuildInputs = lib.subtractLists [ ninja meson ] qemu_kvm.nativeBuildInputs; postPatch = '' sed -i '/$(INSTALL_DIR) "$(DESTDIR)$(qemu_localstatedir)/d' Makefile # The virtio-vhost-user implementation tries to allocate a huge # PCI bar, that's bigger than some CPUs can support! If you see # a kernel panic in vp_reset(), lower this further. substituteInPlace hw/virtio/virtio-vhost-user-pci.c \ --replace '1ULL << 36' '1ULL << 34' ''; inherit (qemu_kvm) buildInputs configureFlags meta; }; qemuInitramfs = makeInitrd { contents = [ { symlink = "/init"; object = writeScript "init" '' #!${busybox}/bin/sh -eux export PATH=${busybox}/bin mkdir -p /nix/store /run /var mount -t sysfs none /sys mount -t proc none /proc mount -t tmpfs none /run mount -t devtmpfs none /dev mkdir /dev/hugepages mount -t hugetlbfs none /dev/hugepages ln -s /run /var # Unbind the virtio-net (host TAP) and virtio-vhost-user devices # from their default drivers, since we'll be passing them # through to DPDK. echo 0000:00:04.0 > /sys/bus/pci/devices/0000:00:04.0/driver/unbind echo 0000:00:05.0 > /sys/bus/pci/devices/0000:00:05.0/driver/unbind # Tell the vfio-pci driver it can support virtio-net and # virtio-vhost-user devices. Since our devices are not # bound to any driver at the moment, doing this will bind # them to vfio-pci automatically. echo 1af4 1000 > /sys/bus/pci/drivers/vfio-pci/new_id echo 1af4 1017 > /sys/bus/pci/drivers/vfio-pci/new_id echo 256 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages ${testpmd}/bin/testpmd \ -l 0-1 \ -w 0000:00:05.0 \ --vdev net_vhost0,iface=0000:00:05.0,virtio-transport=1 \ -w 0000:00:04.0 poweroff -f ''; } ]; }; qemuVm = writeShellScript "qemu-vm" '' exec ${qemu}/bin/qemu-system-x86_64 -enable-kvm -cpu host -smp 2 -m 1G \ -M q35,kernel-irqchip=split \ -initrd ${qemuInitramfs}/initrd \ -netdev tap,id=net0,ifname=qemutap,script=no,downscript=no \ -device virtio-net-pci,netdev=net0,addr=04 \ -chardev socket,id=chardev0,path="$XDG_RUNTIME_DIR/vhost-user0.sock",server,nowait \ -device virtio-vhost-user-pci,addr=05,chardev=chardev0 \ -kernel ${linux}/${stdenv.hostPlatform.linux-kernel.target} \ -append "console=ttyS0 vfio.enable_unsafe_noiommu_mode=1" \ -nographic ''; # Can't use overrideAttrs because of cargoSha256. crosvm = rustPlatform.buildRustPackage rec { name = "crosvm-virtio-vhost-user"; src = fetchFromGitiles { url = "https://chromium.googlesource.com/chromiumos/platform/crosvm"; rev = "8a7e4e902a4950b060ea23b40c0dfce7bfa1b2cb"; sha256 = "1lm6psp0xakb66nhgmmh94valc4wzbb967chk80msk8bcvsfpdn4"; }; unpackPhase = let origSrc = pkgs.crosvm.passthru.src; in builtins.replaceStrings [ "${origSrc}" origSrc.name ] [ "$src" src.name ] pkgs.crosvm.unpackPhase; cargoPatches = [ (fetchpatch { url = "https://spectrum-os.org/lists/archives/spectrum-devel/20210512170812.192540…"; sha256 = "0yzqrpgq35s9wxvbf9s3dgs5cpyxgdc5hr14hsdjr0gd18a6camg"; }) ]; patches = pkgs.crosvm.patches ++ [ (fetchpatch { url = "https://spectrum-os.org/lists/archives/spectrum-devel/20210512170812.192540…"; sha256 = "0g2rvqqa4lvq7bjq0s1ynsjx7lmrxql7lsdv8wyzb7d2z9j6mj13"; }) (fetchpatch { url = "https://spectrum-os.org/lists/archives/spectrum-devel/20210512170812.192540…"; sha256 = "051sz87i8kzc5sbygk2bpiqp4g32y9fxswg2yax1nd3lg4rxh43r"; }) (fetchpatch { url = "https://spectrum-os.org/lists/archives/spectrum-devel/20210512170812.192540…"; sha256 = "1jpas65masn2xg9jxha16vi0y7scarzhl221y9wxh4chi4aa4m3f"; }) ]; cargoSha256 = "07yizbhs64jrb05fq5g7sx812xbz2989bsficacq5l19ziax5164"; passthru = pkgs.crosvm.passthru // { inherit src; }; inherit (pkgs.crosvm) sourceRoot postPatch nativeBuildInputs buildInputs preBuild postInstall CROSVM_CARGO_TEST_KERNEL_BINARY meta; }; crosvmInitramfs = makeInitrd { contents = [ { symlink = "/init"; object = writeScript "init" '' #!${busybox}/bin/sh -eux export PATH=${busybox}/bin mount -t sysfs none /sys mount -t proc none /proc ip link set eth0 up udhcpc -n || : reboot -f ''; } ]; }; crosvmVm = writeShellScript "crosvm-vm" '' # In our patched crosvm, suppling --mac without --host_ip or # --netmask will put it into vhost-user mode. exec ${crosvm}/bin/crosvm run \ --mac 0A:B3:EC:FF:FF:FF \ -i ${crosvmInitramfs}/initrd \ ${linux}/${stdenv.hostPlatform.linux-kernel.target} ''; }

1 0

[RFC PATCH crosvm 0/4] Implement vhost-user-net
by Alyssa Ross 12 May '21

12 May '21

This is a proof of concept, so no need for code review on it. Here I have taken cloud-hypervisor's implementation of vhost-user-net, and ported it to crosvm. I based this on old versions of crosvm and cloud-hypervisor, with the idea being that it would be easier to get an initial working version from a time when the codebases had diverged less, which could then be gradually brought forward. A slight complication is that the cloud-hypervisor version I ported the implementation from didn't quite behave in the way the vhost-user specification[1] said it should, so patches 2 and 3 are backports of cloud-hypervisor fixes that happened later. Alyssa Ross (4): devices: port vhost-user-net from cloud-hypervisor vhost_rs: vhost_user: Add missing protocol features devices: vhost-user: Vring should be enabled after initialization crosvm: use vhost-user-net instead of virtio-net Cargo.lock | 731 ++++++++++------ Cargo.toml | 2 +- devices/Cargo.toml | 8 + devices/src/virtio/mod.rs | 2 + .../virtio/vhost_user/cloud_hypervisor/mod.rs | 1 + .../vhost_user/cloud_hypervisor/net_util.rs | 162 ++++ devices/src/virtio/vhost_user/handler.rs | 123 +++ devices/src/virtio/vhost_user/mod.rs | 82 ++ devices/src/virtio/vhost_user/net.rs | 230 +++++ .../src/virtio/vhost_user/vu_common_ctrl.rs | 110 +++ src/linux.rs | 32 +- src/main.rs | 2 +- vhost_rs/Cargo.toml | 26 + vhost_rs/LICENSE | 202 +++++ vhost_rs/LICENSE-BSD | 27 + vhost_rs/LICENSE-MIT | 24 + vhost_rs/README.md | 10 + vhost_rs/src/backend.rs | 130 +++ vhost_rs/src/lib.rs | 122 +++ vhost_rs/src/vhost_kern/mod.rs | 320 +++++++ vhost_rs/src/vhost_kern/vhost_binding.rs | 405 +++++++++ vhost_rs/src/vhost_kern/vsock.rs | 84 ++ vhost_rs/src/vhost_user/connection.rs | 737 ++++++++++++++++ vhost_rs/src/vhost_user/dummy_slave.rs | 250 ++++++ vhost_rs/src/vhost_user/master.rs | 757 ++++++++++++++++ vhost_rs/src/vhost_user/master_req_handler.rs | 258 ++++++ vhost_rs/src/vhost_user/message.rs | 823 ++++++++++++++++++ vhost_rs/src/vhost_user/mod.rs | 251 ++++++ vhost_rs/src/vhost_user/slave.rs | 48 + vhost_rs/src/vhost_user/slave_req_handler.rs | 582 +++++++++++++ vhost_rs/src/vhost_user/sock_ctrl_msg.rs | 464 ++++++++++ vhost_rs/src/vsock.rs | 30 + virtio-bindings/Cargo.toml | 6 + virtio-bindings/src/lib.rs | 15 + virtio-bindings/src/virtio_blk.rs | 486 +++++++++++ virtio-bindings/src/virtio_net.rs | 734 ++++++++++++++++ virtio-bindings/src/virtio_ring.rs | 453 ++++++++++ 37 files changed, 8461 insertions(+), 268 deletions(-) create mode 100644 devices/src/virtio/vhost_user/cloud_hypervisor/mod.rs create mode 100644 devices/src/virtio/vhost_user/cloud_hypervisor/net_util.rs create mode 100644 devices/src/virtio/vhost_user/handler.rs create mode 100644 devices/src/virtio/vhost_user/mod.rs create mode 100644 devices/src/virtio/vhost_user/net.rs create mode 100644 devices/src/virtio/vhost_user/vu_common_ctrl.rs create mode 100644 vhost_rs/Cargo.toml create mode 100644 vhost_rs/LICENSE create mode 100644 vhost_rs/LICENSE-BSD create mode 100644 vhost_rs/LICENSE-MIT create mode 100644 vhost_rs/README.md create mode 100644 vhost_rs/src/backend.rs create mode 100644 vhost_rs/src/lib.rs create mode 100644 vhost_rs/src/vhost_kern/mod.rs create mode 100644 vhost_rs/src/vhost_kern/vhost_binding.rs create mode 100644 vhost_rs/src/vhost_kern/vsock.rs create mode 100644 vhost_rs/src/vhost_user/connection.rs create mode 100644 vhost_rs/src/vhost_user/dummy_slave.rs create mode 100644 vhost_rs/src/vhost_user/master.rs create mode 100644 vhost_rs/src/vhost_user/master_req_handler.rs create mode 100644 vhost_rs/src/vhost_user/message.rs create mode 100644 vhost_rs/src/vhost_user/mod.rs create mode 100644 vhost_rs/src/vhost_user/slave.rs create mode 100644 vhost_rs/src/vhost_user/slave_req_handler.rs create mode 100644 vhost_rs/src/vhost_user/sock_ctrl_msg.rs create mode 100644 vhost_rs/src/vsock.rs create mode 100644 virtio-bindings/Cargo.toml create mode 100644 virtio-bindings/src/lib.rs create mode 100644 virtio-bindings/src/virtio_blk.rs create mode 100644 virtio-bindings/src/virtio_net.rs create mode 100644 virtio-bindings/src/virtio_ring.rs base-commit: 8a7e4e902a4950b060ea23b40c0dfce7bfa1b2cb -- 2.31.0

1 4

[PATCH nixpkgs] spectrumPackages.makeRootfs: fix messagebus group
by Alyssa Ross 22 Apr '21

22 Apr '21

/etc/passwd set the messagebus user's primary group to 1, but there was no group with that id in /etc/group. The messagebus group had id 4. The 4 comes from NixOS. I think it's probably a good idea to use the same user and group ids where possible -- there's no particular reason to diverge -- so let's fix it in /etc/passwd instead of changing the id in /etc/group to 1. And while we're here, for consistency, let's change the user id of messagebus to 4 as well, so that's consistent with NixOS as well. I don't think the user ID is used anywhere else. This fixes the DBus "Permission denied" errors that have been showing up in vm-net since forever (but never seemed to cause any real harm...). --- pkgs/os-specific/linux/spectrum/rootfs/etc/passwd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pkgs/os-specific/linux/spectrum/rootfs/etc/passwd b/pkgs/os-specific/linux/spectrum/rootfs/etc/passwd index c76fb8f2a99..ddb1f854a6e 100644 --- a/pkgs/os-specific/linux/spectrum/rootfs/etc/passwd +++ b/pkgs/os-specific/linux/spectrum/rootfs/etc/passwd @@ -1,3 +1,3 @@ root:x:0:0:System administrator:/:/bin/sh -messagebus:x:1:1:D-Bus system message bus daemon user:/run/dbus:/bin/sh +messagebus:x:4:4:D-Bus system message bus daemon user:/run/dbus:/bin/sh user:x:1000:1000:User:/:/bin/sh -- 2.31.0

3 4

[PATCH nixpkgs] spectrumPackages.spectrum-testhost: fix driver bind
by Alyssa Ross 22 Apr '21

22 Apr '21

The following operations are available to us in /sys/bus/pci/drivers/vfio-pci: bind, unbind, new_id, remove_id bind and unbind control attachment of a device to a driver. new_id and remove_id control whether a device should be considered "supported" by a driver. But when an unassigned device is newly supported by a driver, the kernel will automatically bind it to that driver. We were relying on the new_id operation binding the device to the driver in this way. But if the driver is in the supported list, but not bound to anything, this won't happen. new_id won't do anything, because it's already in the supported list. So we can't rely on new_id binding for us, and need to try bind as well (at least if new_id fails). To reproduce: echo 8086 1502 > /sys/bus/pci/drivers/vfio-pci/new_id echo 0000:00:19.0 > /sys/bus/pci/drivers/vfio-pci/unbind (Use lspci -nn to find the right values for your hardware.) This will leave you with a device that is supported by vfio-pci but not bound to it, which would previously cause vm-net to fail to start, but should not after this change. --- .../linux/spectrum/testhost/default.nix | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/pkgs/os-specific/linux/spectrum/testhost/default.nix b/pkgs/os-specific/linux/spectrum/testhost/default.nix index 7e1a973e8c6..21c585f1490 100644 --- a/pkgs/os-specific/linux/spectrum/testhost/default.nix +++ b/pkgs/os-specific/linux/spectrum/testhost/default.nix @@ -142,7 +142,9 @@ let printf "%s" $PCI_LOCATION } - # (Re)bind the device to the VFIO PCI driver. + # Tell the VFIO driver it should support our device. This + # is allowed to fail because it might already know that, in + # which case it'll return EEXIST. if { modprobe vfio-pci } backtick -in device_id { if { dd bs=2 skip=1 count=2 status=none if=''${PCI_PATH}/vendor } @@ -155,6 +157,20 @@ let printf "%s" $device_id } + # Bind the device to the VFIO driver. This is allowed to + # fail because the new_id operation we just tried will have + # bound it automatically for us if it succeeded. In such a + # case, the kernel will return ENODEV (conistency!). + foreground { + redirfd -w 1 /sys/bus/pci/drivers/vfio-pci/bind + printf "%s" $PCI_LOCATION + } + + # Because we allow both new_id and bind to fail, we need to + # manually make sure now that at least one of them succeeded + # and the device is actually attached to the vfio-driver. + if { test -e /sys/bus/pci/drivers/vfio-pci/''${PCI_LOCATION} } + foreground { mkdir env } ${cloud-hypervisor}/bin/cloud-hypervisor -- 2.31.0

2 2

[PATCH nixpkgs 00/16] Inter-guest networking
by Alyssa Ross 14 Apr '21

14 Apr '21

In Spectrum, we want the host kernel to include as few drivers as possible, to reduce attack service. To accomplish this, we need to move as much hardware interaction as possible into VMs. This series introduces proof-of-concept network hardware isolation by passing through network devices to a VM, and having that VM handle all interaction with that hardware instead of the host system. Background ---------- Ideally, the Spectrum host system wouldn't have to have support for networking at all. Network hardware could be handled by a VM, which would act a bit like a router, and export virtual network devices directly to other VMs. The hard part of that ideal is the bit where a VM exports a virtual device to another VM. There's work going on in the Linux virtualisation ecosystem to make that possible, through a protocol called virtio-vhost-user[1]. But it looks like it's going to be a long time before virtio-vhost-user is baked enough to be useful to us, and inter-guest networking is so fundamentally important to Spectrum that we can't afford to wait that long. So we need a plan B. Because we have no way to have a VM provide a virtual device, the host is going to need to do it. And this means that some amount of networking is going to have to happen on the host. But we can still get most of what we were going for: * Pass hardware network devices through to a dedicated VM. * Attach a virtual network device to the same VM. * Connect htat virtual network device to virtual network devices for other VMs on the host. With this approach, the host is still doing networking, but there's a lot less code involved, because instead of having loads of drivers for every kind of Wi-Fi card and so on available on the host, the host only ever needs to use the driver for the virtual network devices. If we connect each network client VM to the router VM using a bridge (i.e. a virtual Ethernet switch), we don't even need to worry about IP addresses or routing tables on the host. (There are some drawbacks to this approach, which are elaborated later in the series.) So that's what's implemented here. [1]: https://wiki.qemu.org/Features/VirtioVhostUser Implementation -------------- This series starts with some cherry-picks from upstream Nixpkgs. (We should sync with upstream soon, but I didn't want to delay this any longer by blocking it on a Nixpkgs sync.) Then there's some important refactoring to make it possible to fit multiple VM definitions into spectrumPackages. (Until now, there's only been one.) Finally, I implement two VMs -- one to act as the router and one to be the client -- as well as a service manager that can set everything up on the host to be able to run those VMs, and run them. Don't read too much into the structure of the Nix code for the VMs. I just needed _some_ structure, and this was what I came up with. As we explore the configuration aspect of Spectrum more, I expect it to change dramatically. There's quite a bit of duplication between VM definitions, but I don't think it's worth spending time getting rid of that for now when the entire structure could change before it becomes an issue. The important thing here is what's going on at runtime on the host and in the VMs, not the Nix code used to build that. Everything in the router VM is set up to be able to handle multiple clients coming and going at runtime (with the exception of a small issue with clients going away that I'll get into later in the series). The only reason the client VM is defined up front is because that's what fits our Nix code. There's no reason from the networking point of view that client VMs couldn't be instantiated entirely at runtime. One thing that might jump out to anybody skimming the series is that we're using cloud-hypervisor here. This is because there's a feature it supports that crosvm doesn't (I'll go into detail about what exactly later in the series), and it's more expedient to just use both VMMs as required by the needs of the VMs they're running than it would be to port features from cloud-hypervisor to crosvm or vice versa. Obviously using multiple different VMMs on the host is not good in the long term, but things are so in flux at the moment that it's very likely that any porting work will no longer be in use by the time we need to pare down to a single VMM anyway. Testing ------- If you want to try this out for yourself (and I encourage you to do so and reply to me with a Tested-by!), here's what you need to do: 1. Identify the PCI location of a physical Ethernet device on your system. `lspci -n' should help with this. 2. Modify pkgs/os-specific/linux/spectrum/testhost/default.nix to define PCI_LOCATION to be your one. In future, we can either have a configuration option for this sort of thing, or try to figure it out at runtime. 3. Start the service manager: sudo env XDG_RUNTIME_DIR=/run $(nix-build -A spectrumPackages.spectrum-testhost)/bin/spectrum-testhost A temporary directory will be created for service manager state. Its location will be printed to the terminal. 4. At another terminal, tell the service manager to start the "application VM". This is the client that will be connected to the router VM. s6-rc -u -l /run/spectrum.sy2huQuC3x/s6-rc/live change vm-app Remember to substitute in your temporary directory. You'll need the s6-rc command available (it's in Nixpkgs). This tells the service manager to start the service named vm-app. This service has a dependency on another VM called vm-net, which is the router VM that deals with the network hardware, so both will be started when you ask s6-rc to start vm-app. 5. The terminal running the service manager will be connected to the serial console of client VM. After a few seconds (because the router VM will probably have to sort out DHCP first), you should be able to ping hosts on the internet. Note that the client VM doesn't come with DNS set up. 6. Once you're done, you can tell the service manager to stop the VMs: s6-rc -da -l /run/spectrum.sy2huQuC3x/s6-rc/live change Again, remember to substitute your temporary directory. The service manager itself won't respond to ^C. I believe this is handled better in more recent versions of s6, which we'll get when we sync Nixpkgs. After this, it's a good idea to reboot, to restore all the network devices to their default state. Alyssa Ross (16): linux: enable Xen everywhere it can be cloud-hypervisor: 0.8.0 -> 0.14.1 mdevd: init at 0.1.3.0 spectrumPackages.linux_vm: fix cloud-hypervisor hotplug spectrumPackages.linux_vm: allow config overrides crosvm: support setting guest MAC from --tap-fd spectrumPackages: export makeRootfs spectrumPackages.rootfs: add s6-rc support spectrumPackages.rootfs: make /var/lib and /var/run spectrumPackages.rootfs: add dbus configuration spectrumPackages.rootfs: add connman dbus services spectrumPackages.sys-vms.comp: init spectrumPackages.makeRootfs: move to default.nix spectrumPackages.sys-vms.net: init spectrumPackages.sys-vms.app: init spectrumPackages.spectrum-testhost: init .../cargo-lock-vendor-fix.patch | 53 ---- .../cloud-hypervisor/default.nix | 15 +- ...upport-setting-guest-MAC-from-tap-fd.patch | 294 ++++++++++++++++++ .../linux/chromium-os/crosvm/default.nix | 1 + .../linux/kernel/common-config.nix | 13 +- pkgs/os-specific/linux/kernel/patches.nix | 9 + pkgs/os-specific/linux/mdevd/default.nix | 28 ++ pkgs/os-specific/linux/spectrum/default.nix | 6 +- pkgs/os-specific/linux/spectrum/linux/vm.nix | 7 +- .../linux/spectrum/rootfs/default.nix | 92 +++--- .../linux/spectrum/rootfs/etc/group | 1 + .../linux/spectrum/rootfs/etc/passwd | 1 + .../linux/spectrum/rootfs/generic.nix | 48 --- .../linux/spectrum/rootfs/rc-services.nix | 26 ++ .../linux/spectrum/rootfs/stage1.nix | 25 +- .../linux/spectrum/spectrum-vm/default.nix | 6 +- .../linux/spectrum/testhost/default.nix | 205 ++++++++++++ .../linux/spectrum/vm/app/default.nix | 63 ++++ .../linux/spectrum/vm/comp/default.nix | 86 +++++ .../os-specific/linux/spectrum/vm/default.nix | 9 + .../linux/spectrum/vm/net/default.nix | 165 ++++++++++ pkgs/top-level/aliases.nix | 6 + pkgs/top-level/all-packages.nix | 12 +- 23 files changed, 976 insertions(+), 195 deletions(-) delete mode 100644 pkgs/applications/virtualization/cloud-hypervisor/cargo-lock-vendor-fix.patch create mode 100644 pkgs/os-specific/linux/chromium-os/crosvm/0001-crosvm-support-setting-guest-MAC-from-tap-fd.patch create mode 100644 pkgs/os-specific/linux/mdevd/default.nix delete mode 100644 pkgs/os-specific/linux/spectrum/rootfs/generic.nix create mode 100644 pkgs/os-specific/linux/spectrum/rootfs/rc-services.nix create mode 100644 pkgs/os-specific/linux/spectrum/testhost/default.nix create mode 100644 pkgs/os-specific/linux/spectrum/vm/app/default.nix create mode 100644 pkgs/os-specific/linux/spectrum/vm/comp/default.nix create mode 100644 pkgs/os-specific/linux/spectrum/vm/default.nix create mode 100644 pkgs/os-specific/linux/spectrum/vm/net/default.nix -- 2.30.0

2 19

[PATCH crosvm] crosvm: support setting guest MAC from tap-fd
by Alyssa Ross 14 Apr '21

14 Apr '21

This adds a mac= option to crosvm's --tap-fd option. The virtio-net driver in the guest will read the desired MAC from virtio configuration space. See the documentation for VIRTIO_NET_F_MAC in the Virtio spec[1]. [1]: https://docs.oasis-open.org/virtio/virtio/v1.1/virtio-v1.1.html --- This will be important for host-based networking in Spectrum. devices/src/virtio/net.rs | 31 ++++++++++++++++--- src/crosvm.rs | 8 +++-- src/linux.rs | 20 +++++++----- src/main.rs | 64 ++++++++++++++++++++++++++++++--------- 4 files changed, 96 insertions(+), 27 deletions(-) diff --git a/devices/src/virtio/net.rs b/devices/src/virtio/net.rs index 44a39abd..fe71371f 100644 --- a/devices/src/virtio/net.rs +++ b/devices/src/virtio/net.rs @@ -22,7 +22,9 @@ use virtio_sys::virtio_net::{ }; use virtio_sys::{vhost, virtio_net}; -use super::{DescriptorError, Interrupt, Queue, Reader, VirtioDevice, Writer, TYPE_NET}; +use super::{ + copy_config, DescriptorError, Interrupt, Queue, Reader, VirtioDevice, Writer, TYPE_NET, +}; const QUEUE_SIZE: u16 = 256; const NUM_QUEUES: usize = 3; @@ -373,7 +375,13 @@ where } } +#[derive(Default)] +pub struct NetOptions { + pub guest_mac: Option<net_util::MacAddress>, +} + pub struct Net<T: TapT> { + config: Vec<u8>, workers_kill_evt: Option<EventFd>, kill_evt: EventFd, worker_thread: Option<thread::JoinHandle<Worker<T>>>, @@ -392,6 +400,7 @@ where ip_addr: Ipv4Addr, netmask: Ipv4Addr, mac_addr: MacAddress, + options: NetOptions, ) -> Result<Net<T>, NetError> { let tap: T = T::new(true).map_err(NetError::TapOpen)?; tap.set_ip_addr(ip_addr).map_err(NetError::TapSetIp)?; @@ -401,18 +410,18 @@ where tap.enable().map_err(NetError::TapEnable)?; - Net::from(tap) + Net::with_tap(tap, options) } /// Creates a new virtio network device from a tap device that has already been /// configured. - pub fn from(tap: T) -> Result<Net<T>, NetError> { + pub fn with_tap(tap: T, options: NetOptions) -> Result<Net<T>, NetError> { // This would also validate a tap created by Self::new(), but that's a good thing as it // would ensure that any changes in the creation procedure are matched in the validation. // Plus we still need to set the offload and vnet_hdr_size values. validate_and_configure_tap(&tap)?; - let avail_features = 1 << virtio_net::VIRTIO_NET_F_GUEST_CSUM + let mut avail_features = 1 << virtio_net::VIRTIO_NET_F_GUEST_CSUM | 1 << virtio_net::VIRTIO_NET_F_CSUM | 1 << virtio_net::VIRTIO_NET_F_CTRL_VQ | 1 << virtio_net::VIRTIO_NET_F_CTRL_GUEST_OFFLOADS @@ -422,8 +431,18 @@ where | 1 << virtio_net::VIRTIO_NET_F_HOST_UFO | 1 << vhost::VIRTIO_F_VERSION_1; + if options.guest_mac.is_some() { + avail_features |= 1 << virtio_net::VIRTIO_NET_F_MAC; + } + + let config = options + .guest_mac + .map(|mac| mac.octets().to_vec()) + .unwrap_or_default(); + let kill_evt = EventFd::new().map_err(NetError::CreateKillEventFd)?; Ok(Net { + config, workers_kill_evt: Some(kill_evt.try_clone().map_err(NetError::CloneKillEventFd)?), kill_evt, worker_thread: None, @@ -545,6 +564,10 @@ where } } + fn read_config(&self, offset: u64, data: &mut [u8]) { + copy_config(data, 0, self.config.as_slice(), offset); + } + fn activate( &mut self, mem: GuestMemory, diff --git a/src/crosvm.rs b/src/crosvm.rs index 81344c32..e69f2dfc 100644 --- a/src/crosvm.rs +++ b/src/crosvm.rs @@ -157,6 +157,10 @@ impl Default for SharedDir { } } +pub struct TapFdOptions { + pub mac: Option<net_util::MacAddress>, +} + /// Aggregate of all configurable options for a running VM. pub struct Config { pub vcpu_count: Option<u32>, @@ -177,7 +181,7 @@ pub struct Config { pub netmask: Option<net::Ipv4Addr>, pub mac_address: Option<net_util::MacAddress>, pub vhost_net: bool, - pub tap_fd: Vec<RawFd>, + pub tap_fd: BTreeMap<RawFd, TapFdOptions>, pub cid: Option<u64>, pub wayland_socket_paths: BTreeMap<String, PathBuf>, pub wayland_dmabuf: bool, @@ -224,7 +228,7 @@ impl Default for Config { netmask: None, mac_address: None, vhost_net: false, - tap_fd: Vec::new(), + tap_fd: BTreeMap::new(), cid: None, #[cfg(feature = "gpu")] gpu_parameters: None, diff --git a/src/linux.rs b/src/linux.rs index 3370c1e1..f7f78ad2 100644 --- a/src/linux.rs +++ b/src/linux.rs @@ -60,7 +60,9 @@ use vm_control::{ VmMemoryRequest, VmMemoryResponse, VmRunMode, }; -use crate::{Config, DiskOption, Executable, SharedDir, SharedDirKind, TouchDeviceOption}; +use crate::{ + Config, DiskOption, Executable, SharedDir, SharedDirKind, TapFdOptions, TouchDeviceOption, +}; use arch::{self, LinuxArch, RunnableLinuxVm, VirtioDeviceStub, VmComponents, VmImage}; #[cfg(any(target_arch = "arm", target_arch = "aarch64"))] @@ -586,14 +588,18 @@ fn create_balloon_device(cfg: &Config, socket: BalloonControlResponseSocket) -> }) } -fn create_tap_net_device(cfg: &Config, tap_fd: RawFd) -> DeviceResult { +fn create_tap_net_device(cfg: &Config, tap_fd: RawFd, options: &TapFdOptions) -> DeviceResult { // Safe because we ensure that we get a unique handle to the fd. let tap = unsafe { Tap::from_raw_fd(validate_raw_fd(tap_fd).map_err(Error::ValidateRawFd)?) .map_err(Error::CreateTapDevice)? }; - let dev = virtio::Net::from(tap).map_err(Error::NetDeviceNew)?; + let net_opts = virtio::NetOptions { + guest_mac: options.mac, + }; + + let dev = virtio::Net::with_tap(tap, net_opts).map_err(Error::NetDeviceNew)?; Ok(VirtioDeviceStub { dev: Box::new(dev), @@ -614,8 +620,8 @@ fn create_net_device( .map_err(Error::VhostNetDeviceNew)?; Box::new(dev) as Box<dyn VirtioDevice> } else { - let dev = - virtio::Net::<Tap>::new(host_ip, netmask, mac_address).map_err(Error::NetDeviceNew)?; + let dev = virtio::Net::<Tap>::new(host_ip, netmask, mac_address, Default::default()) + .map_err(Error::NetDeviceNew)?; Box::new(dev) as Box<dyn VirtioDevice> }; @@ -1006,8 +1012,8 @@ fn create_virtio_devices( devs.push(create_balloon_device(cfg, balloon_device_socket)?); // We checked above that if the IP is defined, then the netmask is, too. - for tap_fd in &cfg.tap_fd { - devs.push(create_tap_net_device(cfg, *tap_fd)?); + for (tap_fd, options) in &cfg.tap_fd { + devs.push(create_tap_net_device(cfg, *tap_fd, options)?); } if let (Some(host_ip), Some(netmask), Some(mac_address)) = diff --git a/src/main.rs b/src/main.rs index 3afca8e0..053af465 100644 --- a/src/main.rs +++ b/src/main.rs @@ -21,7 +21,8 @@ use arch::Pstore; use audio_streams::StreamEffect; use crosvm::{ argument::{self, print_help, set_arguments, Argument}, - linux, BindMount, Config, DiskOption, Executable, GidMap, SharedDir, TouchDeviceOption, + linux, BindMount, Config, DiskOption, Executable, GidMap, SharedDir, TapFdOptions, + TouchDeviceOption, }; #[cfg(feature = "gpu")] use devices::virtio::gpu::{GpuMode, GpuParameters}; @@ -1041,17 +1042,52 @@ fn set_argument(cfg: &mut Config, name: &str, value: Option<&str>) -> argument:: } "vhost-net" => cfg.vhost_net = true, "tap-fd" => { - cfg.tap_fd.push( - value - .unwrap() - .parse() - .map_err(|_| argument::Error::InvalidValue { - value: value.unwrap().to_owned(), - expected: String::from( - "this value for `tap-fd` must be an unsigned integer", - ), - })?, - ); + let mut components = value.unwrap().split(','); + + let fd: RawFd = components + .next() + .and_then(|x| x.parse().ok()) + .ok_or_else(|| argument::Error::InvalidValue { + value: value.unwrap().to_owned(), + expected: String::from("this value for `tap-fd` must be an unsigned integer"), + })?; + + let mut mac = None; + for c in components { + let mut kv = c.splitn(2, '='); + let (kind, value) = match (kv.next(), kv.next()) { + (Some(kind), Some(value)) => (kind, value), + _ => { + return Err(argument::Error::InvalidValue { + value: c.to_owned(), + expected: String::from("option must be of the form `kind=value`"), + }) + } + }; + match kind { + "mac" => { + mac = Some(value.parse().map_err(|_| argument::Error::InvalidValue { + value: value.to_owned(), + expected: String::from( + "`mac` needs to be in the form \"XX:XX:XX:XX:XX:XX\"", + ), + })?) + } + _ => { + return Err(argument::Error::InvalidValue { + value: kind.to_owned(), + expected: String::from("unrecognized option"), + }) + } + } + } + if cfg.tap_fd.contains_key(&fd) { + return Err(argument::Error::TooManyArguments(format!( + "TAP FD already used: '{}'", + name + ))); + } + cfg.tap_fd.insert(fd, TapFdOptions { mac }); } #[cfg(feature = "gpu")] "gpu" => { @@ -1295,8 +1331,8 @@ writeback=BOOL - Indicates whether the VM can use writeback caching (default: fa Argument::value("plugin-gid-map-file", "PATH", "Path to the file listing supplemental GIDs that should be mapped in plugin jail. Can be given more than once."), Argument::flag("vhost-net", "Use vhost for networking."), Argument::value("tap-fd", - "fd", - "File descriptor for configured tap device. A different virtual network card will be added each time this argument is given."), + "FD[,mac=MAC]", + "File descriptor for configured tap device. A different virtual network card will be added each time this argument is given. MAC is the MAC address that will be set in the guest."), #[cfg(feature = "gpu")] Argument::flag_or_value("gpu", "[width=INT,height=INT]", -- 2.27.0

2 3