On Wed, Aug 17, 2022 at 04:25:20PM +0300, Ville Ilvonen wrote:
As of now, the spectrum fork for aarch64 just adds another config after rpi configs and replaces the default config to use that to build. With small changes this could be handled like rpi configs. In addition, cloud-hypervisor accepts kernel only in EFI format for aarch64[1]. Anyway, this would allow us to build an aarch64 Spectrum installer - even make it with a more generic kernel. That takes us to ARM vendor/device specific HW quirks which would need to be handled anyway. I'll intentionally leave device specific kernel hardening and disabling kernel module loading for security reasons for now. As of now the vendor/device specifics are not supported unless one builds device specific Spectrum image with all configs build-time and skips installer altogether.
The other option that I see. We discussed earlier nix-hardware and device specific modules. That would bring nixos configuration.nix and installation supporting scripts to Spectrum, though. Those could be called from the Spectrum installer but it would change the installer logic from writing an image to dynamically configuring the device during install based on user selections.
I don't think the full NixOS module system, with rebuilds, etc. belongs in Spectrum. Being able to treat images as immutable makes it easier to provide various strong security guarantees. But not wanting to
This was and still is one important design decision to build on Spectrum. Regardless, it makes development iterations on target HW more challenging than needed. Conceptually we've had discussions on separating concerns between "development system - writable, easily updatable" and "production system - immutable, updated as image". Latter could have more hardening, security policies etc. enabled which makes development more difficult by design. In practice, some developers have remounted the Spectrum file system as writable to make development iterations easier. In many cases, the development must be done on the target HW which brings us back to the need for the "development system" configuration. Update image iteration cycle is too slow.
Yeah, I agree something like this would be good. Especially when testing on hardware as you say. I would like to think more about exactly how this should work. Do you think that, if you it were possible to develop Spectrum on Spectrum, it would be acceptable to have to reboot into a new configuration if the host system was changed? (Assume that the process of actually building the new system is fast — the reboot would be the main overhead.)
integrate the full module system doesn't prevent us taking advantage of nixos-hardware. It's possible to evaluate NixOS modules standalone in a Spectrum build, in fact we already do that to reuse NixOS's list of all redistributable firmware packages[3]. We could do a similar thing to extract the kernel that nixos-hardware configures for a particular device, something like this:
inherit (nixos { configuration = [ <nixos-hardware/pine64/pinebook-pro/default.nix> ]; }.config.boot.kernelPackages) kernel;
And naturally which device that's pulling from should be configurable — we'll want to have a config file somewhere, just not a full NixOS one.
This made me propose nix-hardware usage more and think if we could have what I called "development configuration". In essence, nix-hardware is NixOS channel and we could have a custom channel to support development as well (e.g. dev git repo(s)).
You mean set it to your own, custom version of nixos-hardware that included WIP support for the board you were working on? Yeah, that wouldn't be a problem at all.
In the medium term, I'd like to decouple nixos-hardware's custom kernel packages from NixOS configurations. But that would require somebody finding the time to sit down and make the change, and also convince other nixos-hardware users that it's the way to go. I don't think it would be a problem though, especially if it meant nixos-hardware getting more active maintenance, which it's lacking at the moment because it's not too well advertised and so not enough people are using it.
Ideally yes and I hope we could contribute to that effort. However, we need to focus on getting Spectrum running on aarch64 with imx8 for now. For that I'm reading that nixos-hardware approach is preferred.
Yeah, it's definitely the way to go. And if we can make nixos-hardware better in future, that would just be further progress on top of integrating nixos-hardware as described above.