In practice it is a bit of a pain during the initial setup and package installation -- I was probably rebooting to recovery mode once a day or more to install some command line tool that I had forgotten about. Once the machine is configured then it is a more rare occurrence, although your usage might vary.
Something that I'm not happy about is that the snaps all live on the writable /var since they want to do automatic updates all the time. This is problematic for a locked-down configuration and might recommend against a snap based distribution.
Separating out the bootable bits from the rest of the packages might help, as would running more things in sandboxes. Another option that we're exploring is some lvm magic to create a snapshot, upgrade the snapshot, sign it, and then on the next reboot use it as the real root. This is also useful for fleet management -- the new root filesystem, kernel, initrd, etc can arrive "behind the scenes" and on the next reboot is the one that is used. Since the PCRs can be predicated as well, the PCR policy can be signed and sent along with the upgrade to make it seamless.
> Another option that we're exploring is some lvm magic to create a snapshot, upgrade the snapshot, sign it, and then on the next reboot use it as the real root.
This sounds very nice and similar to Android's A/B partitions!
> Since the PCRs can be predicated as well
This may be a stupid question but… what are PCRs? Google yields "polymerase chain reaction" – a method used, among others, for detecting the coronavirus but I'm sure that's not it. :)
Something that I'm not happy about is that the snaps all live on the writable /var since they want to do automatic updates all the time. This is problematic for a locked-down configuration and might recommend against a snap based distribution.
Separating out the bootable bits from the rest of the packages might help, as would running more things in sandboxes. Another option that we're exploring is some lvm magic to create a snapshot, upgrade the snapshot, sign it, and then on the next reboot use it as the real root. This is also useful for fleet management -- the new root filesystem, kernel, initrd, etc can arrive "behind the scenes" and on the next reboot is the one that is used. Since the PCRs can be predicated as well, the PCR policy can be signed and sent along with the upgrade to make it seamless.