I sympathize with your points, especially rebuilding the box from scratch.

But all the “passwordless” schemes I’ve seen support at least an additional “master key” which you can type in.

So if you’re ok with the security tradeoffs of passwordless tpm, it’s only an added convenience on top of your approach.