Time spent twiddling with the repo is time saved in the future debugging or writing documentation.

You can have large refactoring PRs for which splitting them further really makes little sense(*), but that have a huge risk of introducing regressions. Being able to bisect them is much easier with a properly maintained repository.

(*) And then you spend time twiddling with GitHub, which is the same as twiddling with the repo except with worse tools.