It's a version of the tragedy of the commons. Yes, your software bloat isn't the main factor keeping you slow, and cleaning it up might be barely perceptible.
But... all the other software makes the same decision. Then, suddenly, everything on the computer is running at 10% efficiency.
In a complex, multitasking environment, keeping your code clean benefits other tasks as much or more than your own. It should be considered a responsibility.
I would agree with the sentiment but not necessarily the conclusion.
Like, don't use (the equivalent of) Stooge Sort (for your particular situation).
But unless you are in a very particular situation, it should be OK for everyone to "just use your language's built-in sort function" (hopefully that's a thing and you don't use something where you need to roll your own even in 2024) assuming that it uses a sane default like quicksort or merge sort that will work perfectly fine for most regular common situations.
Another example might be to not stupidly build slow algorithms saying "this won't see much data anyhow" (yes, I've seen that in PRs unfortunately) when a very simple hash lookup will ensure it's fast all the time. But it should be totally fine to assume that your language's hash table implementation is fine for common situations and you don't need to optimize anything unless you're a very special case.
But... all the other software makes the same decision. Then, suddenly, everything on the computer is running at 10% efficiency.
In a complex, multitasking environment, keeping your code clean benefits other tasks as much or more than your own. It should be considered a responsibility.