I have a similar habit getting similar flak, but with another added reason. As a remote worker, going to a place every day (often leading to small interactions with others) helps replace replace some elements of working from an office (commute, small chats with other frequenters, the getting presentable ritual you mentioned, etc).
I'll append a gripe to this and say that employers should allow "work-from-home budgets" to be put towards credit at local coffee shops.
Pre-COVID this was my exact approach. I felt like there was a certain sense of randomness and adventure that was inspiring towards whatever I was working on.
Adding one more data point for your anecdote. These are the exact reasons I started smoking, and the same excuses I use to continue (along with the newfound addiction of course). It weighs heavily into my cost/benefit analysis for smoking despite all known and experienced bad things about it.
I never liked the socializing part, it was more some kind of FOMO with the smokers. The other ones stand. But the way it should be is that everyone gets a break, and that you can do smoking during the break or something else. Something like smoking a cigarette was something to hold in my hands and something to do when nervous. I know a better way to get some exercise: do a warming up, put on a podcast or music, and go run (cardio). You do two in one, and you don't have to go to a coffee shop (I find the idea of a coffee shop ridiculous, like a pub, because these goods are perfectly affordable and reachable at home but then again I'm not into the socializing aspect).
