You are right, but Google search is a terrible way to make the comparison. They are returning all websites that are vaguely related to the phase, not doing a word search.
No, I'm doing a quoted phrase search, not a vague vector similarity search. Turning on verbatim mode doesn't change the results much (13 times more instead of 15).
My feeling has been that "brutally simple" evokes the meaning of Brutalist architecture, it is a phrase applied to something that is simple, honest, and functional.
We do say "elegantly simple". According to Google search result counts, "elegantly simple" is used about 15 times more often than "brutally simple".