Agreed (unfortunately). There is too much of an obsession in avoiding unsafe at all costs in my opinion. When you consider all C/C++ is effectively written in an unsafe block then it seems a bit ludicrous to assume you would never ever use unsafe in tight, performant code.
So you think. It doesn't matter how smart you think you are, statistically speaking you're not smarter than a compiler.