Agreed (unfortunately). There is too much of an obsession in avoiding unsafe at all costs in my opinion. When you consider all C/C++ is effectively written in an unsafe block then it seems a bit ludicrous to assume you would never ever use unsafe in tight, performant code.
