You're correct, but I've personally experienced the kind of referred pain you're describing, for both (at different times) prostate and kidneys, and doctors (and now me!) are pretty able to identify that and treat it.