People can blame their inability to have a relationship on outside factors (age/genetic makeup) if physical attractiveness was set in stone and the most important thing. It's a worldview where they don't have to look at their personality or behavior critically, so its an incredibly appealing worldview to some.