A perfect size doesn't exist for every human being. But the average thumb travel for adult males is 4 inches. For adult women it is 3.8. The width directly impacts usability for over 50% of the human population. Get over 3.5 inches and you're having to fidget with the phone to go side to side while holding the phone.

There are actual limits to width that have to be take into consideration. Just because you might have a hand large enough for a nexus, does not mean every human, or even adult male for that matter shares that trait. And you're guaranteed less women share that trait.