Your point about choosing the things that matter is the pertinent sentence. Apple doesn't want to put any focus on game, as media streaming is the focus of the device.
Consider the possibilities:
1) Apple implements amazing support for games on the Apple TV. Consumers who want to play AAA games on their TV won't buy it - they'll get a traditional console instead. Consumers who play the kinds of games that the iPhone/iPad/iPod support also won't be drawn to it, as they've already got their iPod/iPad/iPhone, so why would they want to play the game using a worse interface, slower, and when they could be watching TV.
2) Apple implements mediocre support for games on the Apple TV. This (like I said) will muddy their brand, as well as possibly taking personnel away from the core focus of the device - streaming, and again muddying their brand.
Either way, it's really not worth it for Apple. In the first case, the fraction of consumers they attract is really so tiny that it's not worth it for them to implement gaming. In the second case it's a net loss for them, both in terms of effort/payoff, and in terms of damage to the brand.
Apple could have made a huge impact in the games market. Their knack for polishing their products to the finest detail is exactly what games need these days.
At $499 that's squarely in AAA-console pricing territory, and Apple is not geared up to fight the PS4 and Xbox One... nor do they particularly want to.
I think there is a miscomprehension of the iOS gaming market, or casual gaming on mobiles in general. People don't play Angry Birds or Temple Run because they are fans of the games (in the same way you'd play Halo or Civilization), they do it to kill time.
Mobile games are intentional time-wasters for when you're standing in line, riding a bus, sitting in the doctor's office, etc. This is diametrically opposite to AAA-console gaming where users actively seek it out as something to do and dedicate time towards it. Platforms like the PS Vita straddle this weird middle ground - and it's arguable that Apple could potentially capture this middle ground also.
Either way, an AppleTV/Mac Mini fusion device wouldn't.
Consider the possibilities:
1) Apple implements amazing support for games on the Apple TV. Consumers who want to play AAA games on their TV won't buy it - they'll get a traditional console instead. Consumers who play the kinds of games that the iPhone/iPad/iPod support also won't be drawn to it, as they've already got their iPod/iPad/iPhone, so why would they want to play the game using a worse interface, slower, and when they could be watching TV.
2) Apple implements mediocre support for games on the Apple TV. This (like I said) will muddy their brand, as well as possibly taking personnel away from the core focus of the device - streaming, and again muddying their brand.
Either way, it's really not worth it for Apple. In the first case, the fraction of consumers they attract is really so tiny that it's not worth it for them to implement gaming. In the second case it's a net loss for them, both in terms of effort/payoff, and in terms of damage to the brand.