This seems to be another side of the problem that it gives code that doesn't actually compile for the target language. They should have a check for that, that should be the baseline.

For me, it sounds like it's closer to dumb copy paste than a smart code generator. AlphaZero wouldn't play a chess move that was against the rules.