Yes, the changes introduced by Godot 4 were a prime motivator for this project.
However, it is not quite as clear cut as OpenAI's models simply being trained on Godot 3.x projects only. Not only do they sometimes produce valid 4.x syntax (gpt-4 more often than 3.5-turbo), indicating there were at least some 4.x projects in the training data, they also hallucinate other invalid syntax, such as Python-specific functionality, or simply non-existent methods.
I do think evaluating against Godot 3.x would increase their scores somewhat, but i have not had time to do so yet.
However, it is not quite as clear cut as OpenAI's models simply being trained on Godot 3.x projects only. Not only do they sometimes produce valid 4.x syntax (gpt-4 more often than 3.5-turbo), indicating there were at least some 4.x projects in the training data, they also hallucinate other invalid syntax, such as Python-specific functionality, or simply non-existent methods.
I do think evaluating against Godot 3.x would increase their scores somewhat, but i have not had time to do so yet.