The problem is that so-called AI isn't just a neutral tool for generating and exploring ideas in the whole idea space, nor is it a particular human being with particular experiences and talents and biases and ideas and a particular personality that can generate something interesting and unique, large language models always give you essentially the average of the entire Corpus of human output given whatever input you gave it. So they are always guaranteed to give you the most lukewarm, generic, uncreative ideas and output imaginable. Just try to get it to write something arguing against a widely held position or for a unique position. You're very quickly find that you can't really get it to do that, because that's fundamentally opposed to how it's reward function works so of course it isn't going to do that. This is because the problem stretches even further than just affecting what kinds of ideas large language models can produce on their own, too, because if you are asking it to expand on and process an idea you've given it, that processing itself will be subject to reverting to the mean, and so even if you have a human inputting unique and interesting ideas into the large language model, all you will get out is eventually more generic sludge with all of the interesting aspects scrubbed out. So ultimately everything created with LLMs will be lowest common denominator populist sludge with no interesting ideas or defining characteristics. And your fundamentally not going to be able to solve that until you come up with a completely different method for doing this with an entirely different reward function, with the problem being that the only quantitative, data based way of determining whether an output is good is to compare it to what's common in the Corpus of human output, since otherwise you get very quickly into very complex philosophical questions of Truth and value and logic that just aren't legible to a computer system.