First: You are absolutely correct to note there a lot of grifters jumping on bandwagons. I've seen artists hate on AI that they see as ripping them off — and the example given by one artist I know personally, clearly looks like someone took one of their actual images, fed it into an img2img mode with a very small strength, and then tried to pass the result off as their own.
Second: Transformer models (and diffusion models) are merely the latest hotness in a long series of increasingly impressive AI models. There is no reason at all to assume either are the final possible model, not even the final word by OpenAI specifically.
Third: There is a direct correlation between the quality of output and the combination of training effort and example set size. This is why both image and text generators have improved significantly since this time last year.
Caveat 1: It may be that, as all the usual sources have responded to ChatGPT by locking down their APIs and saying "no" in robots.txt, they are already at the reasonable upper limit for training data, even though more data exists.
Caveat 2: Moore's Law is definitely slowing down, and current models are about (by Fermi estimation) 1000x less complex than our brains. Even though transistors are faster and smaller than synapses by the factor to which wolves are smaller than hills and faster than continental drift, the cost for a 1-byte-per-synapse model of a 6E14 synapse brain is huge. Assuming RAM prices of €1.80/GB (because that was the cheapest I found on Amazon today), that human-scale model would still cost in the order of a million Euros per instance. Will prices go down? I would neither bet for nor against it.
Will they (or anyone else in the next decade) create AGI? I think that's an argument in terms. Transformer models like the GPT models from OpenAI are very general, able to respond in any domain the training data covered. Do they count as "intelligent"? They can score well on IQ tests, but those are only a proxy for intelligence.
Given the biological analogy would be:
"Mad scientists take a ferret, genetically modify it to be immortal, wires up its nervous system so the only thing it experiences is a timeless sequence of tokens (from Reddit, Wikipedia, StackOverflow, and random fan-fic websites, but without ever giving the ferret any context as to what any of the tokens mean), and then spend 50,000 years rewarding/punishing it based on how well it imagines missing tokens, this is what you get."
I don't know what I was expecting, but it wasn't this.
Second: Transformer models (and diffusion models) are merely the latest hotness in a long series of increasingly impressive AI models. There is no reason at all to assume either are the final possible model, not even the final word by OpenAI specifically.
Third: There is a direct correlation between the quality of output and the combination of training effort and example set size. This is why both image and text generators have improved significantly since this time last year.
Caveat 1: It may be that, as all the usual sources have responded to ChatGPT by locking down their APIs and saying "no" in robots.txt, they are already at the reasonable upper limit for training data, even though more data exists.
Caveat 2: Moore's Law is definitely slowing down, and current models are about (by Fermi estimation) 1000x less complex than our brains. Even though transistors are faster and smaller than synapses by the factor to which wolves are smaller than hills and faster than continental drift, the cost for a 1-byte-per-synapse model of a 6E14 synapse brain is huge. Assuming RAM prices of €1.80/GB (because that was the cheapest I found on Amazon today), that human-scale model would still cost in the order of a million Euros per instance. Will prices go down? I would neither bet for nor against it.
Will they (or anyone else in the next decade) create AGI? I think that's an argument in terms. Transformer models like the GPT models from OpenAI are very general, able to respond in any domain the training data covered. Do they count as "intelligent"? They can score well on IQ tests, but those are only a proxy for intelligence.
Given the biological analogy would be:
"Mad scientists take a ferret, genetically modify it to be immortal, wires up its nervous system so the only thing it experiences is a timeless sequence of tokens (from Reddit, Wikipedia, StackOverflow, and random fan-fic websites, but without ever giving the ferret any context as to what any of the tokens mean), and then spend 50,000 years rewarding/punishing it based on how well it imagines missing tokens, this is what you get."
I don't know what I was expecting, but it wasn't this.