I do think that there is also a "bitter lesson" in optimization: simple optimizations are usually more effective, robust and powerful than complicated ones. Also, optimization becomes progressively harder as the system trends to optimal performance, which is likely to cancel out the advantage of higher intelligence in finding them.

Also, even to the extent model architecture can be optimized, in the short term, thanks to the bitter lesson, the bottleneck seems to be less “human intelligence coming up with new ideas for optimizations” and more “compute time to test ideas on sufficiently large models”. An AI that followed the same basic thought process as humans, even if its ideas were a little better than humans’ on average, probably wouldn’t help that much.

On the other hand, an AI that in some sense ‘understands’ its own structure, which is to us almost entirely a black box, might be able to do much better. Consider OpenAI’s attempt to make GPT-4 explain the purpose of neurons in GPT-2; it basically didn’t work at all, but it would have opened a huge new frontier of research if it had.

But that understanding ability is hard to train, given the lack of any meaningful training data. If it does show up, it could act as an accelerant. But it could just as easily not show up, at least not until AI is already comfortably superhuman in the “thinking like a human” department – in which case it won’t help us reach that point in the first place.

> On the other hand, an AI that in some sense ‘understands’ its own structure, which is to us almost entirely a black box, might be able to do much better.

True... but is this a plausible capability? I don't mean in practice. I mean theoretically.

I feel we often forget about the free lunch theorem when it comes to intelligence. To us it is this magical, omnipotent thing, such that a sufficiently advanced intelligence can understand anything. But mathematically, there is no such thing as an algorithm that is better than another at everything: whatever intelligence is, it has to be intrinsically bad at certain things.

Which brings me to this thought: I think it is highly plausible that understanding one's own internal structure intimately enough to reorganize and optimize it is in fact one of these things Intelligence (with a big I: all levels of intelligence) is intrinsically bad at, both directly and indirectly. What makes me think it is plausible is that intelligence is mainly about manipulating abstractions, but reality is far too messy to be abstracted... unless you simplify it. And so it seems to me that what makes humanity incredibly powerful is the one-two punch of morseling reality into simple pieces that match our abstractions, and building with these pieces. In other words, intelligence is only effective over an ablated version of reality, and we happen to be very good at ablating... reality itself. But if the thing you are trying to understand is truly complex, you can't really ablate it to a point where it is simple enough to be understood, and basically every single level of intelligence will fail at understanding it.

There has to be some kind of tradeoff to intelligence's uncanny ability to manipulate and understand systems, and I think this is where it is. I very well could be wrong, of course.

AI risk argument is that human brain "hardware" is suboptimal as it relies on relatively slow electrochemical communication between neurons. Even with same algorithms, performance of digital intelligence will be orders of magnitude higher simply because of speed and memory capacity. Evolution rarely achieves optimal performance (optimal within constrains of physics). Evolution is iterative and tends to find local maxima. Think how we can build construction equipment thousands times stronger than any animal's muscle. And for AGI to improve itself — it's not necessary to be able to model itself with 100% accuracy. It only needs to understand itself at a high level, kinda like the way human understands its own body. Software engineers work on systems they don't completely understand all the time. E.g. AGI may notice inefficient memory use in its own implementation. It will try to fix it and run copy of itself. If it finds it indeed improves performance it will "upgrade" itself to new version, maybe just keeping one instance of previous version as a backup just in case. Now new version is smarter and is able to come up with even better optimizations. Eventually it will exhaust all possible software optimizations and will need to build new hardware to keep improving. How it will do it? Use your creativity. There are many ways. With ability to build custom hardware the only limit is now physics. It will keep improving and getting smarter until it reaches physical limits of computation. At this point it's billions time smarter than human.

> Think how we can build construction equipment thousands times stronger than any animal's muscle.

And yet the mantis shrimp can throw a punch faster than a bullet. Nature does not optimize for the same things we do. Animals did not evolve muscles as strong as some of our construction equipment, but it is hard to know whether this is because evolution is limited, or because that capability was not helpful enough to compensate for the extra fuel requirements.

> Even with same algorithms, performance of digital intelligence will be orders of magnitude higher simply because of speed and memory capacity.

Maybe. But I think you (and others) have misplaced confidence in the superiority of our technology.

Power consumption of a circuit is proportional to frequency. We have computing units that can work at GHz rates, sure, but you can't exactly cram 80 billions of these into a cubic foot unless you're planning to melt them into plasma. So we don't. Current approaches must multiplex millions of "neurons" through the same computing units, leading to effective firing rates that are not wildly different from biological neurons. If we were to switch to a different approach where each "neuron" is independent, I expect the communication speed to plummet simply because of practical considerations about energy consumption and heat dissipation.

Beyond that, I would argue that our paradigm of hardware design rests on many intentional decisions that are structurally suboptimal for intelligence, and that this is a graver problem than the brain's own inefficiencies. Take the ability that you mention to read one's own implementation and back up one's brain. This is intentional in our design. But this is not free! Unfettered read access to one's brain is a heavy internal communication requirement when you compare it to simple local communication between units, and it is naive to think it would not have a negative impact on performance. AI that can copy itself has a lower performance ceiling than a system that does not have this requirement. That's physics.

In any case, the point I'm trying to make is not about humans vs AI, it is about intelligence in general. I suspect that intelligence is excellent at simple optimization, but that it is not very good at complex optimization, and thus for any intelligence there may be a point where it starts to underperform relative to e.g. evolution.