One thing I rarely see discussed is the possibility that the brain is just too complex to practically reproduce. That is to say, that it is technically possible, but not practical. Evolution had billions of years, working in a massively parallel way to work this out after all. It's possible that the brain is a huge tangled mess of rules and special cases that we will never be able to fully understand and reproduce. Also, even if we are able to produce a basic intelligence, why do we assume it will ever get to the point where it can understand itself well enough to self improve? It's possible there is a threshold we won't be able to get past for self improvement to be possible.
It's possible that the brain is a huge tangled mess of rules and special cases that we will never be able to fully understand and reproduce.
It doesn't seem possible. Our brain, like the rest of our bodies, develops from a single cell. It's DNA (maybe also some other markers) that directs the differentiation of every part. So DNA is where the blueprints of that huge mess would be.
I believe that self-organization is a better way to explain how mind emerges. There's a basic try and error mechanism with strong rewards. The rest is the result of a life of learning.
> Evolution had billions of years, working in a massively parallel way to work this out after all.
And it also got to work on it with a hundred billion stars with earth-like planets, in each of a hundred billion galaxies, and (maybe) in a basically infinite space of universes. All in parallel with infinite time to play with.
Is it so far-fetched to postulate that humans will hit our limits before getting this far?
Great point, the anthropic principle dictates we only see the successful outcome. Evolution is working across the entire universe (or even multiverse) so we really don't know how long it could take for human (or greater) level intelligence to emerge.
