(not OP) It's been a while, but if I recall correctly NB didn't address the technical limitations of Turing machines, e.g. the halting problem. How is a machine supposed to make itself smarter when it can't predict that it won't just crash after a code modification? Or just hack its motivation function (wireheading). The papers I've seen on the latter problem (years ago) start by assuming that the halting problem has been solved, essentially, by giving the agent non-deterministic computational powers. Biological intelligence evolved, so it's perhaps more realistic to imagine an ecosystem of computational agents competing in a trial-and-error race, but that makes the whole thing vulnerable to any threat to that superstructure--much more fragile than AI-go-FOOM.
I think AI doomers are mostly dumb, but this argument is not very good. The Halting Problem is a technical fact: you can't _prove_ that a program terminates in general. This doesn't mean that all programs are ambiguous with respect to termination. In fact, with appropriate limitations imposed, many programs could be proven to terminate. And even without limitations humans wrangle code all the time. If the presumption that a machine intelligence is at least as smart as a person, I don't see why they'd be any more likely than we are to run into non-terminating programs or whatever.
Furthermore, it doesn't matter for both practical and philosophical reasons. Practically, I don't see why a super-intelligence would just shut itself down to run some new code without various kinds of testing procedures. Also, there aren't any Turing machines anyway, since they need infinite memory, which even a super-intelligence doesn't have. I don't really see how the halting problem is a material problem for a super smart agent.
It's not just crashing/halting. See Rice's Theorem. The machine can't predict its own future behavior in most ways that are important.
I'm not an expert in this stuff, and my point was that a serious treatment of super-intelligence should address such limitations to computations. In particular the "super" part seems to imply solving exponential-complexity problems in linear time. I remember looking in NB's book and not finding it.
I don't see why that is the case. I'm much more intelligent than a racoon and I have yet to grapple with Rice's Theorem. There is no reason I can think of to believe that Rice's Theorem is a serious constraint on intelligence beyond my own. In general, such an agent isn't particularly interested in proving facts about its own program (at least I can't see why it would care any more than we are interested in proving mathematical properties of our own brains). It is interested in maximizing some objective function, which can transparently be done without thinking much at all about Rice's Theorem (all systems which train neural networks and indeed, even simpler optimization problems, pursue such maximization with nary a thought towards Turing or Rice's Theorems).
> The papers I've seen on the latter problem (years ago) start by assuming that the halting problem has been solved, essentially, by giving the agent non-deterministic computational powers.
Of course this is not required. An AI system can simply not implement optimizations that it can't prove are correct, per the above link. Alternately, if simply "crashing" is the issue then it could simply register a fault handler that reverts to its previous code.
The notion that these would be any kind of impediment is completely bizarre to me. As a lowly human I already know how to handle these problems, and we're talking here about a superintelligence more capable than either of us.
Some FOOM scenarios are clearly unphysical, but the idea of recursive self-improvement being impossible or infeasible is not one of those reasons.
How far advanced do you think an AI would be where we could say to GPTx “here is your source code - write us a much more powerful AI than yourself”? How far off would you say this was?
It's impossible to say with certainty. I suspect there are at least one or two generalization tricks needed, but that's only speculative. Those generalizations might be simple or a little more complex, and so might take a year or two, or decades to discover. I can only say that they will almost certainly be discovered within my lifetime (say within 40 years). I suspect it will be much sooner than that.
