The problems begin with this concept of a "provably friendly architecture", which probably doesn't exist. Heck, humans aren't exactly provably friendly either.
But the main argument against this design goal of being somehow provable and intelligent at the same time is that we have fundamentally conflicting paradigms:
The first is the idea that in order to build an artificial intelligence, we'd need to build an actual artificial person. Complete with their own internal representation of the world (or as you might say: its own hopes and dreams). It's an approach to AI that is fundamentally uncertain in the way individual AI entities will turn out personality-wise, but it is guaranteed to work. We know this approach will work, because we are ourselves machines built on this principle.
==> good chance of success, somewhat limited danger
Then, conflicting with that comes this notion of a provably friendly/unfriendly design which sounds like it was made up by CS theorists who think intelligence is a function of raw processing power and thought patterns are in any way related to rigid formulas. They're very likely wrong, but luckily that also means this group of researchers will never produce anything dangerous except maybe a lot of whitepapers.
==> virtually no chance of happening, no danger
I do agree though that there might be a third kind of AI, a sort of wildly self-improving problem-solving algorithm that has no real consciousness and simply goes on an optimization rampage through the world. This would be a disaster of possibly grey goo-like proportions. BUT, this approach to AI is also very likely to be used in tools with only limited autonomy. And because the capability of an unconscious non-entity AI to understand the world is limited, the probability of it taking over the world also seems limited.
==> small probability of autonomous takeoff, but if it happens it will be the end of everything
> a sort of wildly self-improving problem-solving algorithm that has no real consciousness and simply goes on an optimization rampage through the world
> I meant the human species as a collective has no single consciousness.
Neither has an AI species, but that's not the issue. The point being made here was that a danger could arise from a very efficient and powerful automaton that has neither self awareness nor recognizes other beings with minds as relevant. From that I argued the threat of it happening is actually low because by its nature this kind of AI would probably lack the means to instigate an autonomous takeover of our planet.
> And also that individual humans do not have metaphysical consciousness.
Ah, I finally see where our misunderstanding comes from. Science doesn't talk about consciousness (or metaphysics) in the spiritual sense. The question whether people have metaphysical consciousness or not really depends on your definition of those terms, so arguing "for" or "against" isn't really gonna do anything besides getting you karma for oneliners.
As far as practical AI research is concerned, the definition of consciousness is the same for humans and non-humans and while there are different degrees of consciousness possible, there certainly is an agreement that the average human has one.
But the main argument against this design goal of being somehow provable and intelligent at the same time is that we have fundamentally conflicting paradigms:
The first is the idea that in order to build an artificial intelligence, we'd need to build an actual artificial person. Complete with their own internal representation of the world (or as you might say: its own hopes and dreams). It's an approach to AI that is fundamentally uncertain in the way individual AI entities will turn out personality-wise, but it is guaranteed to work. We know this approach will work, because we are ourselves machines built on this principle. ==> good chance of success, somewhat limited danger
Then, conflicting with that comes this notion of a provably friendly/unfriendly design which sounds like it was made up by CS theorists who think intelligence is a function of raw processing power and thought patterns are in any way related to rigid formulas. They're very likely wrong, but luckily that also means this group of researchers will never produce anything dangerous except maybe a lot of whitepapers. ==> virtually no chance of happening, no danger
I do agree though that there might be a third kind of AI, a sort of wildly self-improving problem-solving algorithm that has no real consciousness and simply goes on an optimization rampage through the world. This would be a disaster of possibly grey goo-like proportions. BUT, this approach to AI is also very likely to be used in tools with only limited autonomy. And because the capability of an unconscious non-entity AI to understand the world is limited, the probability of it taking over the world also seems limited. ==> small probability of autonomous takeoff, but if it happens it will be the end of everything