1. paperclip optimizers where a very smart computer you tell to do one menial task like producing as many paperclips as possible or proving a mathematical theorem can turn into a catastrophy as that computer turns all iron on earth into paperclips or into computers that all try to find a solution to the theorem. This also includes computers that we task to "protect" humanity coming to the conclusion that humans having power to kill each other is mankind's biggest threat.
2. crazy would-be dictator who wants to rule over the world and tells an AI to do it or kill all humans or something else.
TLDR: First way: forgetting machines to tell to not kill humans (or not doing it in an effective manner). Second way: some really shit individual explicitly telling machines to kill humans.
The first danger is one we already face: basically since we've had machines there have been accidents with them, also ones involving casualties. In general, the more we care about avoiding casualties the less likely they are. However, it only takes one super intelligent paperclip optimizer to "break lose" so given the high amount of possible casualties, there needs to be a lot of care taken to prevent even one such event.
The second danger needs to be coped as well. One could do two things: very slow deployment of super-AI capabilities at the start, while building AIs that can defend governments and somehow encoding into them how the government works (to prevent parts of the government from using that machine in a coup). The same computers will prevent revolutions though, so I guess we'll see less and less of those. You can think of variations of those ideas like AIs that only enforce asimov's laws or only make sure that we don't use any weapons more powerful than $weapon on each other.
What I don't understand though is how neuralink will help with coping with those threats.
1. Unplug the paperclip optimizer. Blow it up. The problem with the less wrong idea is they keep ascribing more and more godlike powers to AI to counter very common objections to technology. Somehow the entire thing becomes a godzilla like self-sustaining organism that ignores anything we can do or throw at it, and has magical powers. Meanwhile it seems apparently tha major websites can have outages if people go on summer vacation and the interns are on duty.
2. They can do that now. What would an AI do differently that couldn't be accomplished by conventional weapons? How would it do so without using said weapons or any sort of thing that could be done so without it?
The AI thing is just a secular form of the rapture, a particular variant of existential dread for people with little to no religious belief.
> Somehow the entire thing becomes a godzilla like self-sustaining organism that ignores anything we can do or throw at it, and has magical powers. Meanwhile it seems apparently tha major websites can have outages if people go on summer vacation and the interns are on duty.
Sure, the risk is low right now, but the more powerful computers we can build, the larger the potential risk is. Before you manage to press the off button the computer might already have deployed a bioagent or killed countless lives with drones.
> They can do that now. What would an AI do differently that couldn't be accomplished by conventional weapons?
A military made out of humans is subject to human failings. It is generally a big problem that soilders shoot in the general direction of the enemy to not get punishments for not shooting but miss on purpose. As an extreme example, the nazis had to give lots of free alcohol to their soilders so that they'd continue shooting civilians and burying them under new bodies before they have even died. They later invented gas chambers as an easier method to kill masses of people. Compared to humans, an AI is doing what it is being told to. If you tell it "Kill all humans" it will do it.
1. paperclip optimizers where a very smart computer you tell to do one menial task like producing as many paperclips as possible or proving a mathematical theorem can turn into a catastrophy as that computer turns all iron on earth into paperclips or into computers that all try to find a solution to the theorem. This also includes computers that we task to "protect" humanity coming to the conclusion that humans having power to kill each other is mankind's biggest threat.
2. crazy would-be dictator who wants to rule over the world and tells an AI to do it or kill all humans or something else.
TLDR: First way: forgetting machines to tell to not kill humans (or not doing it in an effective manner). Second way: some really shit individual explicitly telling machines to kill humans.
The first danger is one we already face: basically since we've had machines there have been accidents with them, also ones involving casualties. In general, the more we care about avoiding casualties the less likely they are. However, it only takes one super intelligent paperclip optimizer to "break lose" so given the high amount of possible casualties, there needs to be a lot of care taken to prevent even one such event.
The second danger needs to be coped as well. One could do two things: very slow deployment of super-AI capabilities at the start, while building AIs that can defend governments and somehow encoding into them how the government works (to prevent parts of the government from using that machine in a coup). The same computers will prevent revolutions though, so I guess we'll see less and less of those. You can think of variations of those ideas like AIs that only enforce asimov's laws or only make sure that we don't use any weapons more powerful than $weapon on each other.
What I don't understand though is how neuralink will help with coping with those threats.