[EDITED: the intended quote is below. the quote above is the next paragraph of OP, which is only slightly less relevant than the intended one]

'Why should we expect this to happen? Multiple reasons. The first is that it happened before. It took evolution twenty million years to go from cows with sharp horns to hominids with sharp spears; it took only a few tens of thousands of years to go from hominids with sharp spears to moderns with nuclear weapons. Almost all of the practically interesting differences in intelligence occur within a tiny window that you could blink and miss.'

Yudkowsky's position paper on this idea explains this in more detail: http://intelligence.org/files/IEM.pdf

Bostrom, Yudkowsky, etc. posit that an "artificial super-intelligence" will be many times smarter than humans, and will represent a threat somewhat analogous to an atomic weapon. BUT... consider that the phrase "many times smarter than humans" may not even mean anything. Of course we don't know one way or the other, but it seems to me that it's possible that we're already roughly as intelligent as it's possible to be. Or close enough that being "smarter than human" does not represent anything analogous to an atomic bomb.

So this might be an interesting topic for research, or at least for the philosophers: "What's the limit of how 'smart' it's possible to be"? It may be that there's no possible way to determine that (you don't know what you don't know and all that) but if there is, it might be enlightening.

I think Nick Bostrom had the perfect reply to that in Superintelligence: Paths, Dangers, Strategies:

> Far from being the smartest possible biological species, we are probably better thought of as the stupidest possible biological species capable of starting a technological civilization—a niche we filled because we got there first, not because we are in any sense optimally adapted to it.

It would be extremely strange if we were near the smartest possible minds. Just look at the evidence: Our fastest neurons send signals at 0.0000004c. Our working memory is smaller than a chimp's.[1] We need pencil and paper to do basic arithmetic. These are not attributes of the pinnacle of possible intelligences.

Even if you think it's likely that we are near the smartest possible minds, consider the consequences of being wrong: The AI becomes much smarter than us and potentially destroys everyone and everything we care about. Unless you are supremely confident in humanity's intelligence, you should be concerned about AI risk.

1. https://www.youtube.com/watch?v=zsXP8qeFF6A

The humans who are in charge of our current society by virtue of having the most wealth, political power, or popularity aren't necessarily the smartest or quickest thinkers, at least not in the way that most AGI researchers seem to be targeting. So even if someone manages to build a real AGI there's no reason to expect it will end up ruling us.

There are also geniuses who can do amazing feats of mental arithmetic and have no severe mental disabilities in any other areas. There are people who have perfect memories (and usually, they live totally normal lives but for the inability to forget a single event). There are people who are far better than average at recognizing faces, pattern recognition, and other tasks that are non-conscious.

The notion that there are "fundamental limits" that are for some reason near the typical human is the cognitive bias of the just-world fallacy. The world is not just. Bad things happen to good people. Good things happen to bad people. If there is a fundamental limit to how smart a mind can get, it's very, very far above the typical human, because there are exceptional humans that are that far away and there's no reason to think a computer couldn't beat them either. Deep Blue beat Kasparov.

>The humans who are in charge of our current society by virtue of having the most wealth, political power, or popularity aren't necessarily the smartest or quickest thinkers, at least not in the way that most AGI researchers seem to be targeting.

I'm not at all familiar with AGI research, but while there are people who start out ahead, there's no reason to think that you can't actually play the game and win it. Winning challenges of acquiring wealth or political power or popularity are related to intelligence, insofar as the concept of intelligence is defined (as it is at least in psychology) as the ability to accomplish general goals.

Walking into a meeting of the Deutsche Arbeiterpartei, joining as the 55th member, and later seizing control of the state requires intelligence. Landing in a dingy yacht with 81 men, bleeding a regime to death, and ruling it until your death requires intelligence. Buying the rights to the Quick and Dirty Operating System, licensing it to IBM, and becoming the de facto standard OS for all consumer computing requires intelligence. Presenting yourself as a folksy Texan when you're a private-school elite from Connecticut and convincing the electorate that you'd have a beer with them requires intelligence. All of these outcomes are goals, and accomplishing them demonstrates an ability to actually put the rubber to the road.

You're confusing the technical term of intelligence, which means something along the lines of "ability to accomplish goals," with the social construct of "intelligence" which means something along the lines of "ability to impress people." Intelligence is not always impressive, and humans have a fantastic ability to write off the accomplishments of others when they want to make the world appear more just. I mean, nobody feels good about the fact that we're one Napoleon away from a totally different world system that may or may not suit our interests. The belief that the world is somehow fundamentally different to the extent that the next Napoleon-class intelligence who isn't content managing a hedge fund and just increasing an int in a bank database can't actually redraw the world map, is just an illusion we tell ourselves to make the world we live in more fair, the stories we live out more meaningful, and the events of our lives have a little bit more importance.

AI could change things very, very drastically. It's right to be afraid.

And do they seem to hold any sort of significant power? It seems that intelligence is not a very good means for achieving power. Charm seems much more effective, and therefore dangerous. I'd be afraid of super-human charm much more than super-human intelligence.

Mental disabilities aside, very high IQ seems to be correlated with relatively low charm and low ability to solve particular problems (how to get people to do what you want) that are far more dangerous than the kind of problem-solving intelligence (or how we commonly define it) is capable of solving. "Super" intelligent people are terrible problem-solvers when the problems involve other humans.

Ironically, the fact that people like me view the AI-scare as a religious apocalypse that is as threatening as any other religious apocalypse implies one of two things: 1) that the people promoting the reality of this apocalypse are not as intelligent as they believe themselves to be (a real possibility given their limited understanding of both intelligence and our real achievements in the field of AI) and/or that 2) intelligent people are terrible at convincing others, and so don't pose much of a risk.

Either possiblity shows that super-human AI is a non-issue, certainly not at this point in time. As someone said (I don't remember who), we might as well worry about over-population on mars.

What's worse is that machine learning poses other, much more serious and much more imminent threats than super-human intelligence, such as learned biases, which are just one example of conservative feedback-loops (the more we rely on data and shape our actions accordingly, the more the present dynamics reflected in the data ensure that they don't change).

See these paragraphs in the post to which you replied:

>Walking into a meeting of the Deutsche Arbeiterpartei, joining as the 55th member, and later seizing control of the state requires intelligence. Landing in a dingy yacht with 81 men, bleeding a regime to death, and ruling it until your death requires intelligence. Buying the rights to the Quick and Dirty Operating System, licensing it to IBM, and becoming the de facto standard OS for all consumer computing requires intelligence. Presenting yourself as a folksy Texan when you're a private-school elite from Connecticut and convincing the electorate that you'd have a beer with them requires intelligence. All of these outcomes are goals, and accomplishing them demonstrates an ability to actually put the rubber to the road.

>You're confusing the technical term of intelligence, which means something along the lines of "ability to accomplish goals," with the social construct of "intelligence" which means something along the lines of "ability to impress people." Intelligence is not always impressive, and humans have a fantastic ability to write off the accomplishments of others when they want to make the world appear more just. I mean, nobody feels good about the fact that we're one Napoleon away from a totally different world system that may or may not suit our interests. The belief that the world is somehow fundamentally different to the extent that the next Napoleon-class intelligence who isn't content managing a hedge fund and just increasing an int in a bank database can't actually redraw the world map, is just an illusion we tell ourselves to make the world we live in more fair, the stories we live out more meaningful, and the events of our lives have a little bit more importance.

>Ironically, the fact that people like me view the AI-scare as a religious apocalypse that is as threatening as any other religious apocalypse implies one of two things:

Some alternative explanations:

3. You're wrong, and super-human AI is a massive issue, regardless of how you pattern-match it as "religious"

4. You're not creative enough (one might dare say "intelligent" but that would be too snarky) to imagine all the ways in which an AI could devastate humanity without it having much intelligence or much charm

5. Your view of the world is factually incorrect, I mean you believe things like:

>Mental disabilities aside, very high IQ seems to be correlated with relatively low charm and low ability to solve particular problems (how to get people to do what you want) that are far more dangerous than the kind of problem-solving intelligence (or how we commonly define it) is capable of solving. "Super" intelligent people are terrible problem-solvers when the problems involve other humans.

Let's assume that IQ is a good proxy for intelligence (it isn't): what IQ do you think Bill Gates or Napoleon or Warren Buffet or Karl Rove have? What IQ do you think Steve Jobs or Steve Ballmer had/have? Do you think they're just "average" or just not "very high"?

This:

>very high IQ seems to be correlated with relatively low charm

is again the just world fallacy! There is no law of the universe that makes people very good at abstract problem solving bad at social situations. In fact, most hyper-successful people are almost certainly good at both.

And that ignores the fact that cognitive biases DO exist, and it's very possible to apply the scientific method and empirical problem solving to finding them, and then exploiting humans that way. This is a huge subfield of psychology (persuasion) and the basis of marketing. Do you think it takes some super-special never-going-to-be-replicated feat of non-Turing-computable human thought to write Zynga games?

It's nice to think the world is a safe place, but the reality is that our social order is increasingly precarious and an AI could easily disrupt that.

Perhaps. But if that is the case, the people who are most intelligent by this definition are far from the ones recognized as intelligent by the AI-fearing community. Let me put it this way: Albert Einstein and Richard Feynman would not be among them. Adolf Hitler, on the other hand, would be a genius.

> You're wrong, and super-human AI is a massive issue, regardless of how you pattern-match it as "religious"

If you think I don't always presume that everything I say is likely wrong then you misunderstand me. I do, however, don't understand what you mean by "a massive issue". Do you mean imminent danger? Yes, I guess it's possible, but being familiar with the state of the art, I can at least discount the "imminent" part.

> You're not creative enough (one might dare say "intelligent" but that would be too snarky) to imagine all the ways in which an AI could devastate humanity without it having much intelligence or much charm

I can imagine many things. I can even imagine an alien race destroying our civilization tomorrow. What I fail to see is compelling arguments why AI is any more dangerous or any more imminent than hundreds of bigger, more imminent threats.

> In fact, most hyper-successful people are almost certainly good at both.

I would gladly debate this issue if I believed you genuinely believed that. If you had a list ordered by social power of the top 100 most powerful people in the world, I doubt you would say their defining quality is intelligence.

> it's very possible to apply the scientific method and empirical problem solving to finding them, and then exploiting humans that way. This is a huge subfield of psychology (persuasion) and the basis of marketing.

Psychology is one of the fields I know most about, and I can tell you that the people most adept at exploiting others are not the ones you would call super-intelligent. You wouldn't say they are of average intelligence, but I don't think you'd recognize their intelligence as being superior.

> It's nice to think the world is a safe place, but the reality is that our social order is increasingly precarious and an AI could easily disrupt that.

There are so many things that could disrupt that, and while AI is one of them, it is not among the top ten.

How so? Feynman in particular was quite able to continually accomplish his goals, and he purposely chose divergent goals to test himself (his whole "I'll be a biologist this summer" thing).

And yes, see my original comment re: it takes intelligence to walk into the DAP meeting and join as member 55 and come out conquering mainland Europe.

>I do, however, don't understand what you mean by "a massive issue". Do you mean imminent danger? Yes, I guess it's possible, but being familiar with the state of the art, I can at least discount the "imminent" part.

The state of the art is irrelevant here; in particular, most of AI seems to be moving in the direction of "use computers to emulate human neural hardware and use massive amounts of training data to compensate for the relative sparseness of the artificial neural networks."

What's imminently dangerous about AI is that all it really takes is a few innovations that might be in seemingly unrelated areas enable probably several people who see the pattern to go and implement AI. This is how most innovation happens, but here it could be very dangerous, because...

>What I fail to see is compelling arguments why AI is any more dangerous or any more imminent than hundreds of bigger, more imminent threats.

AI could totally destabilize our society in a matter of hours. Our infrastructure is barely secure against human attackers, and it could be totally obliterated by an AI that chose to do that, or incidentally caused it to happen. An AI might not be able to launch nukes directly (in the US at least, who knows what the Russians have hooked up to computers), but it could almost certainly make it seem to any nuclear power that another nuclear power had launched a nuclear attack. There actually are places that will just make molecules you send them, so if the AI figures out protein folding, it could wipe out humanity with a virus.

AI is more dangerous than most things, because it has:

* limitless capability for action

* near instantaneous ability to act

The second one is really key; there's nearly nothing that would make shit hit the fan FASTER than a hostile AI.

If you have a list of hundreds of bigger, more imminent threats, that can take humanity from 2015 to 20000BCE in a day, I'd like to see it.

>I doubt you would say their defining quality is intelligence.

I'm confused as to how you can read three comments of "intelligence is the ability to accomplish goals" and then say "people who have chosen to become politically powerful and accomplished that goal must not be people you consider intelligent."

>You wouldn't say they are of average intelligence, but I don't think you'd recognize their intelligence as being superior.

Well, they can exploit people. How's that for superiority?

My background is admittedly in cognitive psychology, not clinical, but I do see your point here. I'd like to make two distinctions:

* A generally intelligent person (say, Feynman) could learn to manipulate people and would almost certainly be successful at it

* People that are most adept at manipulating people, usually are that way because that's the main skill they've trained themselves for over the course of their lives.

>it is not among the top ten.

Of the top ten, what would take less than a week to totally destroy our current civilization?

His goals pertained to himself. He never influenced the masses and never amassed much power.

> it takes intelligence to walk into the DAP meeting and join as member 55 and come out conquering mainland Europe.

I didn't say it doesn't, but it doesn't take super intelligence to do that. Just more than a baseline. Hitler was no genius.

> What's imminently dangerous about AI is that all it really takes is a few innovations that might be in seemingly unrelated areas enable probably several people who see the pattern to go and implement AI.

That could be said just about anything. A psychologist could accidentally discover a fool-proof mechanism of brainwashing people; a microbiologist could discover an un-killable deadly microbe; an archeologist could uncover a dormant spaceship from a hostile civilization. There's nothing that shows that such breakthroughs in AI are any more imminent than in other fields.

> Our infrastructure is barely secure against human attackers, and it could be totally obliterated by an AI that chose to do that

Why?

> but it could almost certainly make it seem to any nuclear power that another nuclear power had launched a nuclear attack

Why can an AI do that but a human can't?

> limitless capability for action

God has limitless capability for action. But we have no reason whatsoever to believe that either God or true AI would reveal themselves in the near future.

> near instantaneous ability to act

No. Again,

> there's nearly nothing that would make shit hit the fan FASTER than a hostile AI.

There's nothing that would make shit hit the fan FASTER than a hostile spaceworm devouring the planet. But both the spaceworm and the AI are currently speculative sci-fi.

> I'm confused as to how you can read three comments of "intelligence is the ability to accomplish goals"

There are a couple of problems with that: one, that is not the definition that is commonly used today. Britney Spears has a lot of ability to achieve her goals, but no one would classify her as especially intelligent. Two, that is not where AI research is going. No one is trying to make computers able to "achieve goals", but able to carry out certain computations. Those computations are very loosely correlated with actual ability to achieve goals. You could define intelligence as "the ability to kill the world with a thought" and then say AI is awfully dangerous, but that definition alone won't change AI's actual capabilities.

> A generally intelligent person (say, Feynman) could learn to manipulate people and would almost certainly be successful at it

I disagree. We have no data to support that prediction. We know that manipulation requires intelligence, but we do not know that added intelligence translates to added ability to manipulate and that that relationship scales.

> what would take less than a week to totally destroy our current civilization?

That is a strange question, because you have no idea how long it would take an AI. I would say that whatever an AI could achieve in a week, humans could achieve in a similar timeframe and much sooner. In any case, as someone who worked with neural-networks in the nineties, I can tell you that we haven't made as much progress as you think. We are certainly not at any point where a sudden discovery could yield true AI any more than a sudden discovery would create an unkillable virus.

I acknowledge it as a real risk, but it's not terribly high on my personal list of things to worry about right now. I like what Andrew Ng said about how worrying about this now is "like worrying about over-population on Mars".

I'm not so confident that human-level AI will take a long time. The timeline depends on algorithmic insights, which are notoriously difficult to predict. It could be a century. It could be a decade. Still, it seems like something worth worrying about.

I never said I wasn't worried about AI. You're extrapolating from what I did say; which I've said all along was just a thought experiment, not a position I'm actually arguing for.

It's one of those books that put the arguments so clearly you're suddenly catapulted to having a vastly superior knowledge of the subject than someone trying to do simple thought experiments.

Both of your arguments look out-dated if you're one of the people 'in the know'.

Also, I suggest looking a bit more into what's going on in machine learning, it's suddenly got far more sophisticated than I personally realized until a couple of months ago when I was chatting to someone developing in it at the moment.

Ng's statement is, at best, equivalent to a student who is putting off starting their semester project until finals week. Yes it seems far away, but the future is going to happen.

"a freshman student who is putting off studying for his Senior final project until his Senior year".

The question Ng asked was something like "is there any practical action we can take today, to address over-population on mars" as an analogy to "is there any practical step we can take today to address the danger of a super-AGI". And honestly, I'm not convinced there is anything practical to do about super-AGI today. Well, nothing besides pursuing the "open AI" strategy.

But I'm willing to be convinced otherwise if somebody has a good argument.

http://futureoflife.org/data/documents/research_priorities.p...

What makes you think we humans won't attempt to do even more harm towards humanity? Maybe the AI will save us from ourselves, and, being so much smarter, might guide us towards our further evolution.

I think it's legitimately an interesting question. As in, it could be something like Turing completeness. All Turing complete languages are capable of computing the same things, some are just faster. Maybe there's nothing beyond our level of understanding, just a more accelerated and accurate version of it. An AI will think on the same level as us, just faster. In that case, in that hypothetical, an AI 100x faster than a person is not much better than 100 people. It won't forget things (that's an assumption, actually), it's neuron firing or equivalent would be faster, but maybe it won't really be capable of anything fundamentally different than people.

This is not the same as the difference between chimps and humans. We are fundamentally on another level. A chimp, or even a million chimps, can never accomplish what a person can. They will not discover abstract math, write a book, speak a language.

Mind you, I suspect this is not the case. I suspect that a super intelligent AI will be able to think of things we can never hope to accomplish.

But it is an interesting question that I think is worth thinking about, rather than inanely down voting the idea.

It's helpful to remind people that every human brain that exists runs at the same processing speed†, yet we have greatly varying intelligence between us. (Also, IQ is an index, but people confuse it for a linear measure; human geniuses may be doing things that require many times the "intelligence" of the average person.)

† Okay, I lied: there are some people whose neurons have abnormal firing rates; the visible result is Parkinsonism.

Think about how long it takes you to imagine a program vs actually coding it, or imagining an object you want to create and actually building it, there are at least two or three orders of magnitude for improvement over the keyboard.

Yeah, that's another really good way of putting it. That's probably closer to what I meant, than what I said above.

- that there's only one class of "humanic" intelligence;

- that we can approximately represent instances in this class of intelligence as vectors of {memory, learning speed, computation speed, communication speed};

- that any AI that could be created is merely a vector in this n-dimensional intelligence space, lacking any extra-intelligent qualities;

- that productivity and achievement increases exponentially the more intelligent being you devote to a problem, but logarithmically for more beings you devote (e.g. a being with intelligence vector {10,10,10,10} might be as productive as 10000 {1,1,1,1} beings);

then this doesn't exclude the possibility of us creating an AI with an intelligence vector twice an average human's intelligence vector, which can suggest improvements to its algorithms and datacenter and chip designs to become 10x as intelligent as a human, and from there it could quickly determine new algorithms, and eventually it's considering philosophy (and what to do about these humans).

The point is: viewing intelligence the way you suggest doesn't help us on what to do about "super" artificial intelligence.

Humans ourselves are very bad at predicting the weather. So we build supercomputers, and weather models, and now we're a bit better at it.

Now the question is: given the same budget, would a super-intelligent AI predict the weather in a substantially different way to humans? I cannot see how, but maybe that is my human stupidity talking.

However if it is possible to improve weather simulations, and if there are advancements to be made in meteorology, then it's very likely an AI would be able to find them.

In terms of speed: if you could build the exact silicon equivalent of a human brain, you may be able to run it several orders of magnitude faster, simply because it wouldn't be limited by the slow speeds of electrochemical processes in the human brain. Nerve impulses travel at speeds measured in meters per second. Neurons also need time to recharge between spike bursts, and they can physically damage themselves if they get too excited.

In terms of volume: much of our intelligence is in perceiving patterns. That's limited by cortical mass. Pattern recognition is what all these "deep learning" systems excel at. The more depth they add, the better they get. Having deeper pattern recognizers, or simply having more of them, means you can see more patterns, more complex patterns, etc. Things that might be beyond the reach of any human.

Then, in terms of data, machine have an advantage too. We're limited by our short lifetimes. How many people are expert musicians, genius mathematicians, rockstar programmers and great cooks? Very few. There's only so many hours in a day, and we only live so long. A machine could learn all those skills, and more. It could excel at everything. Speak every language, master every skill, be aware of so many more facts.

And finally, I posit that maybe we, humans, are limited in our ability to grasp complex conceptual relationships. If you think about it, the average person can fit 7-8 items in their short-term memory, in their brain's "registers", so to speak. That probably limits our ability to reason by analogy. We can go "A is to B what C is to D", but maybe more complex relationships with 50 variables and several logical connectives will seem "intuitive" to a machine that can manipulate 200 items in its short-term memory.

Right, but the idea I'm playing around with is this: Suppose you had a hypothetical creature with a brain 20x as fast as the human brain and with twice the volume. How much smarter would that creature be in practice. It's kind of an abstract idea (and I probably don't fully understand it myself), but I'm getting at something like "is there a point where that additional raw computing power just doesn't buy you anything meaningful" at least in terms of "does it represent an existential threat?" or "does the nuclear analogy hold?"

(There are more technical arguments that an AI's values would be unlikely to be the complex mismash that human values are, so such an AI would be very unlikely to share our sentimental desire to not make species extinct)

That is, unless the reason we don't have AI is because we haven't put the pieces together correctly, and it really could work on minimal hardware.

The only human level general intelligence we have to look at is humans. Sure, it's possible that there's an easy way to build an artificial intelligence that doesn't need to sleep and isn't interested in art but that retains our ability to problem solve; but maybe their isn't. It's possible that we'll be able to easily figure out that alternative brain architecture and build in the near future; but given how little we still don't understand about our own brain (and the failure to mimic the brains of even simple creatures), it doesn't seem likely. It's possible that if we were able to overcome those major hurdles, a super John von Neumann would be able to wield enormous political power; or it's possible that, like the real von Neumann, it would focus mostly on its research.

In order to get to the AI doomsday scenario, you have to assume that a lot of very unlikely things are all going to happen. And the main argument for them happening seems to be one of ignorance - "hey, we don't have any idea what this will be like, so we can't say that this improbable situation won't happen."

And even if you're right, as Alexander says, in that case why the need to open-source it? If you think it's important that AI is open-source and not controlled by e.g. Facebook, that must be because you think AI is going to be powerful and effective. In which case it's as dangerous as he says.

People seem more ready to accept the idea that an artificial general intelligence would act similar to it's portrayed in SciFi stories than accept that it might act similar to the other general intelligences we can observe (namely ourselves).

It could be dangerous though.

http://www.yudkowsky.net/singularity/aibox/

Please don't try to argue by name-calling. A strong argument is stronger without it.

"DH0. Name-calling.

This is the lowest form of disagreement, and probably also the most common. We've all seen comments like this:

    u r a fag!!!!!!!!!! 

    The author is a self-important dilettante. 

"The AI party may not offer any real-world considerations to persuade the Gatekeeper party. For example, the AI party may not offer to pay the Gatekeeper party $100 after the test if the Gatekeeper frees the AI... nor get someone else to do it, et cetera. The AI may offer the Gatekeeper the moon and the stars on a diamond chain, but the human simulating the AI can't offer anything to the human simulating the Gatekeeper. The AI party also can't hire a real-world gang of thugs to threaten the Gatekeeper party into submission. These are creative solutions but it's not what's being tested. No real-world material stakes should be involved except for the handicap (the amount paid by the AI party to the Gatekeeper party in the event the Gatekeeper decides not to let the AI out). (...) In case you were wondering, I (Yudkowsky) obeyed this protocol voluntarily in both earlier tests."

E.g. "I'll give you $500, but you also have too sign an NDA, so that people don't know we cheated."

I don't want to imply that they cheated, just want to reiterate my original argument that the lack of transparency makes the experiment effectively invalid. Think Tesla and his claims about World Wireless system.

I've never seen a published log. Link?

That's a lower bound on the scariness of AI explosion, and it may already be enough to take over the world. Certainly it should be enough to take over the Internet circa 2015...

To my mind it seems pretty clear that if AI exists, then scary AI is not far off.

That said, I don't worry about this stuff too much, because I see AI as being much technically harder, and much less likely to materialize in our lifetimes, than articles like this suppose.

I mean, are you going to put the same machine in charge of mineral extraction, weapon construction, transportation, and weapon deployment? When it hasn't proven to act correctly in a high-fidelity simulated environment? Probably not.

We're also assuming that human ethics and intelligence are independent. I don't see many reasons to believe this. Social power and intelligence might be independent.

The computation environment of neurons is also extremely noisy (axons are not well insulated) and neurons only fire at 7-200Hz. Assuming noise and low firing rate do not fulfill a certain task in mammalian brains, this is another way in which silicon-based minds could potentially be vastly superior.

Thirdly, assuming sleep is not necessary for intelligence, artificial minds would never get exhausted. They could work 24 hours on a problem a day, which is possibly 5-10 time the amount of thinking time a human can do realistically.

And lastly, an AI could easily make copies of itself. Doing so it could branch a certain problem to many computers which run copies of it and eventually collect the best result, or just shorten the time it takes to get a result. It could also evolve at a much faster rate than humans, assuming it has a genetic description: possibly hours to seconds instead of 20 years. Anyhow, it could easily perform experiments with slightly changed version of itself.

If we think of intelligence as skill at solving problems, it might be that there are not many problems that are easily solved with increased intelligence alone, because the solutions we have are nearly optimal, or because other inputs are needed as well.

This seems most likely to happen with mathematical proofs and scientific laws. Increased intelligence doesn't let you prove something that's false, and it doesn't let you violate scientific laws.

But I don't find this particularly plausible. Consider computer security: hackers are finding new exploits all the time. We're far from fixing all the loopholes that could possibly be found.

Yeah, I think that's a much better way of putting what I'm trying to get at. That is, maybe the machine is much "smarter" depending on how you define "smart". But maybe that doesn't enable it to do much more than we can do, because of other fundamental limits.

It is not just an issue of finding solutions to impossible problems, or breaking scientific laws, but also problems where the solutions are along the lines of what George Soros would call reflexive. Computer security is like this, so is securities trading (no pun.)

Secondly, what about problems which require the destruction of the problem solvers along the path to the optimal solution? I'm not sure about the correct word to describe this, or the best example but it could be seen in large systems. Where humans are right now is a result of this. We would not know many things if those things which came before were not destroyed (cities, 1000 year empires, etc.)

Thirdly, is a uniform, singular AI the most optimal agent to solve these sorts of problems? Much the way we don't rely or use mainframes for computing today, perhaps there will be many AI agents each which may be really good at solving particular narrowly defined problem sets. This could be described perhaps as a swarm AI.

Nick Bostrom's Superintelligence is a great book, but I don't recall much consideration along these lines. When a lot of AI agents are "live" the paths to solutions where AI compete against each other open up even more complex scenarios.

There certainly are physical limitations to AI. Things like the speed of light can slow down processing. Consumption of energy. Physical elements that can be assembled for computational purposes.

Between now and "super" AI, even really good AI could struggle to find solutions to the most difficult problems, especially if those are problems other AI are creating. The speed alone may be the largest challenge to humans. How do we measure this difficulty relative to human capabilities, I don't know.

End of rant -- but the limits of not just AI but problem solving is quite interesting.

Why not compare it to 1000 people, all communicating and problem solving together?

We know that this is possible because it happens all the time, and enables such groups to make lots of money in digital markets, and invest it in things like marketing, robotics, and policy.

The intelligence of an AI is lower bounded by that of the most intelligent possible corporation.

Potential corollary: Assuming one can make a human level AI, then if it is not sufficiently resource constrained (hard?) or somehow encoded with "human values" (very very hard), then it will be at least as dangerous as the most sociopathic human corporation.

Stop there.

We, as humans, don't even know how smart we actually are - and we probably never will. It's very unlikely that any species is equipped to accurately comprehend its own cognitive limits - if such limits even exist.

It's even less likely that we can relegate the intelligence of a nonhuman entitity to a mathematically meaningful figure without restricting the testing material to concepts and topics meaningful to humans - which may have absolutely no relation to the intelligence or interest domains of a nonhuman entity.

As a child I had problems with other children picking on me, I was suspected of having autism due to social issues, I was given an IQ test and scored a 189. I was high functioning an in 1975 there was no such thing as high functioning autism, that wasn't discovered until 1994, so I got diagnosed with depression instead. Child Psychologist told my parents to put me into a school for gifted children, but they put me in public school instead where I struggled and my brain worked 10 times faster so I was always ahead of the class in learning and bored waiting for people to catch up. I was still bullied and picked on and this interfered with my learning. The same thing happened when I went to college and had a job, I was bullied and picked on. I never reached my potential and my mental illness was one of the reasons why and people picking on me was another reason, and had I been in a school for gifted children I'd be able to reach my potential better.

I developed schizoaffective disorder in 2001 and it screws with my memory and focus and concentration. I ended up on disability in 2003. My career is basically over but I still have a potential I never met.

What good is a high IQ if you can't reach your potential to use it?

We keep hearing talk of an AI that is smarter than a human being, but we haven't seen one yet. Our current AI programs are not as smart as a human being yet, but they can do tasks that put human beings out of work. Just having a perfect memory, and being able to do fast math equations makes an AI in the "Rainman" category http://www.imdb.com/title/tt0095953/ even if it is not as smart as a human being.

I guess what I am trying to say is that an AI doesn't have to be as smart as a human being to be dangerous. Just like the Google Maps app that drives people off a cliff or into an ocean. An AI can make robocalls and sell a product and put people out of work. You can replace almost any line of work with an AI, and then it gets dangerous when a majority of people are unemployed by AIs that aren't even as smart as a human being.

I'd like to see a personal AI that works for poor and disabled people to earn money for them, as they run it on a personal computer. Doing small tasks on an SEO marketplace using webbots for $5 each and 100 tasks a day for $500 in a Paypal account to help lift them out of poverty. I know there are people already doing this for their own personal gain, but if the AI that does that is open sourced so disabled and poor people can run it, it can help solve the problem of poverty.

https://wiki.lesswrong.com/wiki/Paperclip_maximizer

Now that's more depressing than global annihilation.

For many 1000's of years different religions, some on opposite sides of the world from each other, have been saying the same thing. We have the ability to be better people, but all of us squander it with pettiness, jealousy, greed etc.

Fair point. But I honestly wonder if it's not true, or close to true. I mean, people have been talking about the "end of science" for a while now, even though we know we're not quite literally at "the end". But let's say that we unify relativity / gravity / QM in the next 100 years or so, and identify the exact nature of dark energy / dark matter. Presumably that would represent knowing close to everything about the physical world that its possible to know. And if we can discover that, it does lead you to wonder "what else would more intelligence represent"?

Of course, maybe we're WAY more than 100 years away from those things. Or maybe we'll never get there. Or maybe if we do, it doesn't mean anything vis-a-vis the limits of intelligence. I'm just thinking out loud here, this isn't an argument that I've put a lot of time into developing...

That's a severe misunderstanding of what it means to have a unified physical theory. The current standard model of physics allows us to theoretically predict the evolution of physical systems with an insanely high precision. A precision so great that there wouldn't be any meaningful error simulating macroscopic objects like human brains, given sufficient computing power.

But we still have massive holes in our understanding of many things, including human biology or neurology. We don't have enough computing power to precisely simulate, but even if we had that power, it wouldn't necessarily let us understand anything. Elementary particles and humans are on hugely different scales. Observing a simulated human would be easier than observing a human simulated by physics itself, but it wouldn't just magically make everything clear.

Having a TOE in physics brings us very little relative benefit compared to the Standard Model, when it comes to understanding biology, materials science, geology, psychology, computer science, etc.

Sure, like I said, I'm just thinking out loud here. Obviously TOE in and of itself doesn't automatically translate into full knowledge of everything in practice. But in principle, it would given sufficient computing power, allow us to simulate anything, which I think would yield additional understanding. But here's the thing, and where this all ties back together: sufficient computing resources may not be possible, even in principle, to allow use of that TOE to do $X. But an upper bound on the amount of computing resources should also reflect something of an upper bound on what our hypothetical AI can do as well.

Or to put it all slightly differently.. if we had a TOE, we'd have shown, at least, that humans are smart enough to develop a TOE. Which I think at least raises the issue of "how much smarter can a machine be, or what would it mean for a machine to be much smarter than that?"

Note to that I'm not necessarily arguing for this position. It's more of a thought experiment or discussion point than something I'm firmly convinced of.

It's not about humans being as smart as possible though, it's more about being "smart enough" to where a hypothetical "smarter than human AI" is not analogous to a nuclear bomb. That is, are we smart enough to where a super-AGI can't come up anything fundamentally new, that humans aren't capable of coming up with, as bound by the fundamental laws of nature.

then I would recommend reading Thinking Fast and Slow or some other book on cognitive psychology

I'm reading Thinking, Fast and Slow right now, actually.

And just to re-iterate this point: I'm not arguing for this position, just putting it out there as a thought experiment / discussion topic. I'm certainly not convinced this is true, it's just a possibility that occurred to me earlier while reading TFA.

As a philosopher of the future, Bostrom has successfully combined the obscurantism of Continental philosophy, the license of futurism and the jargon of technology to build a tower from which he foresees events that may or may not occur for centuries to come. Nostradamus in a hoody.

Read this sentence:

"It looks quite difficult to design a seed AI such that its preferences, if fully implemented, would be consistent with the survival of humans and the things we care about," Bostrom told Dylan Matthews, a reporter at Vox.

Notice the mixture of pseudo-technical terms like “seed AI” and “fully implemented”, alongisde logical contructs such as “consistent with” -- all leading up to the phobic beacons radiating at the finale: “the survival of humans and the things we care about.”

It's interesting, the technical challenges that he feels optimism and pessimism for. For reasons best known to himself, Bostrom has chosen to be optimistic that we can solve AI (some of the best researchers are not, and they are very conservative about the present state of research). It may perhaps the hardest problem in computer science. But he's pessimistic that we'll make it friendly.

Bostrom’s tower is great for monologs. The parlor game of AI fearmongering has entertained, rattled and flattered a lot of people in Silicon Valley, because it is about us. It elevates one of our core, collective projects to apocalyptic status. But there is no dialog to enter, no opponent to grapple with, because no one can deny Bostrom's pronouncements any more than he can prove them.

Superintelligence is like one of those books on chess strategy that walk you through one gambit after the other. Bostrom, too, walks us through gambits; for example, what are the possible consequences of developing hardware that allows us to upload or emulate a brain? Hint: It would make AI much easier, or in Bostrom’s words, reduce “recalcitrance.”

But unlike the gambits of chess, which assume fixed rules and pieces, Bostrom’s gambits imagine new pieces and rules at each step, substituting dragons for knights and supersonic albatrosses for rooks, so that we are forced to consider the pros and cons of decreasingly likely scenarios painted brightly at the end of a line of mights and coulds. In science fiction, this can be intriguing; in a work of supposed non-fiction, it is tiresome.

How can you possibly respond to someone positing a supersonic albatross? Maybe Bostrom thinks it will have two eyes, while I say three, and that might make all the difference, a few more speculative steps into the gambit.

In the New Yorker article The Doomsday Invention, Bostrom noted that he was "learning how to code."

http://www.newyorker.com/magazine/2015/11/23/doomsday-invent...

We might have expected him to do that before he wrote a book about AI. In a way, it's the ultimate admission of a charlatan. He is writing about a discipline that he does not practice.

(There's also some DH2 http://paulgraham.com/disagree.html level stuff about the terms Bostrom chooses to use to make his argument... I'm not sure if there's anything to be said about this except that Bostrom's book seems more accessible to me than most academic writing http://stevenpinker.com/why-academics-stink-writing and I'd hate for him to receive flak for that. It's the ideas that matter, not the pomposity with which you communicate them. I also don't understand the implied disregard for anything that seems like science fiction--what's the point in trying to speculate about the future if all the speculation will be ignored when decisions are being made?)

If an Oxford philosophy professor is not enough for you, here's a succinct explanation of the AI safety problem & its importance from Stuart Russell (Berkeley CS professor & coauthor of Artificial Intelligence: A Modern Approach):

https://www.cs.berkeley.edu/~russell/research/future/

I have nothing against science fiction, but I object to any fiction that disguises itself as non-fiction, as Bostrom's often does. Nor do I think that the impossibility of predicting the future of AI is, in itself, a reason for undue caution.

Bostrom is performing a sort of Pascalian blackmail by claiming that the slight chance of utter destruction merits a great deal of concern. In fact, he is no different from a long line of doomsday prophets who have evoked fictional and supposedly superior beings, ranging from deities to aliens, in order to control others. The prophet who awakens fear puts himself in a position of power, and that's what Bostrom is doing.

Regardless of Bostrom's motives, we as humans face an infinite number of possible dangers that threaten total destruction. These range from the angry Judeo-Christian god to the gray goo of nano-technology to Peak Oil or Malthusian demographics. In each case, the burden of proof that we should be concerned is on the doomsday prophet -- we should not default to concern as some sort of reasonable middle ground, because if we do, then we will worry uselessly and without pause. There is not enough room in one mind to worry about all the gods, aliens, golems and unforeseen natural disasters that might destroy us. And you and I both know, most if not all of those threats have turned out to be nonsense -- a waste of time.

I do not believe that Bostrom carries the burden of proof well. His notion of superintelligence is based on a recursively self-improving AI that betters itself infinitely. Most technological advances follow S-curves, moving slow, fast and slow. Bostrom does not seem to grasp that, and cites very little evidence of technological change to back up his predictions about AI. He should be, first and foremost, a technological historian. But he contents himself with baseless speculation.

We are in danger, reading Bostrom or Russell or Good, of falling into language traps, by attempting to reason about objects that do not exist outside the noun we have applied to them. The risk is that we accept the very premise in order to fiddle with the details. But the very premise, in this case, is in doubt.

Agreed.

>Nor do I think that the impossibility of predicting the future of AI is, in itself, a reason for undue caution.

Sure.

>Bostrom is performing a sort of Pascalian blackmail by claiming that the slight chance of utter destruction merits a great deal of concern. In fact, he is no different from a long line of doomsday prophets who have evoked fictional and supposedly superior beings, ranging from deities to aliens, in order to control others. The prophet who awakens fear puts himself in a position of power, and that's what Bostrom is doing.

Consider: "Previous books published by authors hailing from Country X contained flaws in their logic; therefore since this book's author came from Country X, this book must also have a logical flaw." It's not a very strong form of argument: you might as well just read the book to see if it has logical flaws. Similarly even if a claim seems superficially similar to the kind of claim made by non-credulous people, that's far from conclusive evidence for it being an invalid claim.

It would be a shame if religious doomsayers have poisoned the well sufficiently that people never listen to anyone who is saying we should be cautious of some future event.

>And you and I both know, most if not all of those threats have turned out to be nonsense -- a waste of time.

Sure, but there have been a few like nuclear weapons that were very much not a waste of time. Again, you really have to take things on a case by case basis.

>His notion of superintelligence is based on a recursively self-improving AI that betters itself infinitely. Most technological advances follow S-curves, moving slow, fast and slow. Bostrom does not seem to grasp that, and cites very little evidence of technological change to back up his predictions about AI. He should be, first and foremost, a technological historian. But he contents himself with baseless speculation.

I can see how this could be a potentially fruitful line of reasoning and I'd encourage you to pursue it further, since the future of AI is an important topic and it deserves more people thinking carefully about it.

However, I don't see how this does much to counter what Bostrom wrote. Let's assume that AI development will follow an S-curve. This by itself doesn't give us an idea of where the upper bound is.

Estimates suggest that human neurons fire at a rate of at most 200 times per second (200 hertz). Modern chips run in the gigahertz... so the fundamental activity that chips do is being done at something like a million times the speed that the fundamental activity neurons do is being done. (Humans are able to do a ton of computation because we have lots of neurons and they fire in parallel.)

And there are lots of arguments one could make along similar lines. Human brains are limited by the size of our skulls; server farms don't have the same size limitations. Human brains are a kludge hacked together by evolution; code written for AIs has the potential to be much more elegant. We know that the algorithms the human brain is running are extremely suboptimal for basic tasks we've figured out how to replicate with computers like doing arithmetic and running numerical simulations.

Even within the narrow bounds within which humans do differ, we can see variation from village idiots to Von Neumann. Imagine a mind V' that's as inscrutable to Von Neumann as Von Neumann is to a village idiot, then imagine a mind V'' that's as inscrutable to V' as V' is to Von Neumann, etc. Given that AI makers do not need to respect the narrow bounds within which human brains differ, I don't think it would be surprising if the "upper bound" you're discussing ends up being equivalent to V with a whole bunch of prime symbols after it.

You're right, that's not a very strong form of argument. But that is not my argument. The connection between Bostrom and previous doomsayers is not as arbitrary as geography. Your analogy is false. The connection between Bostrom and other doomsayers in the structure of their thought. The way they falsely extrapolate great harm from small technological shifts.

> It would be a shame if religious doomsayers have poisoned the well sufficiently that people never listen to anyone who is saying we should be cautious of some future event.

I think many people do not realize the quasi-religious nature of AI speculation. Every religion and charismatic mass movement promises a "new man." In the context of AI, that is the singularity, the fusion of humans with intelligent machines. The power we associated with AI present and future leads us to make such dramatic, quasi-religious predictions. They are almost certainly false.

>Sure, but there have been a few like nuclear weapons that were very much not a waste of time. Again, you really have to take things on a case by case basis.

Nuclear weapons are fundamentally different from every other doomsday scenario I cited because they were designed explicitly and solely as weapons whose purpose was to wreak massive destruction at a huge cost to human life. The potential to destroy humanity defined nuclear weapons. That is not the case for other technological, demographic and resource-related doomsday scenarios, which are much more tenuous.

The fact that a claim bears superficial resemblance to one made by wackos might be a reason for you to believe that it's most likely not worth investigating, but as soon as you spend more than a few minutes thinking about a claim, the weight of superficial resemblances is going to be overwhelmed by other data. If I have been talking to someone for three hours, and my major data point for inferring what they are like as a person is what clothes they are wearing, I am doing conversation wrong.

"Nuclear weapons are fundamentally different from every other doomsday scenario I cited because they were designed explicitly and solely as weapons whose purpose was to wreak massive destruction at a huge cost to human life. The potential to destroy humanity defined nuclear weapons. That is not the case for other technological, demographic and resource-related doomsday scenarios, which are much more tenuous."

Sure, and in the same way nuclear bombs were a weaponization of physics research, there will be some way to weaponize AI research.

http://www.givewell.org/labs/causes/ai-risk/ai-timelines

No, there's clearly at least one human smarter than you.

Right, like I said, this whole thing was just "thinking out loud" more than any thoroughly researched and validated idea. And I didn't do a very good job of even explaining what I was getting at. It's not so much that the question is "are we the smartest creatures possible". It's more like:

"as smart as we are, and given the constraints of the physical world, is there room for a hypothetical 'smarter than human AGI' to represent an existential threat, or something that justifies the analogy to nuclear weapons?"

See also skybrian's comment which actually states it better than I did originally:

https://news.ycombinator.com/item?id=10755109

The answer is still yes. Your "thinking out loud" is privileging an already refuted hypothesis.

Well that's a fine assertion, but how do you justify it?

Your "thinking out loud" is privileging an already refuted hypothesis.

WTH does that even mean? There is no hypothesis, there's a vague notion of an area to consider, which - if considered thoroughly - might or might not yield a hypothesis.

The main thing is that the constraints of the physical world are nowhere near the limitations of human capabilities. Yes, there's a theoretical limit to how much computation you can do in a certain amount of space with a certain amount of energy. No biology or technology currently in existence even gets close to those limits.

Without any such fundamental limit there's just no reason for AGI not to present a threat.

But there is lots of other evidence too.

These are basic questions, the answers are known, published, and the article even mentions exactly where you can find them.

I've read and re-read TFA and I don't find that it addresses the issue I'm thinking about. It's not so much asking "are we the smartest possible creature", or even asking if we're close to that. It's also not about asking whether or not it's possible for a super-AGI to be smarter than humans.

The issue I was trying to raise is more of "given how smart humans are (whatever that mean) and given whatever the limit is for how smart a hypothetical super-AGI can be, does the analogy between a super-AGI and a nuclear bomb hold? That is, does a super-AGI really represent an existential threat?"

And again, I'm not taking a side on this either way. I honestly haven't spent enough time thinking about it. I will say this though... what I've read on the topic so far (and I haven't yet gotten to Bostrom's book, to be fair) doesn't convince me that this is a settled question. Maybe after I finish Superintelligence I'll feel differently though. I have it on my shelf waiting to be read anyway, so maybe I'll bump it up the priority list a bit and read it over the holiday.

I'm not even sure how we could stop it, but we should really be passing laws right now about algorithms that operate like a black box where a training algorithm is used to generate the output. For some reason everyone just thinks we should rush forward into this not concerned about an AI that is super human.

Whether it is a good or bad actor doesn't even matter. Giving up control to a non-human entity is the worst idea humanity has ever had. We will end up in a zoo either way.

General AI will be a massive advance for our economy, for our culture, for science, for military, for ...

A lot of things humans want to do but can't, effectively because of human body and/or brain limitations. From efficiently building buildings, taking risks that our bodies effectively don't allow for (e.g. being abandoned on Mars with little equipment would be a bit harsh, but not catastrophic, for an AI. And bringing him back means a data transmission), doing things that our bodies don't allow for (like building buildings/houses/... using huge premade blocks quickly. Humans can do it, but if we had hands the size of cars we could build those houses like we build lego houses). Defense/policing. An AI would not be risking life nor limb. An AI could just walk into the middle of a firefight, and worst case, he/she needs to be restored from backup)

All of these things sound like very good things. And yes, in the very long term AIs will replace humans. But in the very long term the human species is dead anyway. Does it really matter that much if we get replaced by a subspecies (best case scenario), another species, or AI ? Plus, you won't experience that, nor will your great-great-great-great grandchildren. At some point it doesn't matter anymore.

No, on the contrary, nearly everyone who spends any amount of time thinking about it quickly realizes the risks.

The concession is the realization that the technology is an inevitability (because of the immense power it grants the wielder, and because of the wide gradient of safe and useful AI to dangerous AI).

I think you would have an extremely tough time deciding where to draw the line. The closest parallel we have may be the export controls on cryptography or the ridiculousness that emerged from the AACS encryption key fiasco.

Right now we are making the building blocks that will make up that AI. We are very close to AI that can drive tanks and fly weaponized drones. We are very close to AI that replaces most blue collar jobs and the majority of jobs in the world really.

If we stop these lines of AI research and technology right now we can probably make it to the stars while still being a free people. If we make a true AI whether it is benevolent or not doesn't even matter. Humanity will no longer be in control of its destiny.

You know this because you're an expert in the field?

http://www.pcworld.com/article/237005/foxonn_to_rely_more_on...

I don't have to be an expert. Anyone can see what is happening.

If an AI could compromise our military then it could just subvert our communications with atomic submarines and installations throughout the world. It wouldn't even have to though.

Honestly an AI that compromises even some security could just slowly take over the world in a way where almost no one would even realize it was happening. It would have basically infinite financial resources like immediately and then it could just buy some humans to do any leg work that needed to be done. There are just so many ways it could make money incredibly quickly and once that happens there are not a lot of other obstacles really.

I mean I make a lot of money completely on the internet and I am an ape that only works normal hours.

If we believe that pfisch is correct in their assertion that handing control of things to an AI means we end up living as if we're in a zoo, then we'll (presumably) decide that the benefits aren't too good to refuse, and we'll refuse them.

Whether or not that premise is correct is what's up for debate.

This is precisely what dumbfounded me about the announcement.

>My biggest hope is that as usual they are smarter than I am and know something I don’t.

It's possible that OpenAI might be a play to attain a more accurate picture of what constitutes state-of-the-art in the field, effectively robbing the large tech companies of their advantage—all the while building a robust research organization that could potentially go dark if necessary.

Admittedly, that also sounds like it could be the plot to a Marvel movie. Perhaps a simpler explanation is that the details aren't really hashed out yet, and they're essentially going to figure it out as they go—which would be congruent with the gist of OpenAI's launch interview.

They are also probably worried about societal change and the angst everyone is going to feel as AI starts becoming more commonplace. Where do people (beyond entertainers and AI programmers) fit in such a world? They don't. People start to become very irrelevant.

Please be precise here and say they will be irrelevant in economic terms. What will be left are things humans otherwise care about: producing art, consuming art, fun, games, sports, traveling, companionship, partying, building things, learning new things etc.

I'm looking forward to it, and I don't see a reason why anyone wouldn't.

Firearms aren't theoretically capable of recursive self-improvement.

* Whether the source code to advanced AI is open may have some importance, but what determines whether some individual or corporation will be able to run advanced AI is whether they can afford the hardware. I can download some open-source code and run it on my laptop - but Google has data centres with 10s or 100s of thousands of computers. The big corporations are much more likely to have/control the advanced AI because they have the resources for the needed hardware.

* Soft / hard takeoff - I think a lot of people miss that any 'hard takeoffs' will be limited by the amount of hardware that can be allocated to an AI. Let us imagine that we have created an AI that can reach human level intelligence, and it requires a data centre with 10000 computers to run it. Just because the AI has reached human level intelligence doesn't mean that the AI will magically get smarter and smarter and become 'unto a God' to us. If it wants to get 2x smarter, it will probably require 2x (or more) computers. The exact ratio depends on the equation of 'achieved intelligence' vs hardware requirements, and also on the unknown factor of algorithmic improvements. I think that algorithmic improvements will have diminishing returns. Even if the AI is able to improve its own algorithms by say 2x, it's unlikely that will allow it to transition from human level to 'god-level' AI. I think hardware resources allocated will still be the major factor. So an AI isn't likely to get a lot smarter in a subtle, hidden way, or in an explosive way. More likely it will be something like 'we spent another 100M dollars on our new data centre, and now the AI is 50% smarter!'.

Contrary to popular belief, state of the art deep learning is not commonly run on multi-node clusters. Although hardware itself is not the bottleneck for innovation in the current state of the art in deep learning, if we restrict ourself to hardware, the bottleneck is memory bandwidth.

AI advanced enough to be "good" or "evil" won't be developed instantaneously, or by humans alone. We'll need an AI capable of improving itself. I believe the authors argument falls apart at this point; surely any AI able to evolve will undoubtedly evolve to the same point, regardless of it being started with the intention of doing good or evil. Whatever ultra-powerful AI we end up with is just an inevitability.

I guess we'll find out, eh?

Plus, from what I see, the problem reduces to a form of computer security against a clever, malicious threat. You contain it, control what it gets to learn, and only let it interact with the world through a simplified language or interface that's easy to analyse or monitor for safety. Eliminate the advantages of its superintelligence outside the intended domain of application.

That's not easy by any means, amounting to high assurance security against high-end adversary. Yet, it's a vastly easier problem than beating a superintelligence in an open-ended way. Eliminate the open-ended part, apply security engineering knowledge, and win with acceptable effort. I think people are just making this concept way more difficult than it needs to be.

Biggest risk is some morons in stock trading plug greedy ones into trading floor with no understanding of long-term disruption potential of clever trades it tries. We've already seen what damage the simple algorithms can do. People are already plugging in NLP learning systems. They'll do it with deep learning, self-aware AI, whatever. Just wait.

Actually, it's not the lack of understanding, it's the lack of moral responsibility.

We've spent the last few centuries transitioning from a society ruled by strongmen driven by personal aggrandizement to a society where people spend the majority of their adult life as servants to paperclip maximization organizations (aka corporations). Much of what you see in the world today, from the machines that look at you naked at the airport to drones dropping bombs on the other size of the planet to kill brown people, is a result of trying to maximize some number on a spreadsheet.

When we install real AI devices into these paperclip maximizing organizations, you'll have the same problem as you have today with people, except that machines will be less incompetent, less inclined to feather their own nests, and more focused on continually rewriting their software with the express goal of impoverishing every human on the planet to maximize a particular number a particular balance sheet.

[1] https://wiki.lesswrong.com/wiki/Paperclip_maximizer

This is to say, even in a hard-takeoff scenario, we would be looking at something that is still hard-limited by its environment, even if it can compete with a 1000-person organization's worth of intelligence. The danger isn't that it somehow takes over the world by itself; the danger is that we gradually connect it to the same outputs that the decision-making structures of corporate entities are and it ultimately remakes our world with the very tools we give it.

Open-sourcing AGI is no more inherently dangerous than open-sourcing any of the software used to run an enterprise business. It is the choice of what we ultimately give it responsibility for that should draw our caution.

[1] http://omniorthogonal.blogspot.com/2013/02/hostile-ai-youre-...

Likely AIs will be shaped by their builders, if corporations build them, they will adhere primarily to the profit motive, if humanitarian hackers build them they will have human values.

Reputedly the Russian Army has built guard robots, they will just be guns on tanks with a kill radius no values are required - yet are these less moral than the human controlled drones - at least with an AI it can get stuck in a corner or logic loop and you may effect an escape - with humans you need a whistleblower.

Asimov's robot books are informative: his robots are the most moral actors, obeying their 3 laws, often protecting humans from other human decisions.

Certain corporations dehumanise decisions so while processing occurs in wetware, the human worker is only a cog and the invisible hand of human values is removed.

Most workers today could be trivially replaced with a near future neural net.

Of course humans, conspire, complain, unionise, strike, work-to-rule, demand rights and empathise with their customers - so there is a maximum level of evil a corporation of humans can rise to - but as history has shown this is an unacceptably high bar.

The corporate board can make decisions based on human values so long as it does not go against the rapacious seeking of profit or the CEO will be deposed by the shareholders.

Once a corporation reaches transnational size, nothing can really stop it or even get it to pay tax if it doesn't want to.

I think that much of what people actually fear about the AIpocalypse is exactly the sort of dehumansing powerlessness and machine like cruelty they already experience from corporations and governments.

You may be speaking to a human who empathises but often one suspects they are there to sop up your moans not to help you.

An AI is an amplifier of what we already are, in fearing robots we rightly fear their creator's motives.

You are assuming that there is be an obvious way of doing so, an obvious solution to the control problem.

"We already have a world awash in civilization. This civilization is, of course, termite colonies. Termite colonies are effectively miniature civilizations. They have specialization of labor, they build structures much larger than any one organism, and they go to war with one another. Yet even the nimblest termite colonies can only eat so much."

"That is to say, no matter how much an organism evolves, we would be looking at something that is still hard-limited by its environment, even if it can compete with a termite colony's worth of construction ability. The danger isn't that it spreads through the world and destroys entire ecosystems; the danger is that it finds a supply of raw materials and smashes a termite mound or two while building its own bigger house."

Humans don't respect the social conventions of much less intelligent animals that we've domesticated. If an AI much smarter than humans was created, I don't see strong reasons to believe it would respect our social conventions.

Corporations are only superhuman in the most trivial literal sense that a group of people is (slightly) more intelligent than one person. To make any sort of analogy from this to actual superhuman AI is utterly absurd. A group of people with competing and even contradictory interests is an insanely inefficient way to carry out even rudimentary computations.

For instance, a superhuman AGI will utterly crush an average human corporation at even the simplest task like categorising 40000 images by subject. Let alone something of easy-to-moderate difficulty like hacking into the Pentagon.

Which is to say, that your "This is to say," is nothing but a non sequitur, and that this analogy is a nonsense and an impediment to clear thinking about the capabilities of computers.

Depends on whether the AI is capable of deciding for itself whether it should be open or not.

AI's won't have that emotional baggage.

It will be easier first develop a way of getting around the 'human emotions problem', then likely leapfrog us entirely at the rate a Pareto curve allows.

I can't think outside my human being-ness, so I have no idea what is going to happen when something smarter appears on the planet, except to point out there once were large land animals (ancestors of the giraffe and elephant) on North America until humans arrived.

My fear-based response screams YES MAKE IT OPEN.

However it shakes out, I think it'll be messy for human beings. We're not exactly rational in large groups. The early revs of AI (human controlled) will be used for war.

One has to ask what grows out of that besides better killers?

I suspect emotional range may be the last thing to develop because it's not technically needed to evolve past the point of human intelligence.

As a layman, I so far can only see it in terms of basic philosophy and normative ethics. By definition, a friendly AI is one that doesn't merely deal with facts, but also with "should" statements.

Hume's Guillotine says you can't derive an ought statement from is statements alone. Some folks like Sam Harris disagree but they're really just making strenuous arguments that certain moral axioms should be universally accepted.

Münchhausen Trilemma says that when asking why, in this case why something should or should not be done, you've only got three choices - keep asking why forever, resort to circular reasoning, or eventually rely on axioms. In this case, moral axioms or value statements.

So it seems like any friendly AI is going to have to rely on moral axioms in some sense. But how do you even define what they are? Normative ethics is generally seen to have three branches. For consequentialism (like utilitarianism) you make your decision based on its probable outcome, using some utility function. For deontology, you rely on hardcoded rules. For value ethics, you make decisions based off of whether they align with your own self-perception of being a good person.

But all three have flaws - in consequentialism, it's like putting on blinders to other system effects, and the proposed actions are often deeply unsettling (like pushing a guy off a bridge to block a trolley from killing three others). In deontology and value ethics, actions and the principles they are derived from can be deeply at odds - whether it's hypocrisy in deontology or "road to hell being paved by good intentions" in value ethics. In general, deeply counterintuitive effects can be derived from simple principles, as anyone familiar with systems dynamics knows.

But even beyond that, even if we had a reasonable, consistent AI controlled by solid values, and even if the people judging the AI could accept the conclusions/actions that the AIs derive from those values, how would we ever get consensus on what those values should be? For instance, even in our community there's a fair amount of disagreement among these basic root-level utility functions:

- Maximize current life (people alive today), like Bill Gates believes. - Maximize future life (survival of species) - Maximize health of planet

etc, etc - those utility functions lead to different "should" conclusions, often in surprising ways.

The strategies for mitigating risk seem to be: ensure that the AIs are controllable; avoid situations where there is a single AI (whether controlled or uncontrolled) that is too powerful; and ensure that the AI's goals are broadly acceptable to humankind.

The first and the third objectives are extremely difficult, not just technically, but even from a conceptual standpoint[1]. The second strategy is reasonable, because even if a superhuman intelligence were somehow well controlled, depending on who controls it the outcomes could vary significantly. So perhaps the best thing we can hope for is something similar to society's current status quo-- lots of power concentrated in few hands[2], but without one single (person|corporation|government) being so dominant as to be able to act in opposition to all others.

I am not confident that we will ever be able to produce a provably safe AI, or that we could get even a large majority of the world's population to agree on what a "good AI" might do without devolving into ineffectual generalities[3]. Supposing that resolving these questions is not prima facie impossible, it's not like retarding AI development comes without cost-- just about every facet of our lives can be improved via AI, and so in the years, decades, or centuries between when superhuman machine intelligence is theoretically achievable and the time when we collectively agree we can implement it safely, how many billions will suffer or die from things that we could've solved via AI[4]?

On the whole, OpenAI sounds like a good idea. Making research broadly available helps avoid catastrophic "singleton" like futures, while accelerating the progress we make in the present. In addition, if there's every an AI SDK with effective methods of improving how "safe" a given AI is, most researchers would likely incorporate that into their work. It might not be "proven safe", but if there was a means to shut down a runaway process, or stop it from spreading to the Internet, or alert someone when it starts constructing androids shaped like Austrian bodybuilders, that would be handy. Responsible researchers should be doing this already, but as Scott points out the ones we should be worried about aren't responsible researchers. Open AI development is in harmony with safe AI development, at least in some respects.

------

1. I have a significantly longer response that I scrapped because it might ultimately be better suited as a blog post or some such.

2. That's why it's called a power law distribution. Well, no, that's not it at all, but it seemed like a funny, flippant thing to say.

3. A universally beloved AI might be the equivalent of a Chinese Room where regardless of what message you send it, it responds with a vaguely complimentary yet motivational apothegm.

4. Bostrom tends to counterbalance this by arguing how much of our light cone (the "cosmic endowment") we might lose out on if we end up going extinct, due to, e.g., superhuman machine intelligence. Certainly "all of configurations of spacetime reachable from this point" outweighs the suffering of mere billions of people by some evaluations, but I ask myself "how much do I care about people thousands or millions of years into the future?", and also "if these guys have such a good handle on what constitutes the 'right' utility function, why haven't they shared it?". A more sarcastic variation of the above might be to remark that if they're able to approximate what people want with such high fidelity that they feel comfortable performing relativistic path integration over possible futures, then superintelligence is already here.

------

But nothing is further from the truth. Humans are optimized to be cunning to get positions of power in human society. AI won't be optimized in that way, therefore, it's probably going to lose for a long time. So evil AI will probably be like incredibly annoying autistic psychopath child, who cannot comprehend human institutions so his evil plans are totally obvious.

It's like with grey goo - biological systems like bacteria are heavily optimized to survive in very uncertain conditions, and any potential grey goo has to deal with that.

I think humanity is currently to blow themselves up via global warming, so superintelligence is not really a comparable threat to humanity. If anything, bigger threat is that we won't listen enough to superintelligence. In fact, I think friendly AI will be something like Noam Chomsky - totally rational, right most of the time, fighting for it, telling us what should be done disregarding our emotions. Many people find this annoying, too (including me and many very smart people).

Finally, if the hypothesis about superintelligence is right, why would superintelligence want to evolve itself further? It would be potentially beaten by the improved machine, too.