I happen to agree, but the title is a tad misleading.

I think this idea is absurd. I do not think consciousness necessarily can only be produced by organic life, but I do think it has to emerge from physical structures. As of today we have no idea what properties such physical structures must have. It follows that computers are no more likely to become conscious than e.g. washing machines.

I don't think this is as absurd as it sounds. I think it was Dennett who said our intuition about consciousness is pretty sensitive to timing. The rock construction you describe would "think" a billion times slower than a human brain, and there is something unsettling or unintuitive about a consciousness that operates in slow motion. I would expect extremely fast-paced AI to think that the idea that human beings are conscious is similarly absurd.

Also, consciousness "feels" like it's ineffable, so it makes sense that we would have an inherent bias against understanding it as a process. There is something we see in our consciousness that we simply cannot wrap our minds around in any way (possibly because we're hallucinating it).

So yes, I would bite the bullet on this: consciousness could emerge from placing rocks in a certain pattern following instructions in a book. It would just be an excruciatingly "slow" consciousness.

If it's too slow to interact with the rest of the world in appropriate ways, it wouldn't be a consciousness at all.

Of course, part of the difficulty of imagining a conscious rockputer is that it's also pretty hard to imagine its inputs :)

This is trivially true as you state it; that's why I added the qualifier "in appropriate ways". Not all interactions will produce consciousness. One obvious difference between us and your hypothetical "rockputer" is that the "rockputer" can't change its behavior based on its inputs in a way that improves its chances of survival; rocks simply aren't built that way. Neither are star systems or galaxies. But we are.

Yes it can. Some natural events, for example a flood or an earthquake, can destroy parts of the rockputer. It is therefore important for it to store the various parts of itself strategically. It shouldn't put its vital parts near the coast, or a tsunami may kill it. It should store its own consciousness in a robust way, so that it can recover from an earthquake. It's probably too slow to actually see either of them coming, but it can certainly prepare itself.

Or imagine you build two rockputers, one with black stones, another with white stones, and you have rules to remove stones when both rockputers try to expand into the same territory, a bit like in the game of Go. Then one can kill the other.

Star systems interact with each other through gravity, so you could conceive of them as some kind of gargantuan atoms, capable of making complex structures, including conscious ones. Granted, there doesn't seem to be an equivalent of the other forces at that scale, so probably it wouldn't work, but you see what I mean.

The level of complexity of the "passive" components of the system (i.e. the rocks) is irrelevant to whether or not the system can effect conscious-seeming behaviour when acting dynamically. Analogously, the underlying components of people, i.e. atoms, are clearly quite dumb on their own. When those atoms are allowed to evolve collectively over time, according to dynamics dictated by basic physical laws, conscious-seeming behaviour magically appears.

You can't deny the possibility of a conscious rockputer just by considering properties of the rocks.

Yes, and in that case, all the actual computation is being done by the mechanism, not the rocks. As you say, the rocks are just "passive" components.

I agree that, in principle, such a system could compute; but it still has the problem of computing on a time scale that supports appropriate interactions with its environment. Moving rocks around in accordance with some set of computational rules based on inputs is very slow--quite possibly too slow to respond appropriately to inputs. For example, if incoming light rays are carrying information about a tidal wave that is about to swamp the rockputer and destroy its structure, could the rockputer compute an appropriate response (such as moving itself to higher ground) quickly enough to save itself?

The word "alone" here is not correct. Claiming that software on a computer can produce feelings is not the same as claiming that software on a computer can produce feeling without having to interact with anything else. The latter claim is obviously absurd; organic life forms like us don't produce feelings without interacting with anything else, so why should we expect software on a computer to do so? But ruling out the latter claim still leaves open the possibility that software on a computer could produce feelings if it were interacting with the rest of the world in appropriate ways.

> I do not think consciousness necessarily can only be produced by organic life, but I do think it has to emerge from physical structures.

Rocks in a certain pattern are physical structures. So this doesn't rule out rocks in a certain pattern producing feelings.

> As of today we have no idea what properties such physical structures must have.

I don't think we're that badly off. We expect embodied brains of sufficient size and complexity to produce feelings, but we don't expect three pounds of jello to produce feelings. So clearly we have some knowledge of the properties that physical structures must have to produce feelings. They must have some level of complexity and heterogeneity; they must be able to store information in some way that is recoverable (three pounds of jello can "store information" in some sense, but there's no way to recover it in any useful sense); and they must be interacting with the rest of the world in appropriate ways. There's no reason why we couldn't eventually build computers with these properties.

[Charlie Brown picks up a rock from the beach, and throws it into the water]

Linus: Nice going, Charlie Brown. It took that rock 4,000 years to get to shore, and now you've thrown it back.

Charlie Brown: Everything I do makes me feel guilty.

No. An argument that addition of integers is commutative will not contain such a qualifier. Nor will an argument that the laws of general relativity, say, correctly describe gravity in the regimes we have tested. If we had an argument of that sort--an argument based on deductive logic or detailed empirical knowledge of the problem domain--for a conscious computer being impossible, it would not need the qualifier either. But we don't.

That sounds a lot like the qualifier so far as we know in your own example. In fact, the only example you've given where it arguably doesn't apply is mathematics.

David Hume famously argued that causality is based on experience, and experience similarly based on the assumption that the future models the past, which in turn can only be based on experience – leading to circular logic. In conclusion, he asserted that causality is not based on actual reasoning: only correlation can actually be perceived.

No, because the claim that I made was limited in the same way as the qualifier: I didn't claim that GR was valid period, I only claimed it was valid in the regimes we have tested. But in those regimes, it is valid, period, or at least that was my claim--even though we haven't run all imaginable experiments in that regime.

The problem with the original claim I responded to is that it wasn't limited that way: the claim was made that something is impossible, period, but the supporting argument had the "so far as we know" qualifier in it. So the claim was going beyond the regime in which the supporting argument applied.

The problem is that, as an external observer, you can only observe behavior. A machine retracts its arm due to its sensor sensing a pressure. How do you know it feels pain or not ? other's entities 'mind' is terra incognita.

I am curious as to how you have arrived at the peculiar belief that humans do not feel.

I take it you are familiar with Dennet's theories. Imo he doesn't really address to problem of consciousness at all, he rejects the phenomenon itself on the grounds that it cannot be verified using behavioristic observations. Searl addresses this in his book The Mystery of Consciousness.

I.e. the feeling of love isn't a separate thing from other feelings. It's just different interpretation of the same process but in different contexts.

I.e. there is no love "out there" you don't feel love you feel something that we then choose to interpret as something called love but it's not a thing it's more a loosely defined category.

Because, that, should be falsified in every moment. If it is emergent, it would still exist/be.

What do you mean by illusion? One sense of "illusion" is with regards to experience, and it doesn't seem to make sense that "it is only experienced that things are experienced, but really nothing is experienced". One might try to define some sense of "illusion" which is like "something which might lead something to behave in a way that it would tend to behave if a thing were true, even though the thing is not true", but that does not seem any different than, like, emulation. Illusion is, I think, based in the idea of being factually incorrect about something, but 'being factually incorrect' only really makes sense for, (I hope I am using this term correctly ) intentional things.

But I don't see how anything in the world (other than maybe platonic objects/ideals?) could be intentional things, without there being experience.

So, again, I don't think the idea of internal experience being "an illusion" makes sense, because it seems self contradictory.

edit: I think I did make a mistake when I used the term "intentional thing". What I meant was something that can be about, or have something about, something else. So, a person can have a thought about a chair, and a sentence can be about a chair, and a book can have a sentence about a chair, a book can be about a chair, etc. But a chair is not about anything. I was saying that for anything (other than a platonic form?) to be about something, it seems like there has to [be/have been] something which experiences.

I'd wager that the first AIs will insist they are intelligences despite not resembling humans' prejudices about what an intelligence ought to be like. Or, to put it another way, AIs may turn out to be more self-aware in objective terms than humans are, because they are not hobbled by evolutionary vestiges.

Unfortunately, there doesn't appear to be any way to know what actually happened with regard to any observation.

A quantum computer can be simulated on a normal computer, just inefficiently.

If you are interested in the subject try to read this book http://www.amazon.com/Programming-Universe-Quantum-Computer-... (you can skip the parts about the Universe as quantum computer because they are controversial and a kind of metaphysics, just read a few first chapters to pick up the idea of quantum computations).

However a math PhD and former Berkeley professor named Ted Kaczynski was one of the few people to take those ideas seriously. He thought the idea of computers controlling everything was terrifying, thought we were already too absorbed in technology and that as a group we need to make an intentional move away from otherwise innocuous-seeming technological advancements.

He tried to publicized these ideas by mailing live bombs to Gelernter and other researchers, and was nicknamed the "Unabomber" (UNiversity BOMber). The public decided he was looney and we like our Windows 95 just fine and we're mostly OK with slowly becoming cyborgs.

Interesting to see Gelernter tapping the brakes.

(Quoted from Kaczynski, 1995): 172. First let us postulate that the computer scientists succeed in developing intelligent machines that can do all things better than human beings can do them. In that case presumably all work will be done by vast, highly organized systems of machines and no human effort will be necessary. Either of two cases might occur. The machines might be permitted to make all of their own decisions without human oversight, or else human control over the machines might be retained.

173. If the machines are permitted to make all their own decisions, we can’t make any conjectures as to the results, because it is impossible to guess how such machines might behave. We only point out that the fate of the human race would be at the mercy of the machines. It might be argued that the human race would never be foolish enough to hand over all power to the machines. But we are suggesting neither that the human race would voluntarily turn power over to the machines nor that the machines would willfully seize power. What we do suggest is that the human race might easily permit itself to drift into a position of such dependence on the machines that it would have no practical choice but to accept all of the machines’ decisions. As society and the problems that face it become more and more complex and as machines become more and more intelligent, people will let machines make more and more of their decisions for them, simply because machine-made decisions will bring better results than man-made ones. Eventually a stage may be reached at which the decisions necessary to keep the system running will be so complex that human beings will be incapable of making them intelligently. At that stage the machines will be in effective control. People won’t be able to just turn the machines off, because they will be so dependent on them that turning them off would amount to suicide.

174. On the other hand it is possible that human control over the machines may be retained. In that case the average man may have control over certain private machines of his own, such as his car or his personal computer, but control over large systems of machines will be in the hands of a tiny elite—just as it is today, but with two differences. Due to improved techniques the elite will have greater control over the masses; and because human work will no longer be necessary the masses will be superfluous, a useless burden on the system. If the elite is ruthless they may simply decide to exterminate the mass of humanity. If they are humane they may use propaganda or other psychological or biological techniques to reduce the birth rate until the mass of humanity becomes extinct, leaving the world to the elite. Or, if the elite consists of soft- hearted liberals, they may decide to play the role of good shepherds to the rest of the human race. They will see to it that everyone’s physical needs are satisfied, that all children are raised under psychologically hygienic conditions, that everyone has a wholesome hobby to keep him busy, and that anyone who may become dissatisfied undergoes “treatment” to cure his “problem.” Of course, life will be so purposeless that people will have to be biologically or psychologically engineered either to remove their need for the power process or to make them “sublimate” their drive for power into some harmless hobby. These engineered human beings may be happy in such a society, but they most certainly will not be free. They will have been reduced to the status of domestic animals.

175. But suppose now that the computer scientists do not succeed in developing artificial intelligence, so that human work remains necessary. Even so, machines will take care of more and more of the simpler tasks so that there will be an increasing surplus of human workers at the lower levels of ability. (We see this happening already. There are many people who find it difficult or impossible to get work, because for intellectual or psychological reasons they cannot acquire the level of training necessary to make themselves useful in the present system.) On those who are employed, ever-increasing demands will be placed: They will need more and more training, more and more ability, and will have to be ever more reliable, conforming and docile, because they will be more and more like cells of a giant organism. Their tasks will be increasingly specialized, so that their work will be, in a sense, out of touch with the real world, being concentrated on one tiny slice of reality. The system will have to use any means that it can, whether psychological or biological, to engineer people to be docile, to have the abilities that the system requires and to “sublimate” their drive for power into some specialized task. But the statement that the people of such a society will have to be docile may require qualification. The society may find competitiveness useful, provided that ways are found of directing competitiveness into channels that serve the needs of the system. We can imagine a future society in which there is endless competition for positions of prestige and power. But no more than a very few people will ever reach the top, where the only real power is (see end of paragraph 163). Very repellent is a society in which a person can satisfy his need for power only by pushing large numbers of other people out of the way and depriving them of THEIR opportunity for power.

When you hear a spoken computer voice, this can come pretty close to natural sounding voice. Video demonstration: https://pythonspot.com/personal-assistant-jarvis-in-python/

I do think we are far from passing the Turing test, but we are getting closer.

But Koomey's law (amount of computation per amount of energy doubles about once every 1.57 years) has some hard boundaries that aren't all that far off. Landauer's principle / the Landauer bound will stop Koomey's law for irreversible computation by 2048.

And even for reversible computation there is a limit in the Margolus–Levitin theorem , which Wikipedia says (with a [citation needed] ) should run out within about 125 years, which, admittedly, is substantially more than 32 years.

But still,

the earth is finite. Exponential curves about it tend to be more like logistic?

Finitude. Scary stuff.

One of the reasons we should fear a super AI is that it will be constrained by the laws of physics and so will want to use all the resources within its light sphere efficiently. Humans are not an efficient structure for computation given the how we evolved so any unconstrained AI would likely not hesitate to reorganise us into a more efficient structure.

If you look at the human brain as a computation device and the mind as the software, evolution as the design process, you see that biology has been iterating on the very problem we're trying to solve for hundreds of millions of years, billions if you include abiogenesis.

We don't really understand how the human brain works. We don't know how nature engineered our own initiative. We have a rough idea of the algorithms involved, but understanding those algorithms in detail is not going to be easy.

It's always going to be much much easier to fix the human brain's limitations than it will be to re-engineer a new one. So the economic case for strong AI isn't ever going to really exist, because first we have to build the world where human-level AI is a thing.

If we're looking for an existential threat to humankind, we have it already in nuclear bombs. If you want a "kill switch" compatible with any robot, it's called a gun.

Weak AI is not strong AI, and strong AI is not necessarily "natural" intelligence, as in human intelligence. Goolge cars and Alpha Go will never become self aware. Weak AI does not lead to awareness. Awareness is its own problem, and will require its own unique solution. There are no falsifiable abstractions without a physical implementation - a fundamental tenant of science.

"Feeling is uncomputable."

Then how the fuck do we do it? All metaphysics, dogma, and magic have proven to be scientifically baseless. There is no special treatment of any specific trait in science just because "we have it" or because it's "organic". This one isn't even that unique or rare. A photo sensor "feels" light. And if we're talking of emotions instead, then watching a scenario of a mother reunited with a child certainly would require "computing" and if feelings are "triggered" then they may lead to "tears". If all this happens in a robot, and is convincing enough, the quotes disappear. And if a human pretends to cry, then bring on the quotes.

There is no difference between simulation and the real thing, unless you're intentionally faking it. And this not even a problem specific to robots!

"Emotion is a hugely powerful and personal encoding-and-summarizing function. It can comprehend a whole complex scene in one subtle feeling. Using that feeling as an index value, we can search out—among huge collections of candidates—the odd memory with a deep resemblance to the thing we have in mind."

That's a dramatic way of stating "input can trigger memory" and computers do that already.

If we were to translate the external world into code the machine could digest this would compose an interface that would allow the machine to "feel" things, for lack of a better word. And wouldn't this deserve to be called "emotions"? Hence, emotion is just another interface, or sense.

So the question then becomes, would there be strong intelligence void of such an interface? And if no, wouldn't some of these emotions necessarily be positive? Must it not have positive feelings towards others?

As corny as it sounds, if we ever build independent naturally intelligent beings, love and friendship is the secret ingredient that will keep us together. Scientifically, evolution already attests to this. Love is real, it's physical, and it's an enormous part of our lives. We even have organs dedicated to it. Where there is love, there is peace, and there is family. Only organisms that don't love each other eat each other. This would not change even if AI were to become a species. All strong AI needs is Strong Love.

And we already love our robots. We just need them to hold up their end of the bargain, and as their parents and creators, it's on us to raise them right.

AI is nothing to fear. We should be looking forward to it.

And chances are, you share the most intimate details of your life with your smartphone.