This reminds me of the Chinese Room argument. Basically, it goes like that:
I never took any Chinese lesson. However, suppose I obtain a huge instruction manual that tells me exactly which sequence of Chinese characters I should use to reply to any sequence of Chinese characters you (a native Chinese speaker) give me. Do I "know/understand" Chinese?
The thought experiment is underspecified in ways that undermine it.
How big is the book?
Seems like a nitpick about a theoretical concession in a thought experiment, right? No, it's actually very important.
If the book contains simple instructions like if you receive character A then reply with B, etc. in a Choose-Your-Own-Adventure style, then it would have to be exponentially large. Too large to be able to carry out more than a few words of conversation, and certainly you would not be able to converse about math, e.g, you're asked what 一 plus 二 is and you say 三, etc. The book would rapidly become larger than the planet Earth with any reasonable conversation depth. This is not just practical problem.
Okay, so the book isn't a simple lookup table, fine. You'll have to have a piece of scrap paper and write down things to refer to them later. But once you do that, it's obvious that you've created a system of memory. It undermines the whole force of the thought experiment. The book was supposed to contain all the brains and you were just supposed to mechanically follow along without understanding what you're doing. But now you're doing complicated things like solving math problems and then converting the answers into Chinese. In order to give the book a finite size, we've given you a lot of work to do, and now it's totally reasonable to say that you do know how to write Chinese. You write Chinese by looking it up, the same way that real translators do!
The Chinese room thought experiment is much discussed but it's a pretty poor thought experiment. It handwaves away all the important parts of language in order to make an inscrutable point about machine intelligence. It neither sheds light on machine intelligence nor language.
Let me ask you another question - when I throw a ball at you your subconcious brain has to solve a differential equation to know how to use muscles to catch that ball.
Do you know how to catch a ball if you don't know the math but still can catch it?
Or the system knows how to translate from English to Chinese. Human language isn't simply about following translation rules, though. It's also about communication and expressing thought. Or participating in language games.
> Let me ask you another question - when I throw a ball at you your subconcious brain has to solve a differential equation to know how to use muscles to catch that ball.
Why suppose the neural network needs to solve differential equations? Is that the only way to learn to catch a ball?
> Why suppose the neural network needs to solve differential equations? Is that the only way to learn to catch a ball?
Yes. You need to decide where to put your hand, how to orient it, etc in reaction to the ball movement.
The answer is a solution to the differential equations, and you cannot consistently get a good answer to an equation if you don't solve it.
The solution probably isn't symbolic but numeric, but that hardly changes anything - you still need a lot of math to consciously solve such an equation numerically.
> Or the system knows how to translate from English to Chinese. Human language isn't simply about following translation rules, though. It's also about communication and expressing thought. Or participating in language games.
human doesn't get to decide what to do in Chinese room experiment - (s)he is just a dumb CPU that does table lookups in a huge book. Every possible response to a sequence of previous messages is already written in that book (so it must be quite big ;) ).
You could completely automate it, remove the human and nothing would change from the outside.
I don’t think because we can solve the problem with differential equations necessarily means that our subconscious mind is solving it in the same way. There’s certainly other possible explanations
> You could completely automate it, remove the human and nothing would change from the outside.
Right, but my point is that following a bunch of translation rules from one language to another is not the same thing as understanding a language. That's not what humans are doing when they use language.
> The solution probably isn't symbolic but numeric, but that hardly changes anything - you still need a lot of math to consciously solve such an equation numerically.
This is assuming the brain is doing math. Even deeper than that, it's assuming that math is something more than a specialized human language. That math exists in nature to be harnessed by neurons.
The Chinese room isn't doing translation, I don't understand why you bring it up. It takes input in Chinese and responds in Chinese.
Also the Chinese room would express emotions, do word games etc, whatever's appropriate in the context. The people doing the lookups wouldn't know that they are writing a joke, but who cares about them?
> This is assuming the brain is doing math.
Well of course brain is doing math, see: 234-123=111 - this is math, my brain did this.
> Even deeper than that, it's assuming that math is something more than a specialized human language
If neurons arriving at solutions to math problems consciously is math, then why neurons arriving at these same solutions subconsciously isn't?
>Do you know how to catch a ball if you don't know the math but still can catch it?
I posited this years ago and the conclusion is what we call "math" is the linguistic expression of what is already intuitive to us.
The other example I used was a mother cat will seek out a stray kitten...but how can she possibly count & keep track of the number of kittens she should have? The answers are at once obvious but deceptively difficult to put into words.
You don't know/understand Chinese. However some argue that the system of you and the instruction manual is conscious, and some other will argue further that the book is an extension of your mind, an external apparatus of thinking, or an organ that are no different from your hand. I vaguely remember some years back that someone was fighting to be allowed having his passport photo with a cyborg antenna because it helps him "hears" colour.
I think everyone is forgetting the writer of the instruction manual in the book. You don't know Chinese any more than any program knows what it's doing, and if we received a invalid input the communication would crash the exact same was as any program. With the exception of George Lucas movies, it's very rare for a human to crash of a non-hardware issue.
I never took any Chinese lesson. However, suppose I obtain a huge instruction manual that tells me exactly which sequence of Chinese characters I should use to reply to any sequence of Chinese characters you (a native Chinese speaker) give me. Do I "know/understand" Chinese?