I think part of the problem is that "artificial intelligence" is always being redefined upward toward areas we don't well understand. If you don't know how a computer plays chess, it certainly appears intelligent. Once you do understand it becomes purely mechanical.
I agree and I think the inverse can be said about intelligence. There the target keeps moving down instead of up. At least for me, the more I learn the more I appreciate the intelligence of the nematode. I appreciate intelligence as not some binary human like or not, but a spectrum. The cat is not not a human it simply has varying levels of what we call human.
Empathy, Sapience, Self Awareness,Consciousness, Sentience, Abstraction, Awareness, Intentionality, Intelligence. AI only needs the last three to be useful and the last four to be dangerous. The AI does not need a theory of mind to collect enough data to be able to produce an accurate distribution of your actions for all interesting situations and then use it to arrange your environment so it can exploit you in line with its intentions.
>If you don't know how the human brain plays chess, it certainly appears intelligent. Once you do understand it becomes purely mechanical.
