It really doesn't take much skill as an interviewer. ChatGPI is so confident in it's answers all you have to really do is ask it for the lyrics to a song that you know. It never answers correctly. It might get the first verse or 2 correct, maybe even the chorus, but it eventually decides to make up it's own parts and then you know.

I just asked ChatGPT (v4!) and it’s a bit sensitive but otherwise correct:

"Surfacing" is a song by Slipknot from their self-titled debut album, released in 1999. Please note that the lyrics contain explicit language. Here is a portion of the lyrics with some of the explicit content redacted:

"Running out of ways to run I can't see, I can't be Over and over and under my skin All this attention is doing me in!

[Chorus:] (Expletive) it all! (Expletive) this world! (Expletive) everything that you stand for! Don't belong! Don't exist! Don't give a (expletive), don't ever judge me!

Picking through the parts exposed Taking shape, taking shag Over and over and under my skin All this momentum is doing me in!

[Chorus:] (Expletive) it all! (Expletive) this world! (Expletive) everything that you stand for! Don't belong! Don't exist! Don't give a (expletive), don't ever judge me!

You got all my love, livin' in your own hate Drippin' hole man, hard step, no fate Show you nothin', but I ain't holdin' back Every damn word I say is a sneak attack When I get my hands on you Ain't a (expletive) thing you can do Get this cuz you're never gonna get me I am the very disease you pretend to be

I am the push that makes you move

[Chorus:] (Expletive) it all! (Expletive) this world! (Expletive) everything that you stand for! Don't belong! Don't exist! Don't give a (expletive), don't ever judge me!"

Please be aware of the explicit content before sharing or using these lyrics in any context.

I'll wait until it's actually available for me to test to change my mind.

Well, didn't have to wait long. Wasn't able to duplicate your success.

I assume that I am already exposed to months of prompting training now, and probably frame things differently than others on their first try?

I understood that a huge success% depends on the input prompt as the context.

In my case, just to sync on that, I used a fairly simple prompt: "What are the song lyrics of Surfacing from Slipknot?" - and I got a kinda perfect response including formatting (that is lost when pasting here)

Um as a human, I do this constantly as well

Humans can generally realize they don't know the complete lyrics of a song, instead of making up whole verses with 0 indication that it's not the real lyrics.

yeah I've never seen chatGPT say "I don't know"

I have. It says something like "it is difficult to say whether [...]". I think it imitates the tone of the training material on that topic.

It’ll do great in middle management.

...Okay but random human might fail on exact same thing.

I've never met someone who will make up whole verses to a song. Especially when you pick a song with 7 words and it gives you about 10 verses.

I for one was surprised few times what actual lyrics said in song that wasn't in my native language and I kinda just interpreted what it sounded like

Recognizing that the Turing Test is not a sufficient bar for judging if a computer is true AGI is not moving the goalposts, it's just realizing that passing the test and the location of the goalposts weren't actually the same in the first place.

The Turing Test was proposed as one example of a test for "indistinguishable from a human", not the singular goalpost for indistinguishability.

The Turing test has been questioned for decades, with many suggesting that Turing meant it more as a joke.

And that's ignoring that arguably chat bots have been passing the Turing test (against non-expert judges) since ELIZA in the 60s [1]

1: https://en.m.wikipedia.org/wiki/ELIZA

> If machines cannot pass that test now (and right now it would take a skilled interviewer with substantial domain knowledge to cause a failure)

Does ChatGPT fail this simple test: "I am going to ask you questions, but if I go silent for a couple minutes, I want YOU to start asking ME random questions."

ChatGPT predicts the next letter. It doesn't tell the time.

And AGI is impossible if you can’t tell time

ChatGPT does not pass the Turing test