There is at once nothing “wrong” with this response, and it is ridiculous. It is not a question — which without considerable pre-hedging — a person could consider seriously and most would assume an alternative hypothesis for why it is being asked.
Pasting huge prompts into HN comments is really irritating. Make whatever point you’re trying to make without doing that.
It indeed does tend to assume that you’re speaking hypothetically and for a potentially fictional purpose. Which is a better approach, as far as I’m concerned. I prefer that to it constantly being a nanny that questions everything.
I’ve been writing lots of Prolog recently and asking ChatGPT questions. Many of my questions have been sincere but a bit like the bear trainer — ridiculous for someone who knows what they are doing. Meanwhile ChatGPT will answer it as if the premise is valid. The answer is valid sounding nonsense, which may lead you on a wild goose chase — a bit like the aspiring space bear trainer.
It isn’t assuming that I’m asking from a “hypothetical and for a potentially fictional purpose”. If GPT is in effect a conditional probability distribution over tokens, it isn’t “assuming” at all.
This IMO is a clear challenge to sense making for ChatGPT which is not obviously fixable through fine tuning. I don’t think factfulness is either because low contrast examples are hard to train for, especially if they are compounds of true things . Eg “tell me about logic regression”
In my experience, if you just tell it to do something first, (e.g., "before answering this question, tell me if it makes logical sense") it'll generally do it. Giving it a one sentence, vague prompt isn't going to be useful regardless.
The fine-tuning aspect I meant was mostly about factual data being incorrect.
