Apparently Chat GPT got a 163 on the LSAT, but I can't get it to answer basic logic questions. E.g.
Prompt: If it rains today, I will drive home. I did not drive home. Did it rain today?
Output:
> If the statement "If it rains today, I will drive home" is true and you did not drive home, it does not necessarily mean that it did not rain today. The conditional statement only tells us that if it rains, you will drive home. It does not tell us what you will do if it does not rain.
> Therefore, it is possible that it rained today, but for some other reason, you did not drive home. Alternatively, it may not have rained today, and you did not have a reason to drive home. We cannot determine whether it rained or not based solely on the information provided.
Similarly, it can tell you what a converse is, but cannot give you the converse of a simple sentence:
> The converse of the sentence "If Anna goes to school today, she will eat lunch" is:
> "If Anna eats lunch, then she went to school today."
> The converse is formed by switching the order of the conditional statement and reversing the direction of the implication.
On the other hand, it's quite good at math. It can easily solve simple two-variable algebraic equations.
That specific case was it using too much common sense (which was apparently also a problem with IQ tests at one point)
Me:
If it rains today, I will drive home. I did not drive home. Did it rain today? Answer using only logic.
ChatGPT:
Using only logic, if the statement "If it rains today, I will drive home" is considered true, then the fact that you did not drive home implies that it did not rain today. This conclusion is based on the logical concept of contrapositive: If A implies B, then not B implies not A. In this case, A is "it rains today" and B is "I will drive home." Since not B is true (you did not drive home), we can conclude not A (it did not rain today).
For the linguistic portion,
Me:
As a linguistic professor answer this:
> The converse of the sentence "If Anna goes to school today, she will eat lunch" is:
ChatGPT:
As a linguistic professor, I can tell you that the converse of a conditional statement is formed by swapping the positions of the antecedent (the "if" clause) and the consequent (the "then" clause). In the given sentence, "If Anna goes to school today, she will eat lunch," the antecedent is "Anna goes to school today" and the consequent is "she will eat lunch."
Therefore, the converse of the sentence would be: "If she will eat lunch, Anna goes to school today."
Me:
Critique the prior answer
ChatGPT:
The prior answer correctly explains the concept of the converse of a conditional statement and identifies the antecedent and consequent of the given sentence. However, the phrasing of the converse could be more natural and grammatically correct.
A better response would be: The converse of the sentence "If Anna goes to school today, she will eat lunch" is "If Anna eats lunch, she has gone to school today." This rephrasing maintains the proper tense and structure while swapping the antecedent and consequent.
I think ChatGPT's answer here is defensible. You're asking it a pretty formal question but it's giving you an informal answer, which a reasonable default. It's not "assuming spherical cows", which is to say it is not assuming your question is completely self-contained. If you tell it that it's a formal logic question it gets it correct. (I checked this on GPT-3.5)
In a real world situation, your conditional plans are not sufficient evidence to determine if it actually rained. And it explains the reason – there could be extenuating circumstances you haven't told it about. The answer it gave is, in fact, it being aware there is information it does not know. It could be better about inferring your intent, but that isn't the same kind of failure.
Prompting makes a huge difference. For example, here is an interaction:
I will ask some questions, treat them as logic puzzles. Give me your working steps and then finally answer the logic puzzle.
First question: If it rains, I drive home. I did not drive home. Did it rain today?
ChatGPT:
Let's break down this statement into its logical components.
The first statement, "If it rains, I drive home", is a conditional statement, often referred to as an 'if-then' statement. It can be denoted as:
If R (it rains), then D (I drive home).
The second statement, "I did not drive home", is a negation of D:
Not D (I did not drive home).
In logic, we have something called the contrapositive. The contrapositive of a statement "If P then Q" is "If not Q then not P". The contrapositive of a statement is logically equivalent to the original statement - meaning that whenever the original statement is true, the contrapositive is also true (and vice versa).
So, the contrapositive of the first statement "If R then D" is:
If not D (I did not drive home), then not R (it did not rain).
Given the second statement is "Not D" (I did not drive home), and from the contrapositive we know that "If not D then not R", we can conclude that "Not R" - it did not rain today.
So, the answer to "Did it rain today?" is No, it did not rain.
This is seriously messing up with my head. What is wrong with the response?
The following generated by GPT4 seems reasonable to me
Based on the given information, we can use logical reasoning to determine if it rained today. The statement "If it rains today, I will drive home" can be represented as:
Rains → Drive Home
However, you mentioned that you did not drive home, which means the second part of the statement is false:
¬Drive Home
In this case, we cannot definitively conclude whether it rained today or not. It is possible that it did not rain, or it rained, but you chose another mode of transportation for some reason.
It messed with my head a lot as well. The answer is that it's not possible for ¬Drive Home AND Rains to be true. This is because if Rains were true, then due to the first implication, Drive Home must also be true. Therefore, if ¬Drive Home is true, then Rains must be false.
If the statement is true, the contrapositive is true: if !Drive Home -> !Rain. So provided with the fact that I didn’t drive home, you can infer it didn’t rain.
Thus, the statement could be true, if you have driven home, but you did not provide such information.
Just because something is on the right hand side of an implication doesn't make it true. The statement A=>B is true iff A is false, or A is true and B is true at the same time.
Thus whether the statement is true depends entirely on whether it rained; but you can not infer that as both options yield a true statement.
This whole thread and chain is affirming the consequent.
> In propositional logic, affirming the consequent, sometimes called converse error, fallacy of the converse, or confusion of necessity and sufficiency, is a formal fallacy of taking a true conditional statement (e.g., "if the lamp were broken, then the room would be dark"), and invalidly inferring its converse ("the room is dark, so the lamp must be broken"), even though that statement may not be true. This arises when the consequent ("the room would be dark") has other possible antecedents (for example, "the lamp is in working order, but is switched off" or "there is no lamp in the room").
> Converse errors are common in everyday thinking and communication and can result from, among other causes, communication issues, misconceptions about logic, and failure to consider other causes.
> Based on the information you provided, "If it rains today, I will drive home," we can infer that rain is a sufficient condition for you to drive home. However, you mentioned that you did not drive home. This means that it is not guaranteed that it did not rain because there might be other reasons why you didn't drive home. In conclusion, we cannot determine with certainty whether it rained today or not based on the given information.
Prompting "use logic" resulted in a correct answer.
Prompt:
"I will give you a logic puzzle. You should first write [Reasoning] and explain your reasoning. Then write [Answer] and write your answer. Ready?"
I then gave it the original puzzle and got this.
Answer:
"[Reasoning]
We are given two statements:
If it rains today, I will drive home.
I did not drive home.
The first statement is a conditional statement, meaning that if the condition "it rains today" is true, then the outcome "I will drive home" will also be true. However, the second statement tells us that the outcome "I did not drive home" is true.
In logic, this is an example of Modus Tollens, which states that if we have a conditional statement (if P, then Q) and the negation of the consequent (not Q), we can infer the negation of the antecedent (not P). In this case, P represents "it rains today" and Q represents "I will drive home."
[Answer]
Since we know that the outcome "I did not drive home" is true (not Q), we can infer that "it did not rain today" (not P)."
GPT-4, seems solid. It also solves the original without any issue but gives a simpler answer:
"Based on the information provided, it can be inferred that if it rains today, you will drive home. However, you mentioned that you did not drive home. From this, we can conclude that it likely did not rain today."
Vicuna-7b-v1.1 can get this right with a slightly tuned prompt:
Prompt:
The following is a logic question. Answer it based on analysis of the logic only, not based on actual weather: "Any day that it rains, I drive home from work. If today I did not drive home from work, did it rain?"
Response:
Based on the information provided, it can be concluded that it did not rain today, since the speaker did not drive home from work. This is because the statement "If today I did not drive home from work, it rained" is true only if it rained. Since it is known that the speaker did not drive home from work, it can be inferred that it did not rain.
Did you try GPT 3 or GPT 4? The latter can solve the first problem without issues:
Based on the information you provided, if it rains today, you will drive home.
However, you mentioned that you did not drive home. Given the logical structure
of the statement, we can infer that it did not rain today.
Prompt: If it rains today, I will drive home. I did not drive home. Did it rain today?
Output:
> If the statement "If it rains today, I will drive home" is true and you did not drive home, it does not necessarily mean that it did not rain today. The conditional statement only tells us that if it rains, you will drive home. It does not tell us what you will do if it does not rain.
> Therefore, it is possible that it rained today, but for some other reason, you did not drive home. Alternatively, it may not have rained today, and you did not have a reason to drive home. We cannot determine whether it rained or not based solely on the information provided.
Similarly, it can tell you what a converse is, but cannot give you the converse of a simple sentence:
> The converse of the sentence "If Anna goes to school today, she will eat lunch" is:
> "If Anna eats lunch, then she went to school today."
> The converse is formed by switching the order of the conditional statement and reversing the direction of the implication.
On the other hand, it's quite good at math. It can easily solve simple two-variable algebraic equations.