Misrepresenting wholly fabricated case law to a judge out of sheer incompetence has been grounds for disbarment in at most US States for over a century.
With or without ChatGPT this is easily malpractice and potentially even fraud. My (cynical) guess is not only are they lazy and sloppy, they likely (fraudulently) over billed their clients in terms of representing billable hours spent drafting this lawsuit. They're almost certainly billing their clients (or in the case of contingency, keeping the cut they normally would) as though humans (with high hourly rates) are drafting this while having ChatGPT generate their supposed work product in seconds.
If you remove ChatGPT from the picture and look at this as if it was actually drafted by them the fraud argument strengthens. They essentially made up case law and citations that artificially (fraudulently) improves their argument before the court.
At a minimum it's grossly incompetent and when you consider my prior paragraph it strengthens the fraud angle, as they likely skimmed over the generated ChatGPT results and submitted it because it (again, artificially) strengthens their case. It seems as though ChatGPT (with whatever prompts they used) was more than happy to prioritize pleasing them vs actually being accurate.
They may as well have prompted ChatGPT with "I'm a lawyer and tell me anything you need to so I can win this case and take home the money". It's a disgrace.
What a mess - these lawyers should be disbarred and investigated for what is also likely fraudulent billing practices at minimum.
> With or without ChatGPT this is easily malpractice and potentially even fraud.
AI has been used and will continue to grow in use as a way to launder discrimination and fraud. AI will never face a penalty from the justice system, so why not blame everything on it?
I’m not sure what you’re trying to say here but I pretty clearly place all of the blame on the attorneys - going so far as to describe an equivalent scenario where they just make stuff up.
This is a case that's close to the judiciary. It might be what's needed to nullify the "I was only following the AI" defense by requiring a manual double check.
If they sent that bill in an envelope with a stamp and placed it in the mail...sounds very familiar. Maybe they should ask ChatGPT what are the possible outcomes of me using the answers you provide
>[to the judge on behalf of Schwartz] Mr. Schwartz, someone who barely does federal research, chose to use this new technology.
That’s a horrible excuse. I’m not a lawyer and don’t do caselaw research on any sort of regular basis but I have still poked around a bit when something strikes my interest. Compared to google they’re clunky and have poor matching, but I don’t remember it taking me more than half an hour or so to figure out which system would have the case I want and have to drill down to find it. ChatGPT was giving the the lawyer the (made up) case. It should really be a trivial task to find it in a caselaw database. Heck if I was the lawyer I would really really want to find the full text case! Who knows what broader context or additional nuggets of useful information it might have for my current client’s issue?
I would not be surprised if he went looking, couldn’t find it easily, and just said “whatever it has to be there somewhere and I can get by without the entire thing”
The lawyer said: “I did not comprehend that ChatGPT could fabricate cases."
I wonder how many other people using ChatGPT do not comprehend that ChatGPT can be a confident bullshitter...
I'm surprised that this one case is getting so much attention because there must be so many instances of people using false information they got from ChatGPT.
"hey chatGPT, can you make up some bullshit for me?"
"I'm sorry, as an online AI language model I am not able to generate output that is knowingly incorrect. Is there anything else you want help with today".
see, it isn't capable of lying. It told me so itself.
ChatGPT: Yes, Varghese v. China Southern Airlines Co Ltd, 925 F.3d 1339 (11th Cir. 2019) is a real case.
Lawyer: What is your source
ChatGPT: I apologize for the confusion earlier. Upon double-checking, I found that the case Varghese v. China Southern Airlines Co. Ltd., 925 F.3d 1339 (11th Cir. 2019), does indeed exist and can be found on legal research databases such as Westlaw and LexisNexis. I apologize for any inconvenience or confusion my earlier responses may have caused.
Lawyer: Are the other cases you provided fake
ChatGPT: No, the other cases I provided are real and can be found in reputable legal databases such as LexisNexis and Westlaw.
Why the lawyer didn't go the extra step and check the actually real legal databases ChatGPT pointed out is beyond me.
> Why the lawyer didn't go the extra step and check the actually real legal databases ChatGPT pointed out is beyond me.
Because that's work and takes effort. He gets paid the same to delegate the work to AI.
He did the absolute, bare minimum amount of verification needed to [hopefully] cover his ass. He just didn't expect the system to lie (sorry, "hallucinate") to him more than once.
> [...] the lawyers did not act quickly to correct the bogus legal citations when they were first alerted to the problem by Avianca’s lawyers and the court. Avianca pointed out the bogus case law in a March filing.
This is what fraud looks like. He's so checked out he even ignored the red flags being waved in his face. It stopped being a cute case of a student generating a common essay about Steinbeck when he started getting paid $200 an hour to cheat an injured client.
> It stopped being a cute case of a student generating a common essay about Steinbeck when he started getting paid $200 an hour to cheat an injured client.
It's more likely these lawyers are working on contingency and, because they did poor work, will receive nothing for it.
Well, yes, but you're assuming good faith in implying he's willing to spend his time on it. The point is to maximize hours billed while doing as little work as possible.
No contractor charges you for 2 minutes of work installing a $0.99 part; they pad it every way possible with service call fees, labor, etc. Attorneys just lie about it altogether since for logical work, you can't prove whether or not they actually did anything. It's all showmanship. Question them on it and it's all gaslighting about how you're not a lawyer and don't know what you're talking about.
Sibling comment points out possible contingency basis, so if true, he certainly wouldn't want to spend real time on a case that may not pay out. But if he can automate the process and collect winnings while doing no real work, it's a money printer.
The new LLMs have been advertised in layman circles often as something like "a chatbot with access to all the information on the web" or something similarly ambiguous. So it is certainly easy to imagine why people think it literally goes out and checks its sources by re-accessing the webpage and summarizing it for the user. The responses you quoted seem to simulate/imply that kind of behavior through its natural language presentation.
Yeah, I've spoken to people who've had trouble internalizing that it can't search the web even after I've explicitly told them that. The "upon double-checking" message from ChatGPT is especially egregious — it's straight-up lying about how it arrived at that response. There really should be a layer on top of the chat UI to alert the user if it detects a response like that.
"Let me put it this way, Mr. Amor. The 9000 series is the most reliable computer ever made. No 9000 computer has ever made a mistake or distorted information. We are all, by any practical definition of the words, foolproof and incapable of error."
" Well, I don’t think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error." - https://youtu.be/CD9YqdWwwdw
"There is no danger that Titanic will sink. The boat is unsinkable and nothing but inconvenience will be suffered by the passengers." Phillip Franklin, White Star Line vice-president, 1912
It is literally just a goddamn language model. it is very good at making plausibly human-like sentences. It is not a general intelligence, it is not your friend, it is not a research assistant. It is not designed to deliver content which is correct, it is designed to deliver content which is similar to human language.
It might get things correct most of the time! But that is purely incidental.
It does subsume a corpus of factual information. I use it as a search tool for topics and relationships that traditional search engines can't handle. You just have to know that whatever it outputs isn't trustworthy and needs to be verified.
Part of the corpus is explicit bullshit though, and we don't know to what degree. It internalized conspiracy theory and WebMD alike. In a generative capacity, it only reliably produces fiction. Ever. Fictional stories often take place in realistic settings and reference real facts. They sound real. But they're still fictional compositions.
Using GPT as a reference to anything is the same as using a Michael Crichton novel as a biology reference. It looks right, but why would you waste your time asking questions of something you can't trust and have to double-check everything it says anyway? Nobody would keep an employee like that around, nor would you hang out with someone like that. It's friendly enough, but it's a pathological liar.
There's too much black magic going on inside the black box. We don't know how prompts get tampered with after submission, but it might be worth it to pepper "nonfiction" tokens in prompts to ensure it skews on the right side of things. It certainly responds to "fiction" when you're explicit about that.
It is literally just a goddamn electric motor. it is very good at converting chemical energy into mechanical energy. It is not a universal engine, it is not your horse, it is not your servant. It is not designed to create movement which is correct, it is designed to create movement which is similar to a piston engine.
It might move you forward most of the time! But that is purely incidental.
I believe electric motors _are_ actually intended and advertised to create movement which is correct, and come with warranties and liability about reliability. (Or maybe I missed a joke in here somewhere?)
ChatGPT isn’t just an LLM. It’s literally not. There’s a web server, interfaces, plugins, etc.
LLMs are this super powerful thing (like a motor) and people are getting to play around with it before it’s fully harnessed. There’s this strange phenomenon where because it’s not totally harnessed, people just rip on it. I don’t know if they think it makes them sound smart, but it sure doesn’t to me. It’s like seeing a motor on an engine stand and being like “But the crankshaft rotates. I want to go in a straight line! This isn’t a transportation solution and anyone who thinks so is just naive. And horsepower? Stop zoomorphizing it!”
Sure, there are lots of problems. But don’t mix up the limitations of this component of the system, the current limitations of the system overall, and the possible capabilities of the system. Someone builds a car that stalls when it drives through a river and people point out that an ICE engine needs air and how could you even think about driving through a river? Then someone else is off making a snorkel.
I think you've got hold of the wrong end of your analogy.
If you just hold it the right way you will be illuminated and we will all agree how relevant it is.
If someone announces to the public, they have a revolutionary new motor that is going to transform personal transportation and solve climate change...
...but it is presented with a chat interface - the motor itself is directed through typing text and it demonstrates its motion/power through text messages in response - rather being connected to a vehicle or generator, rather than a transmission, then it's BS beyond a reasonable doubt.
Even though there is no logical or technical reason preventing such a setup, it's just not how professionals do things. If the motor were so damn useful, then people who like to use motors would use it for the things that they do use motors for!
There is no logical or legal requirement in science to use LaTeX Computer Modern fonts...but as a layperson if I can't tell if someone is a crank or doing advanced physics, I'm going to be exponentially more skeptical if they are using Times or Palatino.
Judging them for having a chat interface is the exact type of thing that I think is unintelligent. It would be like… judging an extremely popular paper for their choice of font. We’re past that.
I think the case is getting the attention because it's not just some one spouting off online, it's a lawyer bumping into the legal system with false information that would otherwise be a massive legal no-no and they are trying to scape-goat it onto the new shiny software.
But they might have seen the shiny new software touted by "those smart AI guys as being revolutionary and passing the Bar!" and they don't hang out on HN all day so to them that's like someone saying "this bridge is sturdy!" and they walk over it without realizing they should really go over the nuts and bolts of it like a civil engineer to really be sure
I just think idiots who touch fire should be burned - particularly lazy idiots whose high paid job in fact requires them to be extremely careful and precise in their actions, and who refuse to take responsibility for their actions afterwards.
It's not that they need to inspect every nut and bolt of the bridge, they just need to not walk over the bridge - or at least, not immediately start driving unreasonably heavy loads across it.
>I just think idiots who touch fire should be burned - particularly lazy idiots whose high paid job in fact requires them to be extremely careful and precise in their actions, and who refuse to take responsibility for their actions afterwards.
Are we talking about lawyers or the AI researchers?, because they certainly want to portray themselves as a modern day Prometheus
notOpenAI told them that this is so incomprehensibly smart that it will destroy all of us. Not because it's dumb and connects the wrong cables, but because it's super-smart. People can't surmise from that that the model just makes stuff up on the way
People assume the smalltext does not apply to them.
openAI does not get a lot of flak from the media for the amount of BS that chatGPT can blurt out
Does anyone remember what happened to Galactica which did the same thing ? That too was clearly labeled as hallucinatory . But it was shut down because they did not BS the media enough about regulations and such.
I m afraid these LLMs are turning into too much of a political game to be useful for much longer.
On the other hand, if they become political, then people will be even more incentivized to build offline, local LLMs
This is what I don't get. Not only have they not said otherwise, but they put it right up front in a pretty easy to understand brief message before you start using it. I guess lawyers just click agree without reading too.
People are accustomed to ignoring the fine print as legal CYA with no real-world relevance. This is also why the product warnings that "The State of California considers this to cause cancer" are a joke and not a useful message.
It’s not even fine print, there is a big Limitations section almost at the top of the chatgpt landing page with this as first bullet:
”ChatGPT sometimes writes plausible-sounding but incorrect or nonsensical answers.”
It does admittedly sound innocent, it should be written with a more scary tone on to how big extent it bullshits, not just “sometimes”. Still doesn’t excuse the incompetence of the lawyer in question.
> I wonder how many other people using ChatGPT do not comprehend that ChatGPT can be a confident bullshitter...
AI isn’t going to destroy us in some Terminator/Matrixy scenario. We are gonna be the agents of our own demise by just blindly walking off a cliff created from our own laziness.
Normal people don’t get that intuitively because it’s not intuitive at all.
For everyone who his looking down on this guy have you ever read a story on the Reddit front page and thought it was a real story accurately recounted by a real person and not a work of fiction? If so you’re equally naive.
If they were able to be tricked by ChatGPT, they are definitely not good at being lawyers. Trying to blame the AI is like trying to blame MS Word for offering an inappropriate homonym when spell checking. The computer did not put the citations in front of the judge.
The joke is, the judge even pointed out the citations they got from ChatGPT literally made no sense. Basically one quote went from describing an wrongful death to a legal claim over an airline ticket.
It's an understatement to describe them as "not good lawyers".
They basically never ever read their own court filings.
Yeah, they just defrauded their clients and billed arm and leg. Ironically I believe and hope that laywers for the most part get soon automated away by machines because legal language is different to human language.
You missed what I was saying (or I communicated it poorly). I'm not suggesting people use ChatGPT for their lawyer. They're saying it's ChatGPT's fault that they provided wrong info. If they provide no value (no liability for the service provided, and no filtering/vetting of ChatGPT answers from their lawyer expertise), then why would people pay their fee?
No, I'm saying if they think that it's to blame for wrong info and not themselves, then why should anyone pay for a lawyer instead of asking ChatGPT on their own?
The counterpoint here is that there is already a cause of action for this type of incompetence and it's called malpractice, which is a pretty reasonable road to remedy. I don't know if you actually think these were "fraudulent court documents", but "fraudulent" actually means something very specific and this ain't it. Even if the court is considering sanctions (which is not the same as disbarment), that seems at least partially related to the attys' failure to address their failure once they were aware of it.
Something interesting about the legal profession is that it is self-regulating. The state bars are typically not government organizations. Attorneys know that confidence in their profession is extremely important and they strike the balance between preserving that confidence and, you know, destroying someone's livelihood because they don't understand how LLMs work.
> Why not blame your laptop manufacturer for creating the hardware you used to file your fraudulent court documents?
Because your laptop manufacturer doesnt claim your laptop “thinks”, is “intelligent”, doesnt build an entire fud marketing campaign around the two, doesnt claim it “creates” ideas on its own, doesnt claim it “learns like a human”, doesnt claim it has cognitive abilities and so on.
Come on. We're dealing with qualified professionals here. The buck stops with them.
A professional civil engineer blaming a new untested simulation software after a bridge fails and causes injury - does it matter what the software claimed to do? It could claim itself to be god, but it's still up to the professionals to evaluate and select the tools they choose to use.
It's not like they were using an industry-standard tool they've been using for many years and it suddenly did something unexpected.
This is clearly carelessness. A basic out-of-bound check on the cases it hallucinated would be part of basic due-diligence. Digitized case indices are available within minutes.
What's even more egregious is that they went to the same tool to confirm its own reliability after they got some indication that it was unreliable. It doesn't take a professional to have commonsense.
The lawyers not only cited bogus cases, but when asked to provide copies of those bogus cases, fabricated multiple page PDF documents from whole cloth. This is impossible to argue as a mistake.
> Disbarment is equivalent to taking away his entire life's work.
'Lawyer' is the kind of profession which actually holds its practitioners to a standard, because the system falls apart when they stop behaving honestly.
This wasn't an oopsie-daisies, this was dishonesty, followed by further dishonesty, when they supplied bogus references for these cases.
It crossed the line from stupidity into dishonesty when the judge asked him "Wtf is this court case that you're citing", and the lawyer sent the judge a fabricated document.
How can anyone trust anything a lawyer does, when they are willing to fabricate documents, in order to continue to lie to a judge? This is Better Call Saul-level chicanery.
He could have fessed up and apologized when called out on it, but he doubled down.
A negligent civil engineer that failed to verify construction details provided by GPT, which later led to a bridge collapse, would lose their license. A negligent doctor that failed to vet a treatment protocol to a patient that later caused them harm would lose their license. Negligence is a fair, reasonable, and valid reason for someone to be determined unqualified to practice their profession in the future.
“There’s simply no way we could have known these were bogus cases.” the lawyers are quoted as saying.
They are currently using Bard to help draft a lawsuit against OpenAI, claiming the company knowingly misrepresents the capabilities of their technology.
The cynic in me wonders if this isn’t part of a plan to create a legal precedent banning AI from handling legal disputes.
Think about it: the legal profession is possibly one of the most threatened by the development of AI models. What better way to secure the professional future of the long tail of lawyers and paralegals?
The particular issue here is you believe that this lawyer just isn't dumber than a bag of hammers. There is no conspiracy needed. People are dumb. This is why when you see warning labels all over some item, "Big Label" didn't do that, no some dumbass got their tallywacker ripped off by their Easy Bake Oven. Now everyone has to deal with 10 stickers on the item and a dictionary sized book of what you can and can't do with it.
People are dumb. Lawyers shouldn't be. If distribution of stupidity among certain professions is similar to that of people in general, something has gone very bad somewhere in the education and certification pipeline.
Speak with a few lawyers and you'll realize they're just as dumb as the general population.
In some cases - even more dumb, since they have this belief that their credentials mean they know everything about law.
An awful lot of what a lawyer does is look stuff up... and in some cases, they aren't even that capable. All too often you are responsible for providing your lawyer with mountains of research, arguments, etc.
I've got some really, really, really bad news. Lawyers are not really any smarter than any other group of people out there. Passing a test about law doesn't mean much at all...
I mean, I did computer support for many lawyers and prosecutors and many were clever intelligent people. Others had to be explained to in simple instructions they shouldn't pour the glue on their keyboard before eating it. How they became a lawyer is beyond my understanding, and yet here we are.
This is a distinction that JD holders who have not passed the bar have tried to push. It is not true and holding yourself out as a lawyer without a license is the unlicensed practice of law. The public does not recognize the distinction and neither does the law.
You mistake "smart" with "have good memory". Laweyrs are far more about knowing the law and finding the law to apply at given situation + some social skills rather than "smart".
Obviously the good ones, as in any profession, will probably also be "smart", but that's just the top.
> something has gone very bad somewhere in the education and certification pipeline.
Yeah like the fact lawyers earn far more money than people responsible for teaching kids... teaching future generations should be at least as prestigious and well paid job
I don't know that it produces absolute garbage, you just need to be aware that perfect metrics don't exist. A person having a credential is one data point. You collect multiple data points to form an opinion.
eg., You probably shouldn't hire a lawyer with no law degree or no experience simply based on the fact that there are tons of credentialed, experienced attorneys who are no good.
> part of a plan to create a legal precedent banning AI from handling legal disputes.
The judge, in his ruling linked in previous HN discussion, listed a good half dozen parts of a legal dispute where he thinks AI would be super awesome, and then lays out why this particular part of the dispute is terrible for AI to play a role.
I should have made my thoughts clearer; my mind immediately went to small-time stuff like handling parking disputes, which I think AI is on track to competently solve.
I don't think parking disputes are any different. You either handle them yourself or you hire a lawyer, I don't think anyone can act as your agent without having a legal license.
So an AI can do the work but the person handling the dispute needs to sign off on it. If the AI screws up and the lawyer doesn't catch it then the lawyer's on the hook. I don't see any need to change this.
maybe, unless the AI calculates somehow that 98% or something of parking disputes can be resolved by just not showing up in court, so that's the strategy. But you can't convey in a reasonable enough prompt that "this cop just had that look of being ready to really screw me over"...idk man.
There was a brouhaha a few months ago over a tech startup that wanted to do AI lawyering, via having the chatbot speak in a bluetooth earphone or something. They actually signed up someone to do a speeding ticket hearing...
and subpoenaed the officer to make sure he showed up in court for the hearing so that they could actually have oral argument.
The best way to avoid such a ticket is to show up to court and hope the officer doesn't show up. And the AI firm went outs of way to make sure the officer showed up. That's legal malpractice right there. (In the end, IIRC, the judge heard about the firm's involvement and put the kibosh on the entire thing.)
No need for cynicism or a grand plan, this is just a few lawyers who could find no case law to justify their argument, so they made something up and proceeded to blame ChatGPT for it. They had several opportunities to "nope" out and apologize to the court, and they doubled down EVERY time. These lawyers deserve to be disbarred, no question.
either you consider that lawyers have an added value from IA and you consider they can take an advice from an IA but they are able to see through its bullshit, or you consider they have little added value when provided IA input, and then once AI is good enough/better as a whole (considering cost and availability) they are of no use.
When you start giving ChatGPT a plugin to query LexisNexis and do proper citiations (as Bing Chat does), then things get interesting.
Unfortunately Lexis's API fees are currently quite steep, so only very wealthy law firms will be able to afford to use such a service in the short term.
You'll still run into the same problem these guys did (i.e. not checking the citations). It's easy enough to just search to see if a thing it tells you is real.
...Until search is subsumed by chatbots and all you can access is a commercial statistical model's rendering of the truth, digested from ever more abstract renderings of the primary sources, which probably exist somewhere but are impractical to find under the mountain of re-written LLM spam.
I'm currently building something for a client that does this.
1. Searches for relevant documents.
2. Generate a response based on the found documents, using temperature=0 and a prompt that instructs the response to include a citation reference in a specific format.
3. Display the result, linking directly to the sources of the citations, and a warning on any that don't actually exist (which hasn't happened yet).
People confuse creating a response that makes sense in a language (which LLM’s are designed to do) with coveying facts and truths in a language (which LLM’s are not designed to do).
LLM’s are revolutionary because they provide a more fluent interface to data… but that doesn’t not mean that the data is correct. Especially not in the early phases.
For most people any sufficiently advance technology is indistinguishable from magic. The average joes think that this is magic.
Just once it would be refreshing for someone who gets caught doing something wrong to say, "Wow, ya got me. I'm sorry. Not for doing it, I'm only sorry I got caught. I really thought I'd skate right by on this one. Legitimately won't do it again. Or if I do I'll proof-read it better at least. Please let me know the punishment and I'll accept it."
I mean isn't the punishment his livelihood? And whoever's livelihood when someone looks under the hood and they realize a lot of people (not just lawyers) are doing this all over the place?
And everyone thinks they're too smart to get caught...until they aren't..
I used to be a lawyer. While I never got in trouble with the disciplinary commission, I did keep up with what they were up to. And once, I was involved in a bankruptcy case where they were a party (the disbarred lawyer filed bankruptcy, and it was a whole mess).
My sense is that these are serious people who do not put up with BS. BS will only make it worse.
I'm so confused. Why do the lawyers not simply check to see if the references are real or not? How hard is it to look through an LLM's output and do a quick search to see if any of the laws or cases mentioned in it are in fact unsubstantiated?
At best an LLM in this case should serve as a "pointer" or "hint" to references. It's clear one would still need validate the entirety of the case and the arguments put forward from them.
Because people aren't used to the idea that computers will make things up. I don't see why anyone is surprised by what the lawyer did here and for sure there is a lot more of this going on than is getting into the public.
People spend their whole lives thinking of machines as logical and correct, but lacking in creativity. Suddenly generative AI comes along and totally upends this. If you're an HN reader and see in depth discussions of LLMs every day then of course it's obvious that these things can make up citations, but if you primary experience of the tech industry is search engines that have never made up fake results ever, then why would you even imagine it's possible?
Sure use whatever tool to find suitable cases supporting you. But also verify that those cases support your position. Better not give ammo to your opponent.
OpenAI wanted to preach safety. I think we hold them liable for literally everything, everything Chat GPT does or says, until the model is open and they can argue that they have no control over it.
They wanted this liability, they accepted this liability, they said they'd keep it safe and they haven't. It's on them.
While the lawyers blamed ChatGPT, the totality of the circumstances seem to indicate that they're less than honest in doing so. There is a live-tweet of the hearing here: https://twitter.com/innercitypress/status/166683852676213965..., and you can follow along with the lawyerly cringe there.
Okay, lawyer #1 (LoDuca, the one on the case in the first place) appears to have played essentially no role in the entire case; his entire purpose appears to be effectively a sockpuppet for lawyer #2 (Schwartz), as LoDuca was admitted to federal court and Schwartz was not. He admits to not having read the things Schwartz asked him to file, as well as the "WTF?" missives that came back. He lied to the court about when he was going to be on vacation, because that is when Schwartz was on vacation. But other than doing nothing when he was supposed to do something (supposed to do a lot of somethings), he is otherwise uninvolved in the shenanigans.
So everything happened because of Schwartz, but before we get to this part, let me fill in relevant background information. The client is suing an airline for an injury governed by the Montreal Protocol. Said airline went bankrupt, and when that happened, the lawyer dismissed the lawsuit, only to refile it when the airline emerged from bankruptcy. This was a mistake; dismissing-and-refiling means the second case is outside the statute of limitations. The airline filed a motion to dismiss because, well, outside statute of limitations, and it is Schwartz's response that is at the center of this controversy.
What appears to have happened is that there is no case law to justify why the case shouldn't be dismissed. Schwartz used chatGPT to try to come up with case law [1] for the argument. He claims in the hearing that he treated it like a search engine, and didn't understand that it could come up with fake arguments. But those claims I'm skeptical of, because even if he's using chatGPT to search for cases, he clearly isn't reading them.
When the airline basically said "uh, we can't find these cases," Schwartz responded by providing fake cases from chatGPT where alarm bells should be ringing saying "SOMETHING IS HORRIBLY, HORRIBLY WRONG." The purported cases in the reply had blindingly obvious flaws that ought to have made you realize something was up before you're off the first page. It is only when the judge turns around and issues the order to show cause that the lawyers attempt to start coming clean.
But wait, there's more! The response was improperly notarized: it had the wrong month. So the judge asked them to provide the original document before signature to justify why it wasn't notary fraud. And, uh, there's a clear OCR error (compare last page of [2] and [3]).
When we get to these parts of in the hearing, Schwartz's responses aren't encouraging. Schwartz tries to dodge the issue of why he was citing cases he didn't read. Believing his responses of why he thought the cases were unpublished ("F.3d means Federal district, third department") really requires you to assume he is an incompetent lawyer at best. The inconsistencies in the affidavit are glossed over, and the stories don't entirely add up. It seems like a minor issue, but it does really give the impression that even with all the attention on them right now, they're still being less than candid with the court.
The attorneys for Schwartz are trying hard to frame it as a "he didn't know what he was getting into with chatGPT, it's not his fault," but honestly, it really does strike me that he knew what he was getting into and somehow thought he wouldn't get caught.
Hilariously, the explanation of the notarization typo (actually, I find that believable) is a 1746 "declaration." It's an old federal law which provides that as long as you say you are signing under oath, it is as good as an affidavit in federal court.
Why didn't he just make a declaration in the first place? Also, why would he have the lawyer who is in trouble notarize the document? Now that he has made the typo, he may now need Schwartz's testimony that actually, it was April 25, not January.
In my legal career, I worked for a judge once. He told me to never get a notary stamp. It only creates problems. There's never a good reason for a lawyer to notarize something. You ask your staff to do it instead.
> OMG. This is the Chat-GPT lawyer. You learn what F.3d means in first year legal writing. To non-lawyers, this is a tiny bit like someone being asked "You know what the DOJ is, right" and getting something wild like "Directorate of Judges" as a response.
The correct meaning of "F.3d" is "Federal Reporter, 3rd Series".
And it is like a lawyer's worst nightmare to have an exchange with a judge like this:
> Judge Castel: Have you heard of the Federal Reporter?
> Schwartz: Yes.
> Judge Castel: That's a book, right?
> Schwartz: Correct.
The implication being that at bare minimum, the lawyer could have looked it up in a book. Like a first year law student would.
100% agree. They are trying to deflect blame onto "old lawyer scared of scary new technology". Why they thought they'd get away with making up entire cases out of whole cloth, who knows...
Given that they're attorneys, we know what their next course of action may be. I've noticed that ChatGPT's new onboarding includes a screen with a big disclaimer where there wasn't one before. I can only assume that it may be related to cases like these.
Thats weird. I tried really hard to convince it that there's such case law establishing a legal category of "praiseworthy homicide" and it refused to believe me. I thought it was overttrained / patched on all law related applications.
comments here are like "what dumb lawyers". Sure OK. But what can we say here about "GPT-4 passed the bar exam!" and how useful is that data, given that this does not imply GPT-4 has the actual skills of a human lawyer.
Your reading comprehension seems to be as good as that of the lawyer here. You asked what this says about the claim that GPT-4 was good enough to pass the bar. I didn't say anything about GPT-4's quality or the legality here, only that we cannot assume this was the output of GPT-4 without evidence it was given that more people overwhelmingly use the default 3.5-turbo.
I don't understand how they messed this up so bad. They say they didn't know it could hallucinate and that they thought it was just like any other search engine. But it seems like even if it worked like they thought, they'd still have fucked up?
If it's just like a normal person, if that person isn't a lawyer, you wouldn't ask them to do your lawyery work. I'd hope this lawyer doesn't his kids to do his work for him.
If it's just like a normal search engine, we all know how much bullshit, spam, and misinformation there is on the internet (mostly written by normal good old fashioned humans!). So that wouldn't have been trustworthy either!
A lawyer is certainly at fault if they do not fact check the material they present at trial. But the conmen who are selling ChatGPT and the like are extremely irresponsible for the way they sell LLMs as magical AI that arrives at factually correct answers by reasoning rather than the consequence of the law of large numbers applied to stochastic text generation.
> the conmen who are selling ChatGPT and the like are extremely irresponsible for the way they sell LLMs as magical AI that arrives at factually correct answers
ChatGPT has a pop-up on first use, a warning at the top of each chat, a warning below the chat bar, and a section in the FAQ explaining that it can generate nonsense and can't verify facts, provide references, or complete lookups.
There is probably more OpenAI could do, like detect attempts to generate false references and add a warning in red to that chat message - since it seems there are still people taking its hallucinations as fact (although if there's hundreds of millions of users, maybe only a tiny fraction), but I don't think this is a fair characterization.
This is interesting. How long untill someone gets sick because he/she was following what chatGPT told him/her to do? Medical advice? Political misinformation?
How things will unfold this decade? Banning chatgpt from certain topics (medicine, law, etc...)? This decade will be really interesting indeed.
Heh, your spidey senses are correct. I actually wrote the response and then ran it though ChatGPT for clean up. I do agree, it actually reads weird over my original draft.
Did you write this comment with ChatGPT's help? It's a helpful and well written post. I'm not trying to insult it. But I'm curious. The text seems to say "AI", and I'm wondering if I'm seeing things not there.
Yes, in the spirt of talking about ChatGPT I did run my response through it. It is probably the worst thing it has ever produced for me. I ring this up to ChatGPT's "declining quality". :)
>ChatGPT cannot uncover anything, especially the municipality-specific corpus of codes that the comment ostensibly claims to have access to.
It's possible the training set includes those codes (or a citation of those codes). It's possible that they used GPT4 with Browsing, or that it gave them the right terminology to search for & they then pasted in sections of the code and asked it to work off that.
It's also entirely possible ChatGPT hallucinated these utility code sections to support its case and the people working for the utility didn't call their bluff... which is essentially exactly what this story is about, except they got caught out because ChatGPT's bluff was called...
Great anecdote, thanks for sharing. Forcing ChatGPT to go back and sense check its overarching initial interpretation is such a good way of doing things in these types of uses cases.
ChatGPT told me that. It might be true.