back to article Lawyers face judge's wrath after AI cites made-up cases in fiery hoverboard lawsuit

Demonstrating yet again that uncritically trusting the output of generative AI is dangerous, attorneys involved in a product liability lawsuit have apologized to the presiding judge for submitting documents that cite non-existent legal cases. The lawsuit began with a complaint filed in June, 2023, against Walmart and Jetson …

  1. MrMerrymaker

    what are you paying for

    I mean if you are giving lawyers money you're already going wrong but I don't see how this is dereliction of duty or inherently fraud

    This sort of thing must be human-done, if it's advertised the humans are doing the research.

    On this evidence their jobs are safe...

    1. doublelayer Silver badge

      Re: what are you paying for

      It's not fraud, unless they promised that they wouldn't use any AI tools. Since they evidently have their own internal tool, they probably did not make that promise. So fraud is definitely off the table. What it is, however, is incompetence. Their clients have every reason to be angry with them about that, and they may be able to sue these lawyers, but for negligence rather than fraud. I'm not sure how likely they are to win, but I suppose I can find out by giving an LLM the prompt "Show me case law demonstrating that lawyers citing nonexistent cases is considered negligence".

      Of course, the events described in the article aren't at that level yet. This is a court potentially punishing the lawyers, not their clients. Courts can penalize lawyers for a variety of things with a lot of latitude for the judge. Even if this wasn't sufficient negligence for the clients to win against them, a judge can assess some penalties, either on the lawyer, for example finding them in contempt of court, or to the client, for example by rejecting their motion to exclude evidence because they failed to make their case. These things are complicated because the terms for what a judge is allowed to do are a little vague and some of this might be overturned on appeal if the evidence really should have been excluded but wasn't due to attorney incompetence.

      1. Lazlo Woodbine Silver badge

        Re: what are you paying for

        It shouldn't be a case of companies saying they won't use AI, rather companies should declare if they do use AI, and give a clear indication that they will accept any consequences relating to inaccurances from their use of AI.

        1. jdiebdhidbsusbvwbsidnsoskebid Silver badge

          Re: what are you paying for

          Exactly. If you say it, you own the consequences.

          Whether you use AI or not, you should always accept any consequences relating to inaccuracies.

          1. John Brown (no body) Silver badge

            Re: what are you paying for

            The incompetence in this case come from the fact it would probably have taken seconds to enter the AI cited case numbers into whatever legal databases they have and checked them out. A few seconds or minutes compared to the hours of research time the "AI" had just saved them, but they were too busy to do that.

        2. Doctor Syntax Silver badge

          Re: what are you paying for

          In this particular case to whom should they say they use AI? The clients? The court? The other side?

          There are all sorts of ramifications. It's one thing accepting the consequences for presenting obviously incorrect references in the pleadings. But if the AI conjures up non-existent reference is it going to miss references that should have been found? And if so how will this be discovered - if it's discovered at all. Their clients might lose the case because the AI doesn't find a reference a competent attorney would have known about.

          Unless the AI assistant can be relied on to make no errors of omission or commission then it would be wiser not to use it at all.

          1. jdiebdhidbsusbvwbsidnsoskebid Silver badge

            Re: what are you paying for

            "to whom should they say they use AI? The clients? The court? The other side?"

            Where I work, it's company policy to include that in the information you are delivering, so the declaration that an LLM was used is there upfront for everyone to see. I prefer to go one step further and include something about how the LLM was used, what the balance of LLM/human input was etc.

            "Unless the AI assistant can be relied on to make no errors of omission or commission then it would be wiser not to use it at all." Why? It's not like humans always meet that standard.

            1. Eclectic Man Silver badge
              Childcatcher

              Re: what are you paying for

              Legal practices have access to the databases of legal cases, and the whole point of the references is that hey can be checked. In this case, the AI referenced cases that the legal team could and should have checked, but did not. All of the people involved in the case should have been made aware of the use of AI, and who accepts liability for any mistakes.

            2. LBJsPNS Bronze badge

              Re: what are you paying for

              Generally human attorneys understand the penalties for making something up out of whole cloth and submitting it to the court.

          2. Lazlo Woodbine Silver badge

            Re: what are you paying for

            "Unless the AI assistant can be relied on to make no errors of omission or commission then it would be wiser not to use it at all."

            Well, yes, this should be obvious; unfortunately far too many people are drinking the AI kool-aid.

            And as for whom they should inform, regarding their use of AI - they should inform their clients in the first instance, with big print at the head of their contract stating they may make us of AI when researching their case. If they do end up using AI, it should be stated in their court submissions.

            The more companies have to come clean about using AI, the less likely they'll use it.

            When I published a novel, I had to state in my submission whether I'd used AI to generate any content (I didn't, every word was typed by my own hands) - so if I have to declare it when submitting something as trivial as a novel, why shouldn't law firms declare it when submitting trial paperwork?

          3. rafff

            Re: what are you paying for

            "to whom should they say they use AI? The clients? The court? The other side?"

            Surely the "other side" should have checked the rerences and flagged them up as fake, and raised merry hell?

            1. Sherrie Ludwig

              Re: what are you paying for

              "to whom should they say they use AI? The clients? The court? The other side?"

              Yes.

          4. DoctorPaul Bronze badge

            Re: what are you paying for

            "Unless the AI assistant can be relied on to make no errors of omission or commission then it would be wiser not to use it at all."

            And we know they can't, ergo ....... ?

        3. doublelayer Silver badge

          Re: what are you paying for

          My post was not about whether they should say it, where, and to whom. I agree that they should say it. My post was about what they can be sued for for their mistake, whether they said they used it or not. Fraud would be a valid charge if they lied about not using it, then used it anyway. Since they have no problem admitting it, it seems almost certain that they either said they would use it, then did, or made no statements about using it, then did. This makes it more of a negligence issue than fraud, and I think the lawyers would probably lose that case.

    2. Richard 12 Silver badge

      Re: what are you paying for

      It's probably not fraud.

      It is clear negligence and probably contempt of court.

      The client can and should sue for negligence, the judge is clearly considering a contempt ruling.

      At some point a lawyer will be disbarred, so I'm really surprised so many US law firms are doing this.

      1. cyberdemon Silver badge
        Devil

        Re: what are you paying for

        Well, if you as a client had paid for legal representation by a qualified lawyer, and you did not in fact get legal representation by a qualified lawyer, but instead were given generative slop from some bullshit machine, then it would be fraud. But here, it is the judge who has been presented with slop, so it is contempt.

        1. Elongated Muskrat Silver badge

          Re: what are you paying for

          They are getting legal representation from a qualified lawyer, it's just that the qualified lawyer has gone and cited something that should hold as much weight as something overheard in a pub, and then hasn't even bothered to go and verify that citation before presenting it to the court.

          It would, on the face of it, appear to be a legitimate use for AI to go and find supporting instances of case law. However, anyone tasked with writing such a system should, at the very least, add a filter in to verify that the cases it comes up with actually exist, and the person using it should certainly go and read at the very least the summary of those cases.

          It seems that the lawyer in this case is about as skilful as Lionel Hutz.

          1. Doctor Syntax Silver badge

            Re: what are you paying for

            "add a filter in to verify that the cases it comes up with actually exist"

            It's one thing to add a filter to pick out stuff that shouldn't be in there but what about the possibility that stuff that should be in there and isn't? If it can make one set of errors it can also make others.

          2. Anonymous Coward
            Anonymous Coward

            Re: what are you paying for

            Flywheel, Shyster and Flywheel could certainly give this lawyer a lesson or two..... Quite frankly, I'm amazed this lawyer wasn't quoting cases involving them.

            1. LBJsPNS Bronze badge

              Re: what are you paying for

              Dewey, Cheatham, and Howe would like a word...

              1. cyberdemon Silver badge
                Devil

                Re: what are you paying for

                As would Messrs Sue, Grabbitt & Runne

          3. John Smith 19 Gold badge
            Unhappy

            "at the very least, add a filter in to verify that the cases it comes up with actually exist,"

            That's not even AI.

            It's essentially guarding against an assertion failure.

            Sooner or later all this Ohh-it's-sooo-mysterious-AI-we-can't-possibly-understand-how-it-works will be recognized as the total bu***hit that it actually is.

        2. martinusher Silver badge

          Re: what are you paying for

          Its often 'generative slop from a legal assistant'. Well, not likely 'generative slop' because legal assistants are likely as well trained as a lawyer but have yet to pass their state bar exam, find a firm to intern with and so on.

          As with a lot of AI these days its a tool to cut out those middle jobs, the actual donkey work, replacing live humans that need salaries and benefits with a cost cutting machine. The question we should all be asking ourselves is how many cases have gone forward so far -- and how many decisions have already been made -- based on bogus citations. (...and if there's enough of them these judgments are likely to be cited in future cases)

          1. MachDiamond Silver badge

            Re: what are you paying for

            "Its often 'generative slop from a legal assistant'. Well, not likely 'generative slop' because legal assistants are likely as well trained as a lawyer but have yet to pass their state bar exam, find a firm to intern with and so on."

            Plenty of assistants aren't on a path to being an attorney themselves but will wind up with the knowledge by doing all of the fact checking and research for an attorney.

            I do some product development for clients and if I had somebody working for me, I could give them a sketch of something to draw up in CAD so I could 3D print a part rather than do it myself. I would double check the CAD drawing before printing the parts since it can take a day or more only to find an obvious issue that should have been caught. If I were a lot busier doing that sort of thing, I would have people to delegate tasks to and review their work. There's only so many hours in a day and my time isn't well spent filing rough edges if there's higher value tasks that need doing.

            A person I know drove an attorney to work and back each day while going to college. The attorney outfitted a van as a mobile office and the hour plus drive time was billable hours at some bloated $/hour. The attorney could drive himself in (and live much closer to the office), but it was obvious that letting somebody else do that gave him another $1,000/day of revenue at the cost of a couple of hundred.

        3. Sherrie Ludwig

          Re: what are you paying for

          Well, if you as a client had paid for legal representation by a qualified lawyer, and you did not in fact get legal representation by a qualified lawyer, but instead were given generative slop from some bullshit machine, then it would be fraud. But here, it is the judge who has been presented with slop, so it is contempt.

          Could also be construed as perjury, since the lawyer must attest to the filing, sign that it is a true setting out of the facts as known to the attestor.

      2. vtcodger Silver badge

        Re: what are you paying for

        "so I'm really surprised so many US law firms are doing this."

        Why are you surprised? A massive hype machine led by blatant scam artists is pushing AI to the public in hopes of recovering huge investments quickly. Lawyers are clever. Probably considerably smarter than the average human. (But not necessarily smarter than the average dog. After all, how many dogs voted for Donald Trump?) Why would it be a surprise that some lawyers actually believe the hyperbole?

        I would be surprised if lawyers don't learn very quickly that they need to actually check any citations found by AI and make sure the cases exist and that the cases actually say whatever AI agents claim they say.

        BTW, can a lawyer sanctioned by the court for citing ficticious cases turn around and sue his or her AI provider for damages?

        1. Doctor Syntax Silver badge

          Re: what are you paying for

          "can a lawyer sanctioned by the court for citing ficticious cases turn around and sue his or her AI provider for damages?"

          Only if they can cite appropriate cases.

          1. Tron Silver badge

            Re: what are you paying for

            quote: "can a lawyer sanctioned by the court for citing ficticious cases turn around and sue his or her AI provider for damages?"

            Probably not, as the pushers of AI all include terms and conditions in which they point out that their AI is unreliable, experimental, just a bit of fun, well, bollocks really. Or words to that effect. They just don't say that in the adverts. GAFA has been run by lawyers for years, so their small print arse-covering is likely to work better than any of their technology.

        2. Anonymous Coward
          Anonymous Coward

          Re: what are you paying for

          "BTW, can a lawyer sanctioned by the court for citing ficticious cases turn around and sue his or her AI provider for damages?"

          Sure, they can try - but they're still sanctioned by the court.

      3. MachDiamond Silver badge

        Re: what are you paying for

        "It's probably not fraud."

        It would be hard to prove fraud in something like this as there would be a need to show intent that can't be explained away by incompetence.

        An attorney is a deputy of the court and has a mandated requirement and affirms that everything they submit to a court is true and correct to the best of their knowledge. It doesn't matter if one of their staff or a machine has made the error, the attorney of record owns it.

      4. MachDiamond Silver badge

        Re: what are you paying for

        "At some point a lawyer will be disbarred, so I'm really surprised so many US law firms are doing this."

        It's happened. I'm pretty sure they were unless it was only a suspension and fine.

    3. Gary Stewart Silver badge

      Re: what are you paying for

      "I mean if you are giving lawyers money you're already going wrong"

      If you (or the state) don't give lawyers money you're(sic) going to end up owing somebody a lot of money, or in jail, or worse (we still have the death penalty in the US). So pick your(/sic) poison.

      1. Elongated Muskrat Silver badge
        Headmaster

        Re: what are you paying for

        I'm not entirely certain why you've put "(sic)" there, since those would appear to be the correct usages of "you're" and "your".

        Typically, sic is used with square brackets, to distinguish between parenthesised text in the original, and the quoter's own parentheses, although this may be seen as more of a style choice.

        1. neilg
          Headmaster

          Re: what are you paying for

          I think all Commentards should search for "expletive grammar mug" & buy one. Worth every penny :-)

    4. Antron Argaiv Silver badge
      WTF?

      Re: what are you paying for

      IANAL, but...

      The first time this happened (at least a couple of years ago), I said, "OK, blame the intern". And that was exactly the case, the lawyer who filed the document had been using a "clever" intern to do their research for them, and the intern had saved some time by using ChatGPT (aka, "cat, I farted"). The judge in the case did NOT blame the intern, and IIRC, the lawyer got slapped down pretty hard.

      And now, after everyone has heard the story, we have ANOTHER lawyer caught using ChatGPT for his citations? That lawyer is not only in trouble, but should never practice law again, because if we in the nerd community have heard about the first case, you have to believe that there's a Big Red Notice posted above the research terminals in every law office in the country, reminding them not to use ChatGPT for citations...and this idiot ignored it.

      1. John Brown (no body) Silver badge

        Re: what are you paying for

        "you have to believe that there's a Big Red Notice posted above the research terminals in every law office in the country,"

        Apparently not, since the article referenced the offending law firms new mitigation as putting a check box[*] on the GUI screen reminding user of the need to fact check. But this is, of course, the USA, where fact checking is just so passé nowadays.

        * or maybe you are right, but no one bothered to read it and the check box is just an extra warning for people to ignore and click though. MS lawyers may be involved as they are the masters at training users to click through without reading by offering so many click through boxes so often to their users :-)

        1. Frankly

          Re: what are you paying for

          "...where fact checking is just so passé nowadays.

          That made me laugh.

    5. MachDiamond Silver badge

      Re: what are you paying for

      "but I don't see how this is dereliction of duty or inherently fraud"

      Blood sucking lawyers can charge the average person's weekly pay PER HOUR for their services and should be verifying the accuracy of their work. If they don't and this sort of thing becomes acceptable, the average person will just use AI to file briefs and motions on their own without an attorney and let the judges sort it out.

      AI can be a good tool to find citations especially in places such as the US where it's very common place to use precedent and rely on other judges poor opinions to make a case rather than making a useful interpretation of the law(s) as written. What must happen is whatever AI kicks out gets checked. That's often a job that's delegated to staff to at least make sure it's correct, make a print for the attorney to review, etc.

    6. LBJsPNS Bronze badge

      Re: what are you paying for

      "I don't see how this is dereliction of duty or inherently fraud"

      I'm paying for an attorney. If I wanted AI to handle my court proceedings I can fuck it up as badly as AI can and not have to pay an attorney. This is without doubt fraud, if not outright contempt of court. Throw the book at them.

  2. Bebu sa Ware
    Big Brother

    motion in limine

    New one on me.

    Apparently in the US it's a motion presented to the court in the absence of the jurors. In limine is Latin which is translated as "at the threshold" (of proceedings.)

    I imagine, in the best tradition of the US judicial system, with the application of sufficient lucre, will in time mean that AI confabulated cases will become admissible and equally valid as existing case law.

    1. I ain't Spartacus Gold badge

      Re: motion in limine

      I'd be surprised if there's any legal system that operates jury trials that doesn't have a system for the judge to decide what evidence to put the jury, with both sets of lawyers arguing about it first.

      My last jury service was quite interesting - assault and GBH (grevious bodily harm) - because the prosecution lawyer broke these rules. He asked a question, and before the witness could answer the judge had interrupted and said, "we already agreed you couldn't go down this line of questioning." To which the lawyer replied that he was trying to develop a different point. So the judge let him continue.

      A couple of minutes later, he asked the same question again, and the judge virtually levitated out of his chair. And shouted, "stop!" very loudly. He then got about ten words into a very loud and angry verbal dressing-down - then stopped - turned to us in the jury and politely said, "I'm very sorry but the jury will need to retire while I address this issue". As the usher led us to the jury room you could hear him shouting through the not-as-soundproof-as-I'd-initially-thought door.

      We found this quite amusing, back in the jury room. And the bollocking must have been substantial, because we weren't allowed back in for nearly half an hour. And were then told to ignore the last bit of evidence as none of it was admissable. Prosecution barrister was a chastened man for the rest of the afternoon.

      1. Doctor Syntax Silver badge

        Re: motion in limine

        I tended to find these the most interesting parts of the trial when observed from the body of the court. The lawyers including the judge could be seen doing what I'd class as real intellectual work producing and discussing huge, bookmarked tomes. It seemed unfair to the jurors to be excluded. I never heard of such episodes being given a title of their own but I suppose it sould be necessary to have one to refer by later.

      2. collinsl Silver badge

        Re: motion in limine

        I'd be surprised if there's any legal system that operates jury trials that doesn't have a system for the judge to decide what evidence to put the jury, with both sets of lawyers arguing about it first.

        We definitely have that in the UK, it's just not usually called a "motion in liminie" - we just term it as a "ruling on the evidence" or similar.

        source

        1. I ain't Spartacus Gold badge

          Re: motion in limine

          collinsl,

          I've never observed a trial - or watched the couple the BBC have been allowed to show, as experiments. So my only experience is jury duty. And of course there's a lot of it you don't see. But I got the impression (I've done jury duty twice) that a lot goes on in the judges offices, as well as the courtrooms. At least on the circuit courts. There's probably a relatively small number of regular barristers and the judges move around, but in a limited area. So I suspect they snatch a lot of trial planning meetings about stuff for the future - if they happen to be in the same place for other trials.

          Talking to the ushers it was clear that there were lots of meetings going on with the judges, that were unrelated to the case we were on that week. I guess some were formal hearings in the court, the others in the offices in the non public bit - that the jury get to walk past on the way from court to jury room (at least in the more modern of my county's two Crown court buildings.

          I was surprised how little time I spent in court, as a juror. Even on a week long case. Though that could partly be down to it only being relatively simple assault cases.

          It's like an iceberg. The jury only see the top. By design. It's a bit scary, when you can see there's movement behind the curtain, but this has been deliberately designed to exclude telling you vital stuff - you'd like to know to make a better decision. But the very idea of a non-expert jury requires that you be kept in the dark about some things. You're then having to give your verdict on trust.

          One case involved a nightclub fight, where everyone had been offered some kind of deal to avoid prosecution in order to try and get their testimony to convinct just one guy of the more serious assault (stamping on someone's head). Understandable, but made me queasy, and slightly disgusted as the person who started the fight smugly told us what happened from the witness box.

          We didn't convict because even with being let off prosecution there weren't enough good witnesses. But I still worry as much about not convicting a possible vicious face-stamping bastard as that we might have got it wrong about the one we did convict in a different case.

      3. This post has been deleted by its author

  3. Neil Barnes Silver badge

    Another query made was 'Add a paragraph...'

    Perhaps this is part of the issue... That's not a query, that's an instruction. The Statistical Machine was told to do something and it did it.

    The fault is with the lawyer, plain and simple: he tried a short cut (er, I mean, of course, he applied the latest technology to assist him in his job) and it failed. He trusted the output and obviously did not perform even the simplest of checks to find out whether the citations/precedents actually existed.

    Why do lawyers keep all this shelves full of books on the wall behind their desks? Do they never read them?

    1. Richard 12 Silver badge

      Re: Another query made was 'Add a paragraph...'

      It's set dressing, always has been.

      If you look closer at the titles it becomes very obvious.

    2. Ian K
      Holmes

      Re: Another query made was 'Add a paragraph...'

      > Why do lawyers keep all this shelves full of books on the wall behind their desks?

      To impress rubes, and look good in Netflix series?

    3. Elongated Muskrat Silver badge

      Re: Another query made was 'Add a paragraph...'

      Those books will not contain case law, or not recent case law at any rate.

    4. Doctor Syntax Silver badge

      Re: Another query made was 'Add a paragraph...'

      "Why do lawyers keep all this shelves full of books on the wall behind their desks? Do they never read them?"

      They were, at least in my time, wielded in court in the sort of legal argument described although it was usually relatively few - Archbold being the most frequently used as far as I could see. However I'd guess that those few most used books would then send someone off on a long trail through the less frequently used reports.

    5. John Brown (no body) Silver badge

      Re: Another query made was 'Add a paragraph...'

      "The Statistical Machine was told to do something and it did it."

      I've said before here and I'll say it again. Most of not all of these LLMs don't appear to have an option of stating "I'm sorry Dave, I'm afraind I can't do that" and so will inevitably "hallucinate" when asked to do something they can't. HAL in 2001: A Space Odyssey going "insane" due to conflicting orders was a warning, not an instructional video.

      1. Neil Barnes Silver badge
        Facepalm

        Re: Another query made was 'Add a paragraph...'

        Searching for information about a new chip which unfortunately doesn't yet have a datasheet...

        In days of olde: attempt to visit www.chipmaker.com/latestchip/datasheet.pdf - 404 not found

        More recently: unable to find, but here's the maker's homepage, and here are datasheets for some similarly named chips

        More more recently unable to find, so here's an advert for a toaster

        Now: here's a datasheet I made up. It looks like lots of others, so it's probably fine...

        1. David Hicklin Silver badge

          Re: Another query made was 'Add a paragraph...'

          > Now: here's a datasheet I made up. It looks like lots of others, so it's probably fine...

          Ah you get that as well, Google is awful for it and I am spending more time in DDG which seems to do it less (so far!)

  4. xyz Silver badge

    Hoo boi...

    >Monday filed a joint response [PDF] acknowledging the error. Their filing includes the following:

    We are a bunch of idle bastards who believe any old shit a computer says, as long as we can leg it to the bar on a Friday afternoon and still bill the customer for the time it would have taken to do the job properly. We're lawyers, we don't do sorry.

    1. A Non e-mouse Silver badge

      Re: Hoo boi...

      We're lawyers, we don't do sorry.

      To be fair, the lawyer did 'fess and apologise. From the article:

      With a repentant heart, I sincerely apologize to this court, to my firm, and colleagues representing defendants for this mistake and any embarrassment I may have caused. The last week has been very humbling for me professionally, and personally, one that I can guarantee shall not ever repeat itself.

      1. Kane
        Joke

        Re: Hoo boi...

        "To be fair, the lawyer did 'fess and apologise"

        Yes, but if you look really close he's got his fingers crossed behind his back.

        1. David 132 Silver badge

          Re: Hoo boi...

          1) "the attorneys that signed the filing" - what are they signing it for, if not to say "we certify that this is true and accurate and we have verified every word of it"? I suspect that of the three named attorneys who, if I'm interpreting the article correctly, signed it, two of them didn't bother to read what they were rubber-stamping and...

          2) ...the third attorney was the most junior of the three, and was "persuaded" to carry the can for the AI oopsie. Hence the groveling and judge-foot-kissing apology.

      2. TRT Silver badge

        Re: Hoo boi...

        Yeah. He got the AI to write it as there wasn't a template for it in the stock company documents.

        1. Doctor Syntax Silver badge

          Re: Hoo boi...

          I very much doubt that there's a company template for such documents other than getting a few recitals in at the start.

      3. An_Old_Dog Silver badge

        Re: Hoo boi...

        If you shoplift something and are caught, you can't just put it back, say you're sorry, and escape the consequences.

        The judge should have rejected their filing, fined 'em, and put 'em in jail for a week or so for contempt of court.

      4. collinsl Silver badge

        Re: Hoo boi...

        That was only to stop him from losing his job or ending up in chokey

      5. ChrisElvidge Silver badge

        To be fair ...

        Only after he got caught out

        Really should have seen this himself before appearing in court.

  5. kmorwath

    Artificial Intelligence, Human Laziness

    AI - what is good for? To enable lazy people to earn money easily without thinking. This is a perfect example. Well paid people who try to work as little as they can, while still invoicing a lot.

    And frankly, if AI really worked, the lawyer job becomes reduntant - just use an AI directly in court.

    I don't understant why the lawyers were not fined directly, they presented false made up data, and it's only they responsibility, regardless of what tools they used.

    1. John Brown (no body) Silver badge

      Re: Artificial Intelligence, Human Laziness

      Agreed! But "on a computer" and computers "glitch" sometimes and "we don't understand computers", so "sorry, not really all our fault, honest Guv". The general public, and I include anyone not directly involved in the IT world here, has been "trained" by the media, news, print, TV and movies, to accept the computers are always right except when they are wrong and only true IT nerds experts could possibly have foreseen it going bad. This despite the fact that most of these people have grown up using computers or at least been using them for decades. It's 2025! The "bright young things" in the court rooms were probably born after Windows 2000 came out. There's no way they went through law school wihtou using computers and seeing the limitations they have. Even with the current trend and BS over AI.

  6. Anonymous Coward
    Anonymous Coward

    Hey chatgpt, write an apology...

    ... Use some waffle that will prevent me from having to face any consequences of my actions.

    Let's just hope the judges are also allowed to use AI to throw the book at these morons, so they don't waste their brain time on more worthwhile things.

    Maybe the whole legal system should just be chatbots arguing with chatbots?

    1. MachDiamond Silver badge

      Re: Hey chatgpt, write an apology...

      "Maybe the whole legal system should just be chatbots arguing with chatbots?"

      Are you sure it isn't? How would you tell as it's all in a different language? The words are the same, but they get the definitions from a much different dictionary.

  7. Pascal Monett Silver badge
    FAIL

    "attorneys [..] have apologized [..] for submitting documents that cite non-existent legal cases"

    You don't get to apologize. Not for the price you charge, nor for the responsibility you are supposed to uphold.

    You get disbarred.

  8. Long John Silver Bronze badge
    Pirate

    Onwards for AI to greater things

    Now 'AI' has shown itself adept at confabulating case-law, it's time to extend AI's reach to concocting legislation.

    Given that the quality of people sitting in legislatures, especially in the USA and UK, is dire - one may say a pinnacle of 'representative democracy' because they truly represent the qualities of the populations from which they are drawn - delegating legislation drafting to AIs makes sense.

    A legislature need only opt to agree a single paragraph of text outlining the intent of planned legislation. This would then be fed into an AI. Shortly afterwards, the AI would, using impeccable legalise, spew out many pages of suitably turgid prose. Should the text wander off into a pornographic recounting of the Tale of Little Red Riding Hood, nobody will notice; in fact, other AIs later ploughing through the clauses may present legal counsel with a plausible 'Riding Hood defence'.

    The draft immediately be will be passed on the nod of the government's majority. It will automatically be sent onwards to the king/president for his AI to rubber stamp. Meanwhile, after a hard one-hour session of legislating (half an hour by each House) the elected representatives and British Upper House appointees of dubious worth, will return to their various schemes for peculation, insider-trading, bribe acceptance, and sucking-up to the ('generous') super-wealthy; also, they will be left with plenty of time to assuage sexual lust of various kinds.

    1. Doctor Syntax Silver badge

      Re: Onwards for AI to greater things

      "Given that the quality of people sitting in legislatures"

      You don't think that lot are trusted with writing legislation do you? There are professionals who do that job.

  9. Coastal cutie
    Facepalm

    Here we go again

    What a trio of pillocks - it's not as though the issues around case hallucination aren't already well known and lawyers backsides been kicked https://www.theregister.com/2023/06/22/lawyers_fake_cases/ https://www.theregister.com/2024/02/05/ai_in_brief/

    1. jdiebdhidbsusbvwbsidnsoskebid Silver badge

      Re: Here we go again

      Makes me wonder how often made up stuff like this is being presented and not getting picked up. Hence, how often the course of justice is being perverted.

      1. Doctor Syntax Silver badge

        Re: Here we go again

        The other side will pick it up as will the judge. If they're presented with a citation they don't recognise they'll look it up. They have to because they'll be expected to comment.

      2. MachDiamond Silver badge

        Re: Here we go again

        "Makes me wonder how often made up stuff like this is being presented and not getting picked up. Hence, how often the course of justice is being perverted."

        If a judge is being lazy, they can just take the citations provided as gospel and base their decisions (or allow the argument in front of a jury) on made-up information.

  10. Anonymous Coward
    Anonymous Coward

    They believe the hype

    This is the problem. The law firm implemented their own "internal AI system", because that's what everyone has to have these days. They probably told their lawyers to use it because it's a great way to produce content quickly. And the users took it at face value.

    Sometimes current AI is described as nothing more than a superpowered search engine with a natural language interface. But it's much worse, because if the search doesn't find factual results it will clearly just make something up that fits.

    1. tmTM

      Re: internal AI system

      They're quite obviously using ChatGPT.

      It's not some 'internal tool' it's a lazy and useless idiot throwing prompts at a website and copying the output word for word.

      They quite rightly deserve to be properly sanctioned and have their case thrown out.

      1. collinsl Silver badge

        Re: internal AI system

        It may well be a "legal ChatGPT" which is just the model hosted by some law firm support company somewhere which they then sell on subscription to law firms.

        1. John Brown (no body) Silver badge

          Re: internal AI system

          and it clearly needs a simple update. Any case cited by it should be looked up on the official legal/court/whatever database automatically and confirmed to at least exist in reality. That should eliminate at least some of the hallucinations. It's still going to be the job of some human to actually look at the cited cases to make sure they are actually what is being cited and are relevant.

          1. Richard 12 Silver badge

            Re: internal AI system

            That would require it being able to identify what a citation is.

            LLMs produce text that looks like a court submission. They don't have any internal concept as to what any of it means.

  11. Mike 137 Silver badge

    Course of least resistance

    'In a modest effort to prevent this from happening again, the law firm [...] "added a click box to our AI platform'

    A more appropriate effort would be to effin' well check the output of the LLM yourselves, or is that too much effort?

    1. I ain't Spartacus Gold badge

      Re: Course of least resistance

      It would be so easy to have the computer bring up all the cases that the AI has found for you - like a proper search tool should - and then you could check if they were relevant. This might even act as further training data for your internal AI - when it turns out half of it is picked randomly, and the other half is made up.

      1. John Brown (no body) Silver badge

        Re: Course of least resistance

        yeah, each link could have "spam or ham" buttons on them. We've been training spam filters like that for decades and they still get it wrong! Why would anyone expect AI, with only a few years (at most!) training to be any better?

    2. Anonymous Coward
      Anonymous Coward

      Re: Course of least resistance

      > 'In a modest effort to prevent this from happening again, the law firm [...] "added a click box to our AI platform'

      Just arse-covering. And given how massive an arse that lawyer has been, their webpage is gonna need more than just widescreens to display it.

  12. STOP_FORTH Silver badge
    Trollface

    AI is really useful .....

    For jobs that don't have any purpose.

    Good for reports that nobody reads, marketing bumf that sort of thing.

    Non LLM AI may be useful for spotting cancer, cracks in airframes or girders.

    Do the chattering classes not realise that their game is nearly up?

    Is this the Schadenfreude icon?

    1. MachDiamond Silver badge
      Trollface

      Re: AI is really useful .....

      " may be useful for spotting cancer"

      It's good at spotting melanomas. All of those have a ruler in the frame.

  13. The Central Scrutinizer Silver badge

    "Hi, I see you're trying to produce total bullshit today. That happens to be my specialty"!

  14. heyrick Silver badge
    FAIL

    Disbar all of these lawyers

    Given how much they charge, it's surely dereliction of duty and bringing their profession into (more!) disrepute to obtain, trust, and quote stuff spat out of an AI without so much as even cross referencing to check the citations made sense.

    Any idiot could do that, and even this idiot knows to double check whatever the likes of ChatGPT say.

  15. Howard Sway Silver badge

    These stories come out because court cases are public

    Which is useful for seeing just how stupid and badly wrong using LLMs can go, and how common that is. I wonder how many mistakes are being made in private companies using them and are getting hushed up, because it would be damaging to their reputation and share price to hear that a chatbot mistakenly ordered 200 tons of cheese when asked to do a stock replenishment for an office.

    1. MachDiamond Silver badge

      Re: These stories come out because court cases are public

      "because it would be damaging to their reputation and share price to hear that a chatbot mistakenly ordered 200 tons of cheese when asked to do a stock replenishment for an office."

      Whoops, sorry, that was me.

      Alexa, order 200 tons of cheese.

      Alexa, confirm purchase.

      https://xkcd.com/1807/

  16. nijam Silver badge

    > With a repentant heart...

    Does that suggest he's not a real lawyer at all?

    1. John Brown (no body) Silver badge

      Good spot. Lawyers don't have hearts and most especially not a repentant one!

  17. Bump in the night
    Joke

    Back in my day . . .

    If you didn't know the answer you'd make stuff up.

    Now the computer does it for you.

  18. Ray Donald Pratt

    Any Bozo Who Can Read and Write Can Be An Attorney

    Since the AI itself is not a source of primary legal materials such as case law and statutes, the attorney should have known that he should have looked at the cited materials directly. The cited materials (the "WL" citations) were readily available online, assuming that the firm had a WestLaw subscription.

    This 'mistake' could probably be traced back to overbooking more clients than the firm can properly serve. The 'mistake' might also be traced back to a legal secretary or paralegal who did the research and drafted the motion while the attorney was out trying to meet his busy golfing schedule. At the end, the attorney was forced to take credit for the mistake because revealing the truth would have been worse.

    As a former litigious prisoner who represented himself in a multitude of cases, I can assure you that most attorneys are just Bozos who have training and experience in reading and writing about law; and, thus, that any Bozo who can read and write can become an attorney. I started with zip-zero training and experience, and I ended up winning appeals. See, e.g., Pratt vs. Sumner, 807 F.2d 807 (9th Cir., 1987). I would not have made an AI mistake unless that was my only current source of access to legal materials, and even then I would have put in a note about why I was unable to access primary legal materials.

    The attorneys in the above news story are straight-up Bozos.

  19. Winkypop Silver badge
    Devil

    Lawyers and shysters

    But I repeat myself…

  20. drankinatty

    I hope Judge Rankin sees it that way...

    Mata v. Avianca, Inc, United States v. Hayes, and United States v. Cohen -- 4 strikes and your out. The patience of US Federal District Court Judges has it's limits. This isn't a hallucination of first impression. Once, maybe twice and the mea culpa may carry the day. Being ID-10-T lawyer number-five begins to strain credulity from the judge's perspective.

  21. Stevie

    Bah!

    I used to think lawyers had to be clever.

    Then I met some and realized they only had to be clever twice in their lives: when they sat for their law degree and when they sat for the bar.

    After that, the majority of those I've met or heard about directly from others have been delightfully thick.

    Waiving AI at lawyers was always going to end up with this result.

  22. teebie

    "added a click box to our AI platform"

    This is not a sufficient response to misleading the court. If the person who wrote this was a lawyer, then they should no longer be.

    "With a repentant heart, I sincerely apologize to this court, to my firm, and colleagues representing defendants for this mistake"

    This is more the sort of thing I would hope for

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like