Judges And Courts Need To Be Asking These Crucial Questions Of Lawyers Caught Red-Handed Using AI Hallucinations

Use Of AI Hallucinations In Court Cases

As readers well know, I’ve been extensively covering and analyzing a myriad of facets regarding the intersection of AI and the law for many years. You can find my writings not only in my Forbes column but also as posted in Bloomberg Law, ABA Law Journal, The National Jurist, The Global Legal Post, Lawyer Monthly, The Legal Technologist, MIT Computational Law Journal, and so on.

I have extensively examined the repercussions for attorneys who have used AI-hallucinated elements in their legal briefs; see my coverage at the link here and the link here. The situation usually is that an attorney opts to include fake legal citations or fictitious quotations about legal cases, having generated those via the sloppy use of generative AI and large language models (LLMs).

I’ve frequently noted that until the courts impose stiffer penalties, or until lawyers wake up and use suitable double-checking procedures and tools, this phenomenon is going to continue. There is a constant drumbeat of lawyers contending they had no idea that AI could produce bad references or bogus quotes. Though this might have seemed plausible when generative AI first hit the scene, the claims of innocence now seem quite hollow. Most savvy attorneys raise their eyebrows when a caught red-handed lawyer tries to plead ignorance of these matters. It does not compute anymore.

Background On The Happenings

Daily bulletins in the legal community keep highlighting lawyers who have used AI to prepare their court filings and ended up with fake legal citations and false quotations in their documents. These errors in legal briefings can potentially occur due to AI hallucinations. An AI hallucination is when generative AI or large language models (LLMs) such as ChatGPT, Claude, Grok, Gemini, CoPilot, Llama, and other AIs veer into generating fictitious confabulations. For my in-depth coverage of AI hallucinations, see the link here and the link here.

I had long ago predicted that attorneys using AI might get careless in their legal efforts and allow the AI to produce bogus content and not double-check by hand what the AI has generated for them (see my prediction in 2023, at the link here).

The problem for attorneys is that when they submit formal court filings, they are supposed to be responsible for the contents of the filings; thus, if AI has slipped in faked citations or false quotations, the lawyer is likely to be held accountable since they didn’t catch the erroneous content. I say “likely” accountable because judges and courts have been quite lenient so far, overall, and allowed excuses such as “the computer did it”. Attorneys often incur nothing more than a minor hand slap or mild rebuke, asserting that AI is new to the legal beagles and they were caught unawares. Sometimes, the reprimands are accompanied by a modest financial sanction or penalty, which is gradually ratcheting up as these instances continue to climb.

Judges and courts are beginning to lessen their patience and sense of charity in giving lawyers the benefit of the doubt. For the time being, the matter is still generally being treated with kid gloves. The expectation is that lawyers are facing a learning curve, and a bit more time is required before they will be able to properly handle the use of AI. In my view, more than enough time has already passed.

Stop coddling legal pros that are supposed to remain on top of the latest technological aspects when performing their legal services (see my coverage of ABA Formal Opinion 512, at the link here, which was promulgated on July 29, 2024, nearly two years ago!).

Twists And Turns Aplenty

I recently examined the somewhat rare situation of attorneys that fail to detect that the opposing side has made use of AI hallucinations in their legal filings in a court case; see the link here. The question there is whether the attorney that doesn’t catch the mistakes of their opposition ought to receive any sanctions or penalties. There are tradeoffs involved, which I analyzed at length in my posting.

There is another new twist that is garnering attention, namely the exasperating circumstance of both sides of a case relying on AI hallucinations at the same time; see my analysis at the link here. You might be shocked to think that this could happen. It’s one thing for an attorney on one side to make such an error, but having both simultaneously do this is astonishing and dismaying. It is a justice-undermining abomination since neither side caught the other side doing so. This is an affront to the adversarial system of justice.

The Handiness Of A Good Set Of Questions

Some judges have no idea what kinds of questions they should ask when an attorney is caught using AI-hallucinated citations or quotations. It is new territory for them. Maybe the judge knows very little about AI. A few years ago, many judges were in the dark about AI hallucinations. Since they didn’t understand AI, they were willing to give grace to attorneys that seemingly fell into AI traps. The judges also didn’t want to venture into a topic that they themselves were confused about or uninformed on. This opened Pandora’s box for attorneys to slide into using AI-hallucinated content.

I’ve put together a helpful set of probing questions that you can easily use as a written inquiry to an attorney that allegedly has made use of AI hallucinations. I say allegedly because the foul citations and quotations don’t necessarily have to be produced by AI. Perhaps a paralegal or assistant composed the false items themselves by hand or by doing online searches. Maybe the attorney did so.

Believe it or not, some attorneys realize they are on safer ground to claim that AI did it, rather than admitting to the fact that they did so by hand or someone else manually did it. Why in the world would an attorney want to pin this on AI? Because they know that AI as an excuse is a handy cover story. The attorney becomes the unfortunate victim. It’s that dastardly AI that tricked them.

This has been working well with some courts, so it is absolutely a worthy Hail Mary plea. Usually, the judge will shrug their shoulders, acknowledge the mysteries of AI, and move on. The lawyer doesn’t have to reveal the true ugly aspects of how they got themselves into the mess.

Hard-Boiled Questions At The Ready

I provide here a set of over one hundred questions that are arranged into twelve distinct sections and can be readily used by judges and courts. The questions at times are purposely semi-overlapping to try and discern that the responses are consistent and fully considered.

The twelve sections consist of:

• Section 1: Identifying And Explaining The AI Hallucinated Content
• Section 2: Attorney Awareness and Timeline
• Section 3: CLE and Professional Education
• Section 4: Bar Association Communications
• Section 5: Law Firm Policies
• Section 6: Prior Exposure to AI Warnings
• Section 7: Use of AI Prior to the Filing
• Section 8: Vendor Warnings
• Section 9: Conduct During Preparation of the Filing
• Section 10: Professional Judgment
• Section 11: Contradictory Conduct
• Section 12: Credibility Testing

The set is not cast in stone. You are welcome to remove sections, add sections, remove questions, add questions, reword questions, and so on. Generally, I don’t recommend removing any of the sections since they each provide a crucial glimpse into the matter at hand. It’s up to you to determine what is best for the circumstance.

Let’s go through the sections and questions. Afterward, I will have some overarching remarks.

Section 1: Identifying And Explaining The AI Hallucinated Content

1. How many AI-hallucinated elements are in your legal filing?
2. Provide a detailed listing of each AI-hallucinated element that is contained in your legal filing, numbering each element for ease of reference, and including a cross-reference to pinpoint where the element can be found in the legal filing.
3. Are you sure that those are all of the AI-hallucinated elements in your legal filing, or do you believe there might be more (and if so, explain why you believe there might be more, and explain why you don’t know whether there are more AI-hallucinations in the legal filing)?
4. Affirmatively attest that these are all AI-hallucinated elements and not devised or composed via some other method or means other than AI (if any are not produced via AI, explain why you have chosen to include them in this listing).
5. Which AI were you using that produced these AI-hallucinated elements (if more than one AI was used, indicate all AIs that were used).
6. For each listed AI-hallucinated element, indicate the date and time when the element was generated by AI (provide a transaction log of your AI usage that can verify the date and time when each was generated by AI), and additionally indicate the date and time when you included the element into your legal filing.
7. For each listed AI-hallucinated element, provide the prompt or series of prompts that led to the generation of the AI-hallucinated element.
8. For each listed AI-hallucinated element, indicate whether it was you that used the AI or whether it was someone else (if it was someone else, explain who they are, why you were using them for this task, and any other pertinent background information about said person, along with current contact information of that person so that the person can be contacted by this court).
9. For each listed AI-hallucinated element, explain how you have subsequently double-checked to verify that the element is indeed false.
10. For each listed AI-hallucinated element, indicate the impact and severity thereof associated with the arguments and points made in your legal filing, stridently stipulating how their respective falsehood undermines and undercuts your filing.
11. Include any additional background, explanation, or other elicitation that you believe is significant to this matter.

Section 2: Attorney Awareness and Timeline

1. When did you first become aware in general of generative AI?
2. When did you first become aware of ChatGPT or similar systems?
3. Before filing the document in question, what did you believe generative AI was capable of doing?
4. What did you believe the AI was incapable of doing?
5. Did you believe the AI could make mistakes?
6. If so, what kinds of mistakes?
7. Did you believe the AI always produced accurate information?
8. If yes, what led you to that belief?
9. Have you ever heard the term “AI hallucination”?
10. If so, when did you first hear or read that term?
11. If not, how do you explain never encountering that term?
12. Include any additional background, explanation, or other elicitation that you believe is significant to this matter.

Section 3: CLE and Professional Education

1. List all CLE courses completed during the past five years.
2. Produce certificates of attendance for those CLE programs.
3. Produce course descriptions for those CLE programs and provide as much additional detail as possible, including syllabi or any other information.
4. Which of the CLE programs involved legal technology?
5. Which of the CLE programs involved AI?
6. Which of the CLE programs involved legal research tools?
7. Which of the CLE programs discussed attorney competence regarding technology?
8. Which of the CLE programs discussed verification of electronically obtained information?
9. Did any of the CLE programs discuss AI-generated hallucinations, fabrications, confabulations, errors, etc. (if so, describe what was covered)?
10. Did any of the CLE materials specifically mention the possibility of AI-hallucinated citations and/or AI-hallucinated quotations (if so, describe what was covered)?
11. Did you take notes during those courses (if so, produce those notes)?
12. Include any additional background, explanation, or other elicitation that you believe is significant to this matter.

Section 4: Bar Association Communications

1. Have you ever been a member of any bar association (if so, list the name of the bar association and indicate the time period of your membership)?
2. Have you ever received or been made aware of any communication from any bar association regarding the topic of AI?
3. At the time that you made use of AI hallucinations in your legal filing, were you aware of the ABA position on generative AI in legal practice (such as, but not limited to, Formal Opinion 512)?
4. Are you currently aware of the ABA position on generative AI in legal practice (such as, but not limited to, Formal Opinion 512)?
5. Explain how your use of AI hallucinations is a potential violation of the ABA position on the proper use of generative AI in legal practice (such as, but not limited to, Formal Opinion 512).
6. Include any additional background, explanation, or other elicitation that you believe is significant to this matter.

Section 5: Law Firm Policies

1. Are you in a solo practice or are you in a law firm with more than one lawyer?
2. Does the law practice have a policy about AI (if so, provide a complete copy of the AI policy, indicate when the policy was first adopted, when the policy became available for you to read, and when you first read the policy)?
3. Does the law practice have any review process associated with verifying citations, quotations, and the verification of legal filings (if so, was that process utilized for the legal filing in question)?
4. Does the law practice provide any training on AI (if so, indicate the type of training, and when the training was available to you, and when you undertook the training if you have done so)?
5. Include any additional background, explanation, or other elicitation that you believe is significant to this matter.

Section 6: Prior Exposure to AI Warnings

1. Have you ever read any news articles about AI (if so, list them)?
2. Have you ever watched any television or online video coverage of AI (if so, list them)?
3. Have you ever listened to any podcasts discussing AI (if so, list them)?
4. Have you ever attended any conferences of any kind that discussed AI (if so, list them)?
5. Were you aware of any judicial opinions involving AI hallucinations (if so, what was your understanding of those judicial opinions)?
6. Include any additional background, explanation, or other elicitation that you believe is significant to this matter.

Section 7: Use of AI Prior to the Filing

1. How long overall have you been using generative AI (indicate the length of time for your general usage, and indicate the length of time for use in your legal efforts)?
2. Have you used AI for legal research before?
3. Have you used AI for drafting before?
4. Have you used AI for summarization?
5. Have you used AI for citation generation?
6. Did you ever observe AI making factual mistakes?
7. Did you ever observe AI producing incorrect information?
8. Did you ever observe AI giving contradictory answers?
9. Did you ever observe AI inventing facts?
10. Did you ever question the reliability of AI outputs?
11. Did you ever verify AI outputs before this incident?
12. Include any additional background, explanation, or other elicitation that you believe is significant to this matter.

Section 8: Vendor Warnings

1. Were you using general-purpose AI or a specialized AI-based product?
2. Did you review the terms of service?
3. Did you review the user documentation?
4. Did you review the FAQ materials?
5. Did you review any training videos?
6. Did you review any onboarding materials?
7. Did those materials contain warnings?
8. Did those materials contain disclaimers?
9. Did they warn that outputs may be inaccurate?
10. Did they warn that outputs should be independently verified?
11. Did they warn that the system can make mistakes?
12. Did they warn against relying on outputs without review?
13. Did you read those warnings (if not, why not?)?
14. Include any additional background, explanation, or other elicitation that you believe is significant to this matter.

Section 9: Conduct During Preparation of the Filing

1. Did the AI generate legal authorities?
2. Did the AI generate quotations?
3. Did the AI generate citations?
4. Did you ask the AI to provide supporting cases?
5. Did you ask the AI whether those cases were real?
6. Did you independently verify any of them?
7. Did you search for them in legal databases?
8. Did you attempt to locate the opinions?
9. Did you read the opinions?
10. Did you quote language from cases you had not personally reviewed?
11. Did you cite authorities you had not personally reviewed?
12. Include any additional background, explanation, or other elicitation that you believe is significant to this matter.

Section 10: Professional Judgment

1. Would you normally cite a case without reading it?
2. Would you normally quote a case without reading it?
3. Would you normally rely on a law clerk’s citation without verification?
4. Would you normally rely on a paralegal’s citation without verification?
5. Why was AI treated differently?
6. What basis did you have for trusting the AI?
7. Did you perform any reliability assessment?
8. Did you test the system before relying on it?
9. Why did you believe verification was unnecessary?
10. Include any additional background, explanation, or other elicitation that you believe is significant to this matter.

Section 11: Contradictory Conduct

1. Have you ever warned anyone about AI errors?
2. Have you ever discussed AI hallucinations with colleagues?
3. Have you ever received emails discussing AI hallucinations?
4. Have you ever attended meetings discussing AI risks?
5. Have you ever discussed AI risks with clients?
6. Have you ever posted on social media about AI?
7. Have you ever commented on articles discussing AI?
8. Have you ever searched online for AI reliability?
9. Have you ever searched online for AI hallucinations?
10. Have you ever asked AI whether it could make mistakes?
11. Have you ever instructed AI to provide sources?
12. Have you ever asked AI to verify its answers?
13. Include any additional background, explanation, or other elicitation that you believe is significant to this matter.

Section 12: Credibility Testing

1. Are you testifying under oath that before this incident you had never heard that AI could generate false information?
2. Are you testifying that you had never heard AI could fabricate cases?
3. Are you testifying that you had never heard AI could fabricate citations?
4. Are you testifying that you had never heard AI could fabricate quotations?
5. Are you testifying that you had never heard AI could provide inaccurate legal research?
6. Are you testifying that no CLE, article, email, colleague, conference, training session, vendor warning, bar publication, law-firm policy, news report, podcast, webinar, or professional communication ever informed you of these risks?
7. Is that your testimony under penalty of perjury?
8. Is there any document, email, training record, conference attendance record, CLE record, policy acknowledgment, or communication that would contradict that testimony?
9. Have you conducted a diligent search to determine whether such documents exist?
10. Will you agree to produce all responsive materials?
11. Is there any reason the Court should conclude that your present testimony is incomplete or inaccurate?
12. Include any additional background, explanation, or other elicitation that you believe is significant to this matter.

Anticipating The Reaction

I am sure that any attorney who is given such a set of questions is going to balk at the length and magnitude of what they are being asked to respond to. Again, it’s up to you as to whether to let the attorney off the hook. For any question you remove, you are generally allowing a loophole that the attorney can possibly use to avoid being fully apparent on what has transpired.

The purpose of the questions is to get to the truth of the matter. It is not some kind of punishment or idle fanfare. Any attorney caught using AI hallucinations should be fully prepared to straightforwardly explain themselves. If they won’t do so, I’d suggest that your Spidey-tingling sensation should be triggered.

The questions are entirely focused on the issues entailing AI hallucinations. There is no dilly-dallying. There isn’t any fluff. Each section and each question is entirely appropriate and applicable.

By providing these written questions to the attorney, you can then hold them accountable to what they are supposed to do when responding to you on this matter. And, importantly, you are to ensure that they provide their answers in writing. This process reduces chances of claims of confusion, bewilderment, and other whininess. They are given a clear-cut set of questions and expected to provide a clear-cut set of responses.

Taking The Hard Road

Some judges might prefer to nonchalantly ask the attorney what happened, listen to their responses, and swiftly decide what to do. It is an easy-going inquiry. This would seem cordial and collegial. Do a brief query. Move on.

That’s not an advisable process.

Keep in mind that any declaration of sanctions, penalties, rebukes, etc. will potentially be considered by the attorney as unfair, unbalanced, improper, and they are bound to legally aim to fight the decree. Even if the attorney during the chat seems low-key, contrite, and eager to conclude the matter, do not expect they will take the thumping with aplomb. Few would. The meek cat will later roar like a lion.

A judge should anticipate that an appeal will possibly be undertaken. If that arises, a judge that did a mere off-the-cuff decision is probably going to be on much looser ground than one that had the attorney complete an extensive questionnaire, and for which the responses became fodder for the ascertained consequences. A mindful judge should explain in writing their logic of how they reached the declared consequences, doing so as cogently tied to the written questions and written answers.

Remember the wise words of John Quincy Adams: “Whoever tells the best story wins.”