The Court of Common Pleas of Butler County has issued the first general administrative order in Pennsylvania (and one of the first in the U.S.) governing court filings produced using “generative artificial intelligence” (also known as “large language models,” such as ChatGPT). The order requires that filings include an affidavit attesting that generative AI was not used in the perparation of the filing or, if AI was used, that “each and every citation to the law or the record in the filing has been verified by a human being as authentic and accurate.” “Use of Generative Artificial Intelligence; MsD No. 2024-40258” (9/10/2024), 54 Pa.B. 5980 (9/21/2024).

Large language models are known to “hallucinate” or fabricate legal citations when asked to write legal briefs. In one publicized instance, a lawyer used ChatGPT to prepare a brief and was sanctioned when the brief contained citations to non-existent cases. “Here’s What Happens When Your Lawyer Uses ChatGPT,” New York Times (5/27/2023). One study of general-purpose “chatbots” found that they hallucinated between 58% and 82% of the time on legal queries and, in a more recent study, legal AI tools from LexisNexis and Thomson Reuters (Westlaw) still produced incorrect information (or hallucinated”) a significant amount of the time (17% and 34%, respectively). “A New Study Reveals the Need for Benchmarking and Public Evaluation of AI Tools in the Law,” Stanford University Human-Centered Artificial Intelligence (5/23/2024).

So practitioners should use care when using AI tools for legal research or first drafts of pleadings or documents.