Fake Cases and Fabricated Evidence: AI and the Future of Decision Making
By: Pilar Walton
Edited by: Harris Horowitz and Alexandra Stillman
Two federal judges from Mississippi and New Jersey relied on AI to draft flawed court orders sparking national conversation about the boundaries of artificial intelligence in the justice system. Article III vests the power of judgment to come from human beings. [1] Senator Chuck Grassley began this investigation and found that the AI assistant used by the Mississippi and New Jersey judges generated fake legal quotes and made-up case citations, leading Grassley to call for strict AI regulation. [2] This incident quickly caused public scrutiny and raised questions about judicial ethics, the integrity of legal decision‑making, and the necessity of a statewide mandate. At a time when artificial intelligence is rapidly reshaping public life, episodes like this force one to confront the controversial role of AI in decision making.
While AI has become increasingly popular in numerous professions such as healthcare, education, and marketing, the conversation surrounding AI in the judicial system takes a different approach because of the requirement for human judgement. However, judges and attorneys are discovering new ways using AI as a means of efficiency, whether that be summarizing files, conducting research, or drafting briefs. This generated content consequently influences their capacity to make decisions, as generated case law hinders their ability to think autonomously. From the increased use of AI key questions arise: Can an algorithm make impartial decisions? To what extent should we utilize AI on a case by case basis? How do we distinguish between AI doing judges' busy work and influencing decision making? These questions may help us understand the extent to which developments in technology will benefit the courtroom.
Lawyers and judges alike are using AI as a tool to complete tasks that would otherwise be left to the human brain. The social contract requires a human to work through the complexities of a case, but what happens when administrative tasks and opinions are being left up to a machine? A report from the American Bar Association found that AI misshapes court outcomes almost 34% of the time. [2] When judges rely on fake case law, it weakens public trust in the system and questions the necessity of judges.
As the use of AI grows, many cases have emerged that highlight the significant threats AI poses to fairness of the court system. In early February, a Wisconsin judge sanctioned Kenosha County District Attorney Xavier Solis for failing to disclose his use of artificial intelligence in court filings and for submitting inaccurate case citations. [3] These fabricated citations, known as “AI hallucinations,” set misleading precedents and have become extremely prominent. A database tracking hallucinations found between 700 to 1,000 instances of fabrication in 2025. [4] The Wisconsin case was later dismissed due to the inaccuracy of AI, but this case highlights a broader message of how AI should be used responsibly in the court system to ensure transparency. A study by Stanford Law found that chatbots “hallucinate between 58% and 82% of the time on legal queries,” exemplifying the risk of incorporating AI into legal practices. [5] Without proper safeguards, the use of AI in the court system threatens the egalitarian premises on which the court system was founded upon. This issue is important not only for legal professionals, but everyone who relies on the court system to protect their rights.
Emerging technology is increasingly harmful by undermining trust in the legal system. AI hurts confidentiality and attorney-client relationships due to concerns about accuracy. [6] Clients and judges across the country have felt that their nuanced cases were being handled by a machine rather than from the knowledge of a legal professional. This issue does not solely affect attorney-client relationships, but justices as well. Justice Sallie Manzanet-Daniels of the New York Supreme Court felt misled in a case where a plaintiff was secretive about his use of AI. [7] Machines are not able to replicate the authenticity, knowledge, and experience of an educated judge. While it has the potential to enhance judicial processes, reduce errors, and increase consistency, it can also produce inaccurate outcomes as seen in cases across the U.S. [8]
Courts are not uniform on their stance on AI, with districts across the country varying with respect to requirements and repercussions. [9] In the eastern district of Pennsylvania and northern District of Illinois, the courts implemented a ban on the use of AI as a whole, while the northern district of California prohibits only content that is explicitly labeled as AI-generated. This issue is not relevant to only judges and attorneys, but all citizens. If the public discovers that judicial outcomes are determined by flawed algorithms, public trust in the system fails. While the idea of completely eliminating AI seems impossible, it is necessary to limit the use of these algorithms in order to promote a just courtroom. In response to this issue, AI powered legal research specifically Retrieval Automated Generation (RAG) was implemented with the job of avoiding “AI hallucinations,” [10] Justice demands transparency and accountability, qualities AI does not consistently provide. While there are increasing debates about the benefits and complications of AI, the justice system remains a place that needs human judgement to function effectively. As AI continues to shape policies replacing human expertise it is the responsibility of lawyers and judges to ensure that AI is not fabricating case law and manipulating verdicts.
Notes:
Grassley Chuck, “Grassley Calls on the Federal Judiciary to Formally Regulate AI Use,” press release, Senate Judiciary Committee, October 27, 2025.
American Bar Association, “May It Please the Court: How Do Judges Use Generative AI?” January 21, 2026.
Lehr Sarah, “Kenosha DA Sanctioned for Undisclosed AI Use Leading to Hallucinations in Court Filings,” news report, Wisconsin Public Radio, February 10, 2026.
Charlotin, Damien. “AI Hallucination Cases” 2024.
Stanford Institute for Human-Centered Artificial Intelligence (HAI), “In AI Trial, Legal Models Hallucinate in 1 Out of 6 (or More) Benchmarking Queries,” May 23, 2024.
Brown Charley, “Standing Orders: Courts’ Nascent Governance of AI Practices,” Ballard Spahr LLP, February 20, 2024.
Klein Ezra. “A.I. Lawyer’s Courtroom Debut in New York Raises Ethical Questions.” New York Times, April 4, 2025.
Fine, A., Berthelot, E. R., & Marsh, S. (2024). Public perceptions of judges’ use of AI tools in courtroom decision-making: An examination of legitimacy, fairness, trust, and procedural justice.
Brown, C. F., Walters, N., Watkins, C. G., & Blasberg, E. (2024, February 20). Standing orders: Courts’ nascent governance of AI practices. Ballard Spahr LLP.
Stanford Institute for Human-Centered Artificial Intelligence (HAI), “In AI Trial, Legal Models Hallucinate in 1 Out of 6 (or More) Benchmarking Queries,” May 23, 2024.
Bibliography:
American Bar Association. “May It Please the Court: How Do Judges Use Generative AI?” January 21, 2026.
Brown, Charley F., Neal Walters, Casey G. Watkins, and Erin Blasberg. “Standing Orders: Courts’ Nascent Governance of AI Practices.” Ballard Spahr LLP. February 20, 2024.
Charlotin, Damien. “AI Hallucination Cases.” 2024.
Fine, A., E. R. Berthelot, and S. Marsh. “Public Perceptions of Judges’ Use of AI Tools in Courtroom Decision-Making: An Examination of Legitimacy, Fairness, Trust, and Procedural Justice.” 2024.
Grassley, Chuck. “Grassley Calls on the Federal Judiciary to Formally Regulate AI Use.” Press release. Senate Judiciary Committee. October 27, 2025.
Klein, Ezra. “A.I. Lawyer’s Courtroom Debut in New York Raises Ethical Questions.” New York Times, April 4, 2025.
Lehr, Sarah. “Kenosha DA Sanctioned for Undisclosed AI Use Leading to Hallucinations in Court Filings.” Wisconsin Public Radio, February 10, 2026.
Stanford Institute for Human-Centered Artificial Intelligence. “In AI Trial, Legal Models Hallucinate in 1 Out of 6 (or More) Benchmarking Queries.” May 23, 2024.