Artificial Jurisprudence: Might AI Ever Be Tasked With Adjudicating the Law?

By: Jonah Berman

Edited by: Regan Cornelius and Colin Crawford

Artificial intelligence and its inevitable confrontation with established practices across numerous industries have become a fervent topic of discussion lately. The realm of law and legal practice stands out as particularly intriguing in this context given that, traditionally, the sector has depended on the nuanced judgment and expertise of practitioners proficient in the complexities of law. Yet, it now faces the fascinating prospect of transformation through AI. For instance, Large Language Models (LLMs) like ChatGPT are increasingly handling substantial volumes of administrative work. An LLM can meticulously search and analyze case files, compile evidence, and construct legal arguments imbued with strong precedents. A Princeton study identified legal services as one of the most susceptible professions to disruption by the advent of LLMs. It's clear that AI will impact the field of law. But, might it ever evolve to actually adjudicate law itself?

Malcolm Gladwell, in his 2019 book Talking to Strangers, cites a compelling case of man versus machine. A Harvard researcher led a significant study comparing bail decisions made by human judges in New York City with those determined by an AI system. This analysis involved records of over half a million defendants from arraignment hearings between 2008 and 2013, with the AI tasked with selecting 400,000 individuals for release based on the same data judges had. The judges had the advantage of meeting the defendants in person and considering additional courtroom information, unlike the AI, which relied solely on the given data. Astonishingly, those selected by the AI were 25% less likely to re-offend while on bail compared to those chosen by human judges. Furthermore, the AI accurately predicted a high probability of reoffending in over half of a "high-risk" group it identified. In short, the machine won this judicial “face-off”.

The potential of AI in law adjudication offers a range of benefits. AI's capacity for processing vast data impartially may lead to more consistent and precise legal decisions, as demonstrated in Gladwell's case study. An LLM can effortlessly recall any case detail, law, or even recite the entire Constitution word for word. Such comprehensive knowledge grants LLMs a significant advantage in law adjudication, allowing them to consider a broad spectrum of implicit factors and historical contexts with complete objectivity–ideally.

Another aspect worth considering is LLMs’ competence in addressing moral and ethical dilemmas. Remarkably, LLMs have already shown high proficiency in understanding human morals. One study found a .95 Spearman correlation coefficient between GPT 3.5 and human responses to moral queries, indicating an extremely strong correlation between the two sets of responses.

Nonetheless, there are significant concerns about LLMs that would hinder their ability to adjudicate law. Evident biases in AI systems have been a major issue. As AI language models are trained on pre-existing data, they may inadvertently perpetuate and amplify societal biases inherent in that data. The COMPASS system, used in the U.S. criminal justice system, exemplifies this; an investigation highlighted its disproportionate labeling of black defendants as high-risk compared to white defendants, raising questions about racial bias in AI evaluations.

Additionally, LLMs sometimes produce inaccurate responses or can be misled by complex or ambiguous prompts. Whether an LLM can adjudicate law differs from whether it should adjudicate law. I argue that LLMs are—and will soon be even more so—exceptionally equipped for legal judgment. Given that ChatGPT was launched a little over a year ago, LLMs are still in an inchoate phase. AI researchers and industry leaders are well aware of the current flaws facing LLMs; it is clear that the issues of inaccuracy and bias will be properly addressed in time. Once these concerns are mitigated, LLMs will be able to adjudicate law with greater knowledge, impartiality, and efficiency than human judges. However, the inherently subjective nature of law, with its myriad interpretations and evolving viewpoints, remains a critical barrier. The capacity for legal interpretation is not something humans are likely to relinquish. For instance, if an LLM were assigned to decide on the federal legalization of abortion in the U.S., its approach would be multifaceted, considering moral, constitutional, and historical aspects. Nevertheless, people would struggle to adhere to the ruling of a machine. The imperfect, sometimes contradictory nature of law, with its capacity for both justice and failure, is intrinsically human. While AI may in theory soon become a superior alternative for legal judgment, it is hard to imagine its full acceptance in practice. More likely, there will start to be a rise in the use of AI to augment judges’ ability to make the best decisions: a symbiotic relationship that harnesses the breadth and power of AI while still keeping the judiciary human.

Notes: 

1. Tawakol, A. (2023, May 25). “Will AI Replace Lawyers?” Forbes. Retrieved from https://www.forbes.com/sites/forbestechcouncil/2023/05/25/will-ai-replace-lawyers/?sh=58271a373124

2. Felten, E., Raj, M., & Seamans, R. (2023). “How will Language Modelers like ChatGPT Affect Occupations and Industries?” Retrieved from https://arxiv.org/pdf/2303.01157.pdf

3. Gladwell, M. (2019). Talking to Strangers. Retrieved from https://www.ericfrayer.com/wp-content/uploads/2019/11/Talking-to-Strangers.pdf

4. Dillon D., Tandon, N., Gu, Y., Gray, K. (2023). Can AI Language Models Replace Human Participants? Trends in Cognitive Sciences. Retrieved from https://www.cell.com/trends/cognitive-sciences/fulltext/S1364-6613(23)00098-0

5. Hao, K. (2019, October 17). AI is fairer than a judge. MIT Technology Review. Retrieved from https://www.technologyreview.com/2019/10/17/75285/ai-fairer-than-judge-criminal-risk-assessment-algorithm/

6. Aboze, J. (2023, August 7). Risks of Large Language Models. DeepChecks. Retrieved from https://deepchecks.com/risks-of-large-language-models/

Bibliography:

Aboze, J. (2023, August 7). Risks of Large Language Models. DeepChecks. Retrieved from https://deepchecks.com/risks-of-large-language-models/

Dillon, D., Tandon, N., Gu, Y., & Gray, K. (2023). Can AI Language Models Replace Human Participants? Trends in Cognitive Sciences. Retrieved from https://www.cell.com/trends/cognitive-sciences/fulltext/S1364-6613(23)00098-0

Felten, E., Raj, M., & Seamans, R. (2023). How will Language Modelers like ChatGPT Affect Occupations and Industries? Retrieved from https://arxiv.org/pdf/2303.01157.pdf

Gladwell, M. (2019). Talking to Strangers. Retrieved from https://www.ericfrayer.com/wp-content/uploads/2019/11/Talking-to-Strangers.pdf

Hao, K. (2019, October 17). AI is fairer than a judge. MIT Technology Review. Retrieved from https://www.technologyreview.com/2019/10/17/75285/ai-fairer-than-judge-criminal-risk-assessment-algorithm/ 

Tawakol, A. (2023, May 25). Will AI Replace Lawyers? Forbes. Retrieved from https://www.forbes.com/sites/forbestechcouncil/2023/05/25/will-ai-replace-lawyers/?sh=58271a373124.