June 23, 2023
ChatGPT-4's ability to understand and respond to natural language makes it a powerful tool for improving communication and automating tasks that would otherwise require human intervention. LRDC Senior Scientist Kevin Ashley and his co-authors explore whether GPT-4 can be used to explain legal concepts.
Main Takeways from this Research:
- Using the augmented large language model (LLM) approach described in this paper to generate explanations for laypeople could be a valuable way to increase access to justice.
- Employing the augmented LLM approach to other tasks in the legal domain could be a promising way to harness the power of LLMs.
Legislative bodies enact statutes, which are written laws that establish sets of legally enforceable rules. These statutory provisions can be challenging to understand because they must take into account various circumstances, including those that have not yet occurred. They can be vague, and often rely on abstract language to deal with this uncertainty. Interpreting the meaning of statutory provisions is a key task of legal professionals, and an important source for this interpretation is how the term was applied in previous court cases.
In the work described here, the authors investigated whether large language models (LLM), such as GPT-4, can be combined with existing legal information retrieval (IR) methods to automatically draft explanations of how previous courts explained the meaning of statutory terms. If successful, such a system could have an important impact in assisting lawyers in accomplishing their work more efficiently, or supporting judges in deciding how a term should be applied to a new situation. Eventually, it could also make the law more accessible for the public.
The research team compared the performance of a baseline approach, where GPT-4 was directly asked to explain a legal term, to an augmented approach, where a legal information retrieval (IR) module was used to provide relevant context, in the form of sentences from case law. They then employed two legal annotators with extensive experience in semantic annotation of legal texts, to evaluate the explanations. The talk of the annotators was to compare the explanations in terms of five categories: Factuality, Clarity, Relevance, Information Richness, and On-pointedness.
They found that the direct application of GPT-4 yielded explanations that appeared to be of very high quality on their surface. Yet, detailed analysis uncovered limitations in terms of the factual accuracy of the explanation. The augmentation of GPT-4 with a legal information retrieval (IR) component led to improved quality, and appeared to eliminate the issue of hallucination, where LLMs state very plausible sounding, but, made-up information.
The augmented LLM approach, that combined traditional methods in legal information retrieval with the power and flexibility of language models, can play an important role in legal information. These findings can contribute to the building of systems that can autonomously retrieve relevant sentences from case law and condense them into a useful explanation for legal scholars, educators or practicing lawyers.
Read the full paper "Explaining Legal Concepts with Augmented Large Language Models (GPT-4)".
Savelka, J., Ashley, K.D., Gray, M.A., Westermann, H., & Xu, H. (2023). Explaining Legal Concepts with Augmented Large Language Models (GPT-4). Proceedings of the International Conference on Artifical Intelligence and Law (ICAIL 2023), University of Minho Law School, Braga, Portugal, https://arxiv.org/pdf/2306.09525.pdf