Andrew Blair-Stanek recently co-authored a work published in the Proceedings of the Natural Legal Language Processing Workshop 2024 shedding some light on the use of AI in legal practice and further cementing his position in the vanguard of this transformative technology.
Large Language Models (LLMs) have rapidly gained attention in the legal profession for their ability to assist in generating legal analysis, drafting documents, and even streamlining complex tasks like legal research. These models, fueled by vast amounts of legal data and sophisticated machine learning techniques, hold the potential to transform how legal professionals approach their work.
Blair-Stanek, with Nils Holzenberger, and Benjamin Van Durme, published an article entitled, “BLT: Can Large Language Models Handle Basic Legal Text?”. The study examines the ability of LLMs to do the most basic tasks with legal text. For the study, the researchers developed a benchmark for Basic Legal Text (BLT) processing, designed to mimic common tasks that new lawyers and paralegals handle daily. Examples include tasks like retrieving citations, locating statutory references with specific terms or definitions, and finding explicitly referenced contract sections. After testing different models from OpenAI, Google, and Anthropic, the results indicated that the “currently available LLMs perform poorly out-of-the-box on basic legal text handling.” (An example of this poor performance is in the attached figure.). However, with fine-tuning of the BLT’s training data, results showed improved performance on higher-level legal tasks to come closer to the expected abilities of new lawyers or paralegals.
The integration of LLMs into legal practice holds immense potential, but it requires careful consideration of their strengths and limitations. The work of Blair-Stanek et al. is making significant strides in resolving issues in the use of LLMs in legal practice. LLMs could become invaluable tools--helping legal professionals save time, improve productivity, and enhance the quality of their work. However, they still have substantial shortcomings.