Law and Large Language Models, A Glimpse of the Future
Written by Precedent AI
Imagine you are a young legal aid lawyer for immigration and asylum cases. You are rightfully proud of your job and care a lot about the difference you make to people’s lives. You are also chronically overworked. According to a recent report from the Public Law Project, there is a 73% chance that you are working overtime every week, and there is a 33% chance that in just five years’ time you will leave immigration and asylum legal aid practice due to burnout.
Now imagine that there is an AI tool that could make your work not twice, not three times, but 26 times more efficient. A tool that could reduce a roughly 13-hour workflow – of case review, of research and reading legal precedent, and of memo writing evidence-based advice – to under 30 minutes, with even greater thoroughness and attention to detail. Imagine how that would transform your life and those of your clients. It’s almost unimaginable.
A hackathon is nothing if not a concerted effort to manifest the previously unimaginable through the power of computer programming. King’s College recently played host to the LLM X Law Hackathon, organised by King’s Entrepreneurship Lab and Stanford CodeX. 150 participants from 28 universities across the world, arrived to tackle a range of legal challenges set by the event’s sponsors, with hackers travelling in from Australia, China, France, Germany, India, Poland, Slovenia, Switzerland, and the United States to name just a few.
On arrival I met up with my friend and former co-author, the excellent computer scientist Neo. We settled on the Amazon Web Services challenge to devise a method for systematic source tracking in the generated content of LLMs. The challenge bore a close resemblance to my own application of AI in my historical research and was a problem I’d thought about for some time. At breakfast we met two enthusiastic computer science MPhil students, Deepro and John, and formed a team.
Two slides in the keynote speech by Kimberly Boadu from Google DeepMind really set the tone for the day: dinosaurs and an asteroid. Clearly AI is poised to transform the legal profession. With this inspiration in mind, our team found a quiet space and got to work. As I discussed some 14 exhausting but rewarding hours later with Jay Mandal from Stanford, the true value of these hackathon events resides precisely in the time-constraint. This focuses interdisciplinary teams on a powerful common purpose that helps melt away barriers between subject areas, and between strangers, often leading to remarkable outcomes.
We named our tool Precedent AI and, deciding that we wanted to help legal aid lawyers in their important work, we scraped a large dataset of cases from the United Kingdom Asylum and Immigration Tribunal 1973-2010. This is a key corpus of precedent for lawyers in this field, but current tools for searching such corpora are poor, comprising of minimal keyword search. With limited time and the thrill of competition for generous prizes, we divided forces to focus on different facets of our solution.
We wanted the user to be able to ask a question about legal precedent and have the LLM accurately deploy direct quotations from the original sources in its reply, as well as providing bibliographic links to these sources so that the user could verify them and even ask follow-up questions. Incredibly, John and Deepro achieved all this during the hackathon. We also ensured that if asked about something that was not contained within the dataset, the model would decline to answer citing a lack of sources, rather than making something up, further mitigating the hallucination risk.
At the same time, Neo was working to implement a new technique from Stanford which had been published just a few days prior. This TextGrad technique enables users to optimise their question for the LLM to improve the quality of the answer they receive. Remarkably Neo managed to improve the accuracy of the Claude 3 LLM by 10% on a small dataset of LSATs (US Law School Admission Tests) all on the day of the hackathon.
While the other team members focused on the precedent search component of our solution, I designed a new end-to-end workflow that took the needs of the lawyer-users into account. This included a case review stage in which a large-context-window model summarises the facts of the case and suggests key legal concepts for which to research precedent, and a memo drafting stage in which another model automatically combines the outputs of the Case Review model and the Precedent Search model to prepare a draft.
I can vividly remember the moment that our tool first began to work, shortly before the deadline. I typed the question, “how has parenthood of minors affected the chances of asylum seekers in their application for citizenship?”. There was a tense silence for a few seconds as the model worked. Then euphoric shouts as it began to reply, “based on the legal cases discussed… the presence of children… weighs in favour of allowing appeals…” The model went on to accurately cite Article 8 of the European Convention on Human rights and directly quoted 4 cases from our dataset, providing a bibliographic link to each. It was a moment in which we all glimpsed the future of legal research and felt the thrilling profundity of the changes such technology unlocks.
After a first round of pitching, we had made it into the top 12 teams and assembled in Keynes Hall to hear the final presentations. One judge later told us that our sheer enthusiasm shone through in our final pitch, as we took it in turns to share our ideas with our fellow hackers. What a perfect end to the day that we won first place! I cannot remember a more intellectually stimulating (or financially rewarding) single day during my time in academia, and many others who participated gained the greater prize, as did I, of new friends.
Precedent AI
Jacob Forward
Jacob is an E-Lab member and PhD student in the Faculty of History, evaluating AI Language Models as tools for Historical Research. He previously read for an MPhil in American History at Cambridge (King’s College), and a BA in History at Oxford (Keble College). He has worked for History and Policy at the Institute for Historical Research and consulted on digital research projects at the School of Advanced Study.
Xiaochen Zhu (Neo)
Xiaochen Zhu is a PhD student in the Department of Computer Science and Technology, supervised by Professor Andreas Vlachos. His research primarily focuses on Natural Language Processing, particularly in Dialogue Agents and Non-Autoregressive Generation. He previously read for an MPhil in Advanced Computer Science at Cambridge (King’s College) and a BSc in Artificial Intelligence and Computer Science at the University of Edinburgh.
Deepro Choudhury
Deepro is an MPhil student in Advanced Computer Science at the University of Cambridge, holding a BEng in Joint Mathematics and Computer Science from Imperial College London. He is interested in the intersection of mathematics and computer science, working on machine-learning techniques for automating formal mathematical theorem proving. He is also interested in interpretable machine learning and understanding the internal mechanisms of large language models.
John Poole
John likes using computers for good. He’s previously worked on population epidemiology for adolescent suicide prevention and neural networks for discovering scientific formulae. His pastime is theoretical computer science. He reads the MPhil in Advanced Computer Science, studying Graph Isomorphism and Polynomial Factoring over Finite Fields, and holds a BA in Computer Science from Reed College in the United States.
