Detection of Suicidality Through Privacy-Preserving Large Language Models
This is a virtual seminar. For a Zoom link, please see "Venue". Please consider subscribing to mailing list: web.maillist.ox.ac.uk/ox/subscribe/ai4mch
Importance Attempts to use Artificial Intelligence (AI) in psychiatric disorders show moderate success, high-lighting the potential of incorporating information from clinical assessments to improve the mod-els. The study focuses on using Large Language Models (LLMs) to manage unstructured medi-cal text, particularly for suicide risk detection in psychiatric care. Objective The study aims to extract information about suicidality status from the admission notes of elec-tronic health records (EHR) using privacy-sensitive, locally hosted LLMs, specifically evaluating the efficacy of Llama-2 models. Main Outcomes and Measures The study compares the performance of several variants of the open source LLM Llama-2 in extracting suicidality status from psychiatric reports against a ground truth defined by human experts, assessing accuracy, sensitivity, specificity, and F1 score across different prompting strategies. Results A German fine-tuned Llama-2 model showed the highest accuracy (87.5%), sensitivity (83%) and specificity (91.8%) in identifying suicidality, with significant improvements in sensitivity and specificity across various prompt designs. Conclusions and Relevance The study demonstrates the capability of LLMs, particularly Llama-2, in accurately extracting the information on suicidality from psychiatric records while preserving data-privacy. This suggests their application in surveillance systems for psychiatric emergencies and improving the clinical management of suicidality by improving systematic quality control and research.
Date: 3 December 2024, 15:00 (Tuesday, 8th week, Michaelmas 2024)
Venue: https://zoom.us/j/96325047461?pwd=zGAc6FLGXwg4fq7noWrlaGu6JV8aO8.1
Speaker: Dr Isabella C Wiest (Else Kröner Fresenius Zentrum for Digital Health, TU Dresden)
Organising department: Department of Psychiatry
Organiser: Dr Andrey Kormilitzin (University of Oxford)
Organiser contact email address: andrey.kormilitzin@psych.ox.ac.uk
Host: Dr Andrey Kormilitzin (University of Oxford)
Part of: Artificial Intelligence for Mental Health Seminar Series
Booking required?: Not required
Booking url: https://web.maillist.ox.ac.uk/ox/info/ai4mch
Booking email: andrey.kormilitzin@psych.ox.ac.uk
Audience: Public
Editor: Andrey Kormilitzin