2024-07-24-ahsan24a.md

title	abstract	year	volume	publisher	series	software	layout	issn	id	month	tex_title	firstpage	lastpage	page	order	cycles	bibtex_author	author	date	address	container-title	genre	issued	pdf	extras
Retrieving Evidence from EHRs with LLMs: Possibilities and Challenges	Unstructured data in Electronic Health Records (EHRs) often contains critical information—complementary to imaging—that could inform radiologists’ diagnoses. But the large volume of notes often associated with patients together with time constraints renders manually identifying relevant evidence practically infeasible. In this work we propose and evaluate a zero-shot strategy for using LLMs as a mechanism to efficiently retrieve and summarize unstructured evidence in patient EHR relevant to a given query. Our method entails tasking an LLM to infer whether a patient has, or is at risk of, a particular condition on the basis of associated notes; if so, we ask the model to summarize the supporting evidence. Under expert evaluation, we find that this LLM-based approach provides outputs consistently preferred to a pre-LLM information retrieval baseline. Manual evaluation is expensive, so we also propose and validate a method using an LLM to evaluate (other) LLM outputs for this task, allowing us to scale up evaluation. Our findings indicate the promise of LLMs as interfaces to EHR, but also highlight the outstanding challenge posed by “hallucinations”. In this setting, however, we show that model confidence in outputs strongly correlates with faithful summaries, offering a practical means to limit confabulations.	2024	248	PMLR	Proceedings of Machine Learning Research	https://github.com/hibaahsan/chil_diagnosis_evidence/	inproceedings	2640-3498	ahsan24a	0	Retrieving Evidence from EHRs with LLMs: Possibilities and Challenges	489	505	489-505	489	false	Ahsan, Hiba and McInerney, Denis Jered and Kim, Jisoo and Potter, Christopher A and Young, Geoffrey and Amir, Silvio and Wallace, Byron C	given family Hiba Ahsan given family Denis Jered McInerney given family Jisoo Kim given family Christopher A Potter given family Geoffrey Young given family Silvio Amir given family Byron C Wallace	2024-07-24		Proceedings of the fifth Conference on Health, Inference, and Learning	inproceedings	date-parts 2024 7 24	https://raw.githubusercontent.com/mlresearch/v248/main/assets/ahsan24a/ahsan24a.pdf