2024-09-10-giovannotti24a.md

title	booktitle	year	volume	series	month	publisher	pdf	url	abstract	layout	issn	id	tex_title	firstpage	lastpage	page	order	cycles	bibtex_editor	editor	bibtex_author	author	date	address	container-title	genre	issued	extras
Calibrated Large Language Models for Binary Question Answering	Proceedings of the Thirteenth Symposium on Conformal and Probabilistic Prediction with Applications	2024	230	Proceedings of Machine Learning Research	0	PMLR	https://raw.githubusercontent.com/mlresearch/v230/main/assets/giovannotti24a/giovannotti24a.pdf	https://proceedings.mlr.press/v230/giovannotti24a.html	Quantifying the uncertainty of predictions made by large language models (LLMs) in binary text classification tasks remains a challenge. Calibration, in the context of LLMs, refers to the alignment between the model’s predicted probabilities and the actual correctness of its predictions. A well-calibrated model should produce probabilities that accurately reflect the likelihood of its predictions being correct. We propose a novel approach that utilizes the inductive Venn–Abers predictor (IVAP) to calibrate the probabilities associated with the output tokens corresponding to the binary labels. Our experiments on the BoolQ dataset using the Llama 2 model demonstrate that IVAP consistently outperforms the commonly used temperature scaling method for various label token choices, achieving well-calibrated probabilities while maintaining high predictive quality. Our findings contribute to the understanding of calibration techniques for LLMs and provide a practical solution for obtaining reliable uncertainty estimates in binary question answering tasks, enhancing the interpretability and trustworthiness of LLM predictions.	inproceedings	2640-3498	giovannotti24a	Calibrated Large Language Models for Binary Question Answering	218	235	218-235	218	false	Vantini, Simone and Fontana, Matteo and Solari, Aldo and Bostr\"{o}m, Henrik and Carlsson, Lars	given family Simone Vantini given family Matteo Fontana given family Aldo Solari given family Henrik Boström given family Lars Carlsson	Giovannotti, Patrizio and Gammerman, Alexander	given family Patrizio Giovannotti given family Alexander Gammerman	2024-09-10		Proceedings of the Thirteenth Symposium on Conformal and Probabilistic Prediction with Applications	inproceedings	date-parts 2024 9 10