2024-09-10-singh24a.md

title	booktitle	year	volume	series	month	publisher	pdf	url	abstract	layout	issn	id	tex_title	firstpage	lastpage	page	order	cycles	bibtex_editor	editor	bibtex_author	author	date	address	container-title	genre	issued	extras
Distribution-free risk assessment of regression-based machine learning algorithms	Proceedings of the Thirteenth Symposium on Conformal and Probabilistic Prediction with Applications	2024	230	Proceedings of Machine Learning Research	0	PMLR	https://raw.githubusercontent.com/mlresearch/v230/main/assets/singh24a/singh24a.pdf	https://proceedings.mlr.press/v230/singh24a.html	In safety-critical applications, such as medicine and healthcare, decision makers are hesitant to deploy machine learning models unless the expected algorithmic errors are guaranteed to remain within pre-defined tolerances. However, since ML algorithms are statistical in nature, a bounded error cannot be ensured for all possible data inputs. To the contrary, practitioners could be provided with an estimate of the probability the error exceeds the pre-defined tolerance interval. Thus, they will be able to better anticipate high magnitude ML errors and thus manage them more effectively. We refer to this as the risk-assessment problem and propose a novel solution for it. We propose a conformal prediction approach that translates the risk-assessment task into a prediction interval generation problem. The conformal prediction approach results in prediction intervals that are guaranteed to contain the true target variable with a given probability. Using this coverage property, we prove that our risk-assessment approach is conservative i.e., the risk we compute, under weak assumptions, is not lower than the true risk resulting from the ML algorithm. We focus on regression tasks and computationally study, and compare with other related methods, the performance of the proposed method both with and without covariate shift. We find that our method offers superior accuracy while being conservative.	inproceedings	2640-3498	singh24a	Distribution-free risk assessment of regression-based machine learning algorithms	44	64	44-64	44	false	Vantini, Simone and Fontana, Matteo and Solari, Aldo and Bostr\"{o}m, Henrik and Carlsson, Lars	given family Simone Vantini given family Matteo Fontana given family Aldo Solari given family Henrik Boström given family Lars Carlsson	Singh, Sukrita and Sarna, Neeraj and Li, Yuanyuan and Lin, Yang and Orfanoudaki, Agni and Berger, Michael	given family Sukrita Singh given family Neeraj Sarna given family Yuanyuan Li given family Yang Lin given family Agni Orfanoudaki given family Michael Berger	2024-09-10		Proceedings of the Thirteenth Symposium on Conformal and Probabilistic Prediction with Applications	inproceedings	date-parts 2024 9 10