2008-07-09-laber08a.md

title	abstract	layout	series	publisher	issn	id	month	tex_title	firstpage	lastpage	page	order	cycles	bibtex_editor	editor	bibtex_author	author	date	note	address	container-title	volume	genre	issued	pdf	extras
Small sample inference for generalization error in classification using the CUD bound	Confidence measures for the generalization error are crucial when small training samples are used to construct classifiers. A common approach is to estimate the generalization error by resampling and then assume the re-sampled estimator follows a known distribution to form a confidence set [Kohavi 1995, Martin 1996, Yang 2006]. Alternatively, one might bootstrap the resampled estimator of the generalization error to form a confidence set. Unfortunately, these methods do not reliably provide sets of the desired confidence. The poor performance appears to be due to the lack of smoothness of the generalization error as a function of the learned classifier. This results in a non-normal distribution of the estimated generalization error. We construct a confidence set for the generalization error by use of a smooth upper bound on the deviation between the resampled estimate and generalization error. The confidence set is formed by bootstrapping this upper bound. In cases in which the approximation class for the classifier can be represented as a parametric additive model, we provide a computationally efficient algorithm. This method exhibits superior performance across a series of test and simulated data sets.	inproceedings	Proceedings of Machine Learning Research	PMLR	2640-3498	laber08a	0	Small sample inference for generalization error in classification using the CUD bound	357	365	357-365	357	false	McAllester, David A. and Myllym{"a}ki, Petri	given family David A. McAllester given family Petri Myllymäki	Laber, Eric B. and Murphy, Susan A.	given family Eric B. Laber given family Susan A. Murphy	2008-07-09	Reissued by PMLR on 30 October 2024.		Proceedings of the 24th Conference on Uncertainty in Artificial Intelligence	R6	inproceedings	date-parts 2008 7 9	http://proceedings.mlr.press/r6/laber08a/laber08a.pdf