2024-04-18-barrainkua24a.md

title	software	abstract	layout	series	publisher	issn	id	month	tex_title	firstpage	lastpage	page	order	cycles	bibtex_author	author	date	address	container-title	volume	genre	issued	pdf	extras
Uncertainty Matters: Stable Conclusions under Unstable Assessment of Fairness Results	https://github.com/abarrainkua/UncertaintyMatters	Recent studies highlight the effectiveness of Bayesian methods in assessing algorithm performance, particularly in fairness and bias evaluation. We present Uncertainty Matters, a multi-objective uncertainty-aware algorithmic comparison framework. In fairness focused scenarios, it models sensitive group confusion matrices using Bayesian updates and facilitates joint comparison of performance (e.g., accuracy) and fairness metrics (e.g., true positive rate parity). Our approach works seamlessly with common evaluation methods like K-fold cross-validation, effectively addressing dependencies among the K posterior metric distributions. The integration of correlated information is carried out through a procedure tailored to the classifier’s complexity. Experiments demonstrate that the insights derived from algorithmic comparisons employing the Uncertainty Matters approach are more informative, reliable, and less influenced by particular data partitions. Code for the paper is publicly available at \url{https://github.com/abarrainkua/UncertaintyMatters}.	inproceedings	Proceedings of Machine Learning Research	PMLR	2640-3498	barrainkua24a	0	Uncertainty Matters: Stable Conclusions under Unstable Assessment of Fairness Results	1198	1206	1198-1206	1198	false	Barrainkua, Ainhize and Gordaliza, Paula and Lozano, Jose A. and Quadrianto, Novi	given family Ainhize Barrainkua given family Paula Gordaliza given family Jose A. Lozano given family Novi Quadrianto	2024-04-18		Proceedings of The 27th International Conference on Artificial Intelligence and Statistics	238	inproceedings	date-parts 2024 4 18	https://proceedings.mlr.press/v238/barrainkua24a/barrainkua24a.pdf