Clarifying the Path to User Satisfaction: An Investigation into Clarification Usefulness (EACL 2024)
Clarifying questions are an integral component of modern information retrieval systems, directly impacting user satisfaction and overall system performance. Poorly formulated questions can lead to user frustration and confusion, negatively affecting the system’s performance. This research addresses the urgent need to identify and leverage key features that contribute to the classification of clarifying questions, enhancing user satisfaction. To gain deeper insights into how different features influence user satisfaction, we conduct a comprehensive analysis, considering a broad spectrum of lexical, semantic, and statistical features, such as question length and sentiment polarity. Our empirical results provide three main insights into the qualities of effective query clarification: (1) specific questions are more effective than generic ones; (2) the subjectivity and emotional tone of a question play a role; and (3) shorter and more ambiguous queries benefit significantly from clarification. Based on these insights, we implement feature-integrated user satisfaction prediction using various classifiers, both traditional and neural-based, including random forest, BERT, and large language models. Our experiments show a consistent and significant improvement, particularly in traditional classifiers, with a minimum performance boost of 45%. This study presents invaluable guidelines for refining the formulation of clarifying questions and enhancing both user satisfaction and system performance.
We have put a sample notebook of our analysis on MIMCS: Open this link
@inproceedings{rahmani-etal-2024-clarifying,
title = "Clarifying the Path to User Satisfaction: An Investigation into Clarification Usefulness",
author = "Rahmani, Hossein A. and Wang, Xi and Aliannejadi, Mohammad and Naghiaei, Mohammadmehdi and Yilmaz, Emine",
booktitle = "Findings of the Association for Computational Linguistics: EACL 2024",
year = "2024",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.findings-eacl.84",
pages = "1266--1277",
}
This research is supported by the Engineering and Physical Sciences Research Council [EP/S021566/1], the Alan Turing Institute under the EPSRC grant [EP/N510129/1] and the EPSRC Fellowship titled “Task Based Information Retrieval” [EP/P024289/1]. Any opinions, findings, conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect those of the sponsors.