IDC: Quantitative Evaluation Benchmark of Interpretation Methods for Deep Text Classification Models

Date
2021-10-29
Authors
Khaleel, Mohammed I.
Qi, Lei
Tavanapong, Wallapak
Wong, Johnny S
Sukul, Adisak
Peterson, David
Journal Title
Journal ISSN
Volume Title
Publisher
The Author(s)
Altmetrics
Authors
Research Projects
Organizational Units
Political Science
Organizational Unit
Computer Science
Organizational Unit
Journal Issue
Series
Abstract
Recent advances in deep neural networks have achieved outstanding success in natural language processing. Due to the success and the black-box nature of the deep models, interpretation methods that provide insight into the decision-making process of the models have received an influx of research attention. However, there is no quantitative evaluation comparing interpretation methods for text classification other than observing classification accuracy or prediction confidence when important word grams are removed. This is due to the lack of interpretation ground truth. Manual labeling of a large interpretation ground truth is time-consuming. We propose IDC, a new benchmark for quantitative evaluation of I nterpretation methods for D eep text C lassification models. IDC consists of three methods that take existing text classification ground truth and generate three corresponding pseudo-interpretation ground truth datasets. We propose to use interpretation recall, interpretation precision, and Cohen’s kappa inter-agreement as performance metrics. We used the pseudo ground truth datasets and the metrics to evaluate six interpretation methods.
Description
Keywords
achine Learning Interpretation, Natural Language Processing, Pseudo Interpretation Ground Truth
Citation
Collections