Overfitting Identification in Machine Learning Models with the Person-Fit Indicator

Overfitting is a phenomenon that can affect the generalization ability of machine learning models, but there is currently no clear criterion for identification. Overfitting models exhibit similar behavior to examinees with item preknowledge. Item preknowledge (IP) describes a situation in which a group of examinees have access to a subset of items from an administered test before the exam. One common method for identifying examinees with IP is based on the person-fit indicators, which are calculated using Item Response Theory (IRT). This study attempts to utilize the person-fit indicator to identify overfitting models. It is discovered that when the models adhere to the fundamental assumptions of IRT, the person-fit indicator exhibits the best recognition performance.