Absence of item origin bias on a Brazilian interinstitutional Progress Test examination: A pooled analysis of items approach

Abstract

Background It has been proposed that the school origin of items for cross-institutional Progress Tests (PTs) may introduce a bias in favour of students from the same school, posing a potential threat to the validity and reliability of PT results and cross-institutional comparisons. The aim of this study was to examine whether origin bias is present in a Brazilian cross-institutional PT examination. Methods This study conducted a cross-sectional analysis of seven schools affiliated with the oldest PT consortium in Brazil, utilising a pooled analysis of differences in students’ performance concerning self and non-self items. A proportional meta-analysis of the items’ rate differences and confidence intervals with random effects was performed, providing an odds ratio (OR) for self and non-self items. Differences between the two groups of items were assessed by scrutinising whether the OR and 95% confidence intervals overlapped. Results The findings indicated no discernible differences in psychometric indices based on the school responsible for item creation. Three schools consistently demonstrate superior performance on items authored by their faculty, however, these they also excelled on non-self items. Furthermore, an overlap in the 95% confidence intervals for both self and non-self items was observed across all seven schools. Conclusions In contrast to prior reports, this study revealed the absence of origin bias, suggesting that adoption of best practices in blueprinting, item writing, and editing may have played a role in mitigating such bias.