BenchMIRT applies multidimensional item response theory (MIRT) to LLM evaluation, allowing researchers to decompose benchmark scores into specific underlying capabilities. By analyzing model performance at the individual prompt level, the method identifies which skills—such as reasoning or safety—are driving accuracy for specific tasks. This approach reveals how aggregate scores can obscure nuanced model behaviors and enables more targeted, interpretable evaluation designs.
Key points
Standard aggregate benchmark scores often conflate multiple underlying model capabilities within a single metric.
Multidimensional Item Response Theory (MIRT) can be adapted to LLMs to statistically separate latent skills like reasoning and safety.
BenchMIRT enables question-level analysis, identifying which prompts are most discriminative for specific capabilities.
The utility of MIRT-based auditing depends on the diversity and coverage of the initial benchmark set.
Fine-grained evaluation provides better diagnostic transparency but introduces risks regarding the potential for gaming or pruning evaluations.