Multi-rater assessment in systematic reviews: A methodological innovation in forest higher education literature

Abstract

Systematic literature reviews (SLRs) are essential for synthesizing evidence in forest higher education, yet the reliability of article selection often hinges on subjective expert judgment. As forest education evolves to meet global challenges, such as climate change, digitalization, and market dynamics, educators must navigate an overwhelming volume of literature to identify high-quality science that fosters critical thinking and holistic understanding. This study introduces the Many Facet Rasch Model (MFRM) as a methodological innovation for evaluating multi-rater assessments of the forest higher education literature, offering a transparent and replicable framework for evidence synthesis. Five experts, who served as raters, assessed ten articles using six calibrated criteria (originality, comprehensiveness of literature review, methodology, scientific value of findings, related issues with forest higher education, and quality of analysis). The results demonstrate that MFRM can identify psychometrically sound evaluations, rank article quality, and diagnose criteria, particularly in comprehensiveness of the literature review and difficulty, most notably in literature review comprehensiveness and methodological rigor. This research provides practical guidance for forest higher education practitioners seeking to select pedagogically valuable resources. By enhancing transparency and reproducibility in literature evaluation, MFRM strengthens forest higher education’s capacity to train future foresters with precision, integrity, and relevance.