2026-10-11T01:16:33Z https://api.figshare.com/v2/oai

oai:figshare.com:article/33974701 2026-09-23T17:21:47Z category_7 category_19 category_21 category_64 category_106 category_132 portal_5 item_type_3 month_year_09_2026

<p>Summary statistics by funder.</p> Jan-Ole Hesselberg (21453807) Pål Ulleberg (522303) Øystein Sørensen (4767861) Knut Inge Fostervold (11509353) Sigrid Hegna Ingvaldsen (15359171) Ida Svege (11509356) Medicine Pharmacology Biotechnology Cancer Science Policy Infectious Diseases general panel membership average iccs ranged human judgment rather structured review formats stable reviewer tendencies reviewer age composition moderate across funders substantially improve reliability systematic reviewer characteristics including reviewer similarity depth review procedures matter expertise rather examines factors associated slightly lower disagreement reviewer similarity review depth specific rather review roles multiple factors four funders reliability persist rater reliability lower disagreement application characteristics yet concerns variability arose temporal trends specific ambiguity reviewers selected reviewers disagree residual level research area registered report rater agreement program features primarily due leveraging data level ambiguity inherent variability greater responsibility grant evaluations expert reviews clearer criteria assessments conducted also associated absolute magnitude 1 percent <div><p>Grant peer review processes are pivotal in allocating substantial research funding, yet concerns about their reliability persist, primarily due to low inter-rater agreement. This study is a registered report that examines factors associated with agreement among peer reviewers in grant evaluations, leveraging data from 134,991 reviews across four Norwegian research funders. Using a cross-classified linear regression model, we explored the relationship between inter-rater agreement and multiple factors, including reviewer similarity, experience, expertise, research area, application characteristics, review depth, and temporal trends. Our analyses revealed several statistically significant associations, but most effects were small in absolute magnitude. Reviewer similarity in gender was associated with lower disagreement. The use of in-depth review procedures, referring to review roles with greater responsibility for evaluating the application, was also associated with slightly lower disagreement. In contrast, expert reviews, defined as assessments conducted by reviewers selected for subject-matter expertise rather than general panel membership, and certain research areas showed modestly higher disagreement. Application amount, reviewer age composition, and temporal trends exhibited limited influence. Only a small share of disagreement stemmed from stable reviewer tendencies (3.1 percent), whereas most variability arose at the application level (24.1 percent) and the residual level (55.2 percent), indicating that disagreement is largely application-specific rather than reviewer-driven. Inter-rater reliability was moderate across funders: average ICCs ranged from 0.50 to 0.65. Our findings only partly support previous research and indicate that disagreement is driven primarily by application-specific ambiguity and inherent variability in human judgment rather than systematic reviewer characteristics or program features. Consequently, interventions targeting reviewer behaviour alone are unlikely to substantially improve reliability; gains are more likely to come from clearer criteria, more structured review formats, and reducing application-level ambiguity, though such measures are expected to yield only incremental improvements.</p></div> 2026-09-23T17:21:34Z Dataset Dataset 10.1371/journal.pone.0356108.t003 https://figshare.com/articles/dataset/_p_Summary_statistics_by_funder_p_/33974701 CC BY 4.0