Grant Peer Review: Improving Inter-Rater Reliability with Training

Simple item page
dc.contributor.authorSattler, David N.
dc.contributor.authorMcKnight, Patrick E.
dc.contributor.authorNaney, Linda
dc.contributor.authorMathis, Randy
dc.date.accessioned2016-08-09T00:45:41Z
dc.date.available2016-08-09T00:45:41Z
dc.date.issued2015-06-15
dc.description.abstractThis study developed and evaluated a brief training program for grant reviewers that aimed to increase inter-rater reliability, rating scale knowledge, and effort to read the grant review criteria. Enhancing reviewer training may improve the reliability and accuracy of research grant proposal scoring and funding recommendations. Seventy-five Public Health professors from U.S. research universities watched the training video we produced and assigned scores to the National Institutes of Health scoring criteria proposal summary descriptions. For both novice and experienced reviewers, the training video increased scoring accuracy (the percentage of scores that reflect the true rating scale values), inter-rater reliability, and the amount of time reading the review criteria compared to the no video condition. The increase in reliability for experienced reviewers is notable because it is commonly assumed that reviewers—especially those with experience—have good understanding of the grant review rating scale. The findings suggest that both experienced and novice reviewers who had not received the type of training developed in this study may not have appropriate understanding of the definitions and meaning for each value of the rating scale and that experienced reviewers may overestimate their knowledge of the rating scale. The results underscore the benefits of and need for specialized peer reviewer training.
dc.description.sponsorshipThis document was prepared under contract number DE-AC05-06OR23100 between the U.S. Department of Energy and ORAU. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Publication of this article was funded in part by the George Mason University Libraries Open Access Publishing Fund.
dc.identifier.citationSattler DN, McKnight PE, Naney L, Mathis R (2015) Grant Peer Review: Improving Inter-Rater Reliability with Training. PLoS ONE 10(6): e0130450. doi:10.1371/journal.pone.0130450
dc.identifier.doihttp://dx.doi.org/10.1371/journal.pone.0130450
dc.identifier.urihttps://hdl.handle.net/1920/10340
dc.language.isoen_US
dc.publisherPublic Library of Science
dc.subjectResearch validity
dc.subjectResearch grants
dc.subjectPeer review
dc.subjectPublic and occupational health
dc.subjectDemography
dc.subjectQuestionnaires
dc.subjectSense of agency
dc.subjectCognition
dc.titleGrant Peer Review: Improving Inter-Rater Reliability with Training
dc.typeArticle

Files

Original bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
2015-06-15-Sattler-Article.PDF
Size:
431.72 KB
Format:
Adobe Portable Document Format
Description:
Main article
Download
License bundle
Now showing 1 - 1 of 1
No Thumbnail Available
Name:
license.txt
Size:
1.71 KB
Format:
Item-specific license agreed upon to submission
Description:
Download

Collections

Recipients of OA Publishing Fund