Development and preliminary evaluation of the Implementation Research Proposal Appraisal Criteria (ImpResPAC) tool: an international expert modified e-Delphi study

Background: High-quality implementation research is fundamental to increasing the uptake of evidence-based interventions in healthcare and ultimately improving healthcare delivery and outcomes. However, the development of high-quality implementation research proposals and the identification of the ‘best’ implementation research proposals to fund are hampered by the lack of criteria for their appraisal. This study describes the development and evaluation of the Implementation Research Proposal Appraisal Criteria (ImpResPAC), a comprehensive tool developed to help research teams identify strengths and weaknesses of their implementation research proposals and provide grant reviews with clear criteria for appraising the conceptual and methodological quality of implementation research proposals.

Methods: A two-stage sequential mixed-methods design, including content development (stage 1) and a two-round modified e-Delphi (stage 2a & 2b), was employed. Stage 1: Informed by the structure and content of the Implementation Science Research Development (ImpRes) tool and supplementary guide, and a previous study in which five of the ten ImpRes domains were operationalised to develop five ImpResPAC domains, the ImpResPAC development team developed the initial content of ImpResPAC. Purposive and snowball sampling was used to recruit an Expert Advisory Panel (EAP), of individuals that have made a significant contribution to the conceptual and methodological advancement of implementation science. Stage 2a: The EAP provided feedback on ImpResPAC. Based on the feedback, the ImpResPAC development team made extensive revisions to ImpResPAC. Stage 2b: The refined ImpResPAC was shared with the EAP for further feedback and evaluation.

Results: Sixty-eight international experts formed the EAP. ImpResPAC includes 71 domain items, each indicative of high-quality implementation research. The domain items are organised over 10 domains each representing a core element of implementation research. Based on feedback, user instructions were refined, and a glossary of terms used in ImpResPAC was developed.

Conclusion: ImpResPAC is a comprehensive and transparent quantitative tool to appraise the conceptual and methodological quality of implementation research proposals in healthcare. Application of ImpResPAC has several potential and immediate benefits; it will support research teams identify strengths and weaknesses of their implementation research proposals and in turn develop high-quality proposals, enable grant reviewers to identify the ‘best’ implementation research proposals to fund, and provide a comprehensive tool to be used by educators to appraise implementation research proposals submitted as part of implementation science teaching and training initiatives.