Projects per year
Abstract
Objective:
Model replication is important as it enables researchers to check research integrity, transparency and, potentially, to inform the model conceptualisation process when developing a new or updated model. The aim of this study was to evaluate the replicability of published decision analytic models, and to identify the barriers and facilitators to replication.
Methods:
Replication attempts of five published economic modelling studies were made. The replications were conducted using only publicly available information within the manuscripts and supplementary materials. The replicator attempted to reproduce the key results detailed in the paper, for example the total cost, total outcomes and if applicable, the incremental cost effectiveness ratio reported. Whilst a replication attempt was not explicitly defined as a success or failure, the replicated results were compared in terms of percentage difference to the original results.
Results:
In conducting the replication attempts, common barriers and facilitators emerged. For the majority of the case studies, the replicator needed to make additional assumptions when recreating the model. This was often exacerbated by conflicting information being presented in the text and the tables. Across the case studies, the variation between original and replicated results ranged from -4.54% to 108.00% for costs and -3.81% to 0.40% for outcomes.
Conclusion:
This study demonstrates that whilst models may appear to be comprehensively reported, it is often not enough to facilitate a precise replication. Further work is needed to understand how to improve model transparency and in turn to increase the chances of replication, thus ensuring future usability.
Model replication is important as it enables researchers to check research integrity, transparency and, potentially, to inform the model conceptualisation process when developing a new or updated model. The aim of this study was to evaluate the replicability of published decision analytic models, and to identify the barriers and facilitators to replication.
Methods:
Replication attempts of five published economic modelling studies were made. The replications were conducted using only publicly available information within the manuscripts and supplementary materials. The replicator attempted to reproduce the key results detailed in the paper, for example the total cost, total outcomes and if applicable, the incremental cost effectiveness ratio reported. Whilst a replication attempt was not explicitly defined as a success or failure, the replicated results were compared in terms of percentage difference to the original results.
Results:
In conducting the replication attempts, common barriers and facilitators emerged. For the majority of the case studies, the replicator needed to make additional assumptions when recreating the model. This was often exacerbated by conflicting information being presented in the text and the tables. Across the case studies, the variation between original and replicated results ranged from -4.54% to 108.00% for costs and -3.81% to 0.40% for outcomes.
Conclusion:
This study demonstrates that whilst models may appear to be comprehensively reported, it is often not enough to facilitate a precise replication. Further work is needed to understand how to improve model transparency and in turn to increase the chances of replication, thus ensuring future usability.
Original language | English |
---|---|
Pages (from-to) | 1018-1025 |
Number of pages | 8 |
Journal | Value in Health |
Volume | 22 |
Issue number | 9 |
Early online date | 16 Jul 2019 |
DOIs | |
Publication status | Published - Sept 2019 |
Keywords
- Replication
- economic models
- health economics
Projects
- 1 Finished