Can you repeat that? Exploring the definition of a successful model replication in health economics

Background: The ISPOR modelling taskforce suggest decision models should be thoroughly reported and transparent. However, the level of transparency and indeed how transparency should be assessed, has yet to be defined. One way, may be to attempt to replicate the model and its outputs. The ability to replicate a decision model could demonstrate adequate reporting transparency.

Objective: To explore published definitions of replication success, across all scientific disciplines, and to consider how such a definition should be tailored for use in health economic models.

Methods: A literature review was conducted to identify published definitions of a ‘successful replication’. Using these as a foundation, several definitions of replication success were constructed to be applicable to replications of economic decision models, with the associated strengths and weaknesses of such definitions discussed.

Results: A substantial body of literature discussing replicability was found, however relatively few studies, ten, explicitly defined a successful replication. These definitions varied from subjective assessments to expecting exactly the same results to be reproduced. Whilst the definitions that have been found may help to construct a definition specific to health economics, no definition was found that completely encompassed the unique requirements for decision models.

Conclusion: Replication is widely discussed in other scientific disciplines, however as of yet there is no consensus on how replicable models should be within health economics or, what constitutes a successful replication. Replication studies can demonstrate how transparently a model is reported, identify potential calculation errors and inform future reporting practices. It may therefore be a useful adjunct to other transparency or quality measures.