Can non-randomised studies of interventions provide unbiased effect estimates? A systematic review of internal replication studies

Hugh Sharma Waddington, Paul Fenton Villar, Jeffrey C. Valentine

Research output: Contribution to journalArticlepeer-review

5 Citations (Scopus)
1 Downloads (Pure)

Abstract

Background: Non-randomized studies of intervention effects (NRS), also called quasi-experiments, provide useful decision support about development impacts. However, the assumptions underpinning them are usually untestable, their verification resting on empirical replication. The internal replication study aims to do this by comparing results from a causal benchmark study, usually a randomized controlled trial (RCT), with those from an NRS conducted at the same time in the sampled population. Objectives: We aimed to determine the credibility and generalizability of findings in internal replication studies in development economics, through a systematic review and meta-analysis. Research Design: We systematically searched for internal replication studies of RCTs conducted on socioeconomic interventions in low- and middle-income countries. We critically appraised the benchmark randomized studies, using an adapted tool. We extracted and statistically synthesized empirical measures of bias. Results: We included 600 estimates of correspondence between NRS and benchmark RCTs. All internal replication studies were found to have at least “some concerns” about bias and some had high risk of bias. We found that study designs with selection on unobservables, in particular regression discontinuity, on average produced absolute standardized bias estimates that were approximately zero, that is, equivalent to the estimates produced by RCTs. But study conduct also mattered. For example, matching using pre-tests and nearest neighbor algorithms corresponded more closely to the benchmarks. Conclusions: The findings from this systematic review confirm that NRS can produce unbiased estimates. Authors of internal replication studies should publish pre-analysis protocols to enhance their credibility.
Original languageEnglish
Pages (from-to)563-593
Number of pages31
JournalEvaluation Review
Volume47
Issue number3
Early online date1 Sep 2022
DOIs
Publication statusPublished - Jun 2023

Keywords

  • design replication
  • internal replication
  • meta-analysis
  • non-randomized study of interventions
  • quasi-experimental design
  • systematic review
  • within-study comparison

Cite this