How to measure the intervention process? An assessment of qualitative and quantitative approaches to data collection in the process evaluation of organizational interventions

Johan S. Abildgaard, Per Ø. Saksvik, Karina Nielsen

Research output: Contribution to journalArticlepeer-review

55 Citations (Scopus)
14 Downloads (Pure)

Abstract

The evaluation of organizational interventions targeting employee health and wellbeing has been found to be a challenging task (Murta et al., 2007). The use of process evaluation, defined as the evaluation of “individual, collective or management perceptions and actions in implementing any intervention and their influence on the overall result of the intervention.” Nytrø et al. (2000) has served to increase focus on the evaluation of the specific intervention processes and not only the outcomes. Although several evaluation frameworks (Nielsen and Abildgaard, 2013; Nielsen and Randall, 2013) have been suggested it has proven to be methodologically challenging to evaluate the processes of implementation of organizational interventions (OIs; Nielsen and Randall, 2013). Two distinct approaches to process evaluation data collection are commonly used. One is a quantitative approach where either standardized or intervention-specific questionnaire items are included in a follow-up questionnaire, and are later integrated into statistical models of implementation and effect (e.g., Nielsen et al., 2007; Nielsen and Randall, 2009, 2012). The other is the collection of qualitative data; often specifically as a supplement to quantitative data, using semi-structured interviews with employees and managers, (Dahl-Jørgensen and Saksvik, 2005; Nielsen et al., 2006), observations of intervention activities (Brannan and Oultram, 2012), or long-term field observations (Czarniawska-Joerges, 2007). Qualitative process evaluation has been used extensively to understand the context of interventions outcomes (e.g., Mikkelsen and Saksvik, 1998; Saksvik et al., 2002; Nielsen et al., 2006; Aust et al., 2010). Each data source has its methodological strengths and weaknesses and the concurrent mixed methods use of both quantitative and qualitative approaches has been proposed as a potential middle ground (Dahl-Jørgensen and Saksvik, 2005; Nielsen and Randall, 2013). Mixed methods is here defined “as a method [which] focuses on collecting, analyzing and mixing both quantitative and qualitative data in a single or series of studies. Its central premise it that the use of [both] approaches in combination provides a better understanding of research problems than either approach alone” (Creswell and Plano Clark, 2011, p. 5). Although much is written about evaluation research in general (Lipsey and Cordray, 2000; Rossi et al., 2004; Pawson, 2013), and mixed methods evaluation in general (Rallis and Rossman, 2003; Nastasi et al., 2007) the particularities and methodological considerations of using qualitative and quantitative data in mixed methods based process evaluation have been sparse (Nastasi et al., 2007), particularly concerning the specifics of evaluating OIs (Nielsen and Abildgaard, 2013). Using a case of an OI in the Danish postal service where questionnaires and semi-structured interviews were used for process evaluation data collection, we compare the epistemological properties of both methods and assess the benefits of different ways to collect process information.

The aim of the present study is to examine the type of knowledge about the intervention process that may be produced by quantitative and qualitative data and discuss how these sources best can be applied in mixed methods designs. It is hence not a study of different forms of mixed methods designs (for such literature see Nastasi et al., 2007; Teddlie and Tashakkori, 2009; Creswell and Plano Clark, 2011) but instead an assessment of the properties and potential roles of specific data sources in mixed methods OI evaluation. We employ a sequential mixed methods analysis to identify a set of factors in the quantitative data that function as an analytical framework with which we comparatively analyze the qualitative data. This approach will help us accentuate what knowledge about the intervention each data collection methods may provide, and allows us to discuss differences and similarities.
Original languageEnglish
Article number1380
JournalFrontiers in Psychology
Volume7
DOIs
Publication statusPublished - 22 Sep 2016

Cite this