Implementation Science: Evaluation Stage

Original Editor - Stacy Schiurring based on the course by Lucio Naccarella

Top Contributors - Stacy Schiurring, Tarina van der Stockt, Kim Jackson and Jess Bell  

Key Implementation Terms[edit | edit source]

  1. Implementation: The term implementation refers to the use of any strategies to adopt and integrate evidence-based interventions to change practice patterns[1]
  2. Implementation Science: The study of methods to promote the uptake of research and evidence-based practices into routine practice. Other names used include dissemination and implementation research, knowledge translation, knowledge-to-action, and continuous quality improvement in learning health systems [1]
  3. Implementation research: broad term referring to research by not only the provider but the organization and policy levels of healthcare[2]
  4. Implementation practice: rehabilitation leaders implement evidence based research and adapting to different contexts and settings[1]
  5. Implementation strategies: how to strengthen the implementation, adoption and sustainability of evidence based practice[1]
  6. Implementation Facilitation: guide rehabilitation professions regarding identifying and addressing implementation challenges and utilising positive outcomes[1]
  7. Implementation outcome: effects of adopting the new interventions into practice[1]

An Implementation Evaluation Mindset[edit | edit source]

The term evaluation has many meanings and myths associated with it, leading to confusion, fear and resistance to its benefits.

Evaluation involves all of the following:

  • Gathering reliable and valid information in a systematic way from all intervention stakeholders
  • Attributing value to the intervention implementation process and strategies or outcomes of implementation process
  • Informing future rehabilitation intervention decision-making

The backbone of any implementation evaluation is having a clear evaluation purpose, and direct relevant and answerable evaluation questions that are aligned with the evaluation approaches and methods. Evaluations are typically associated with judging the effectiveness of the implementation process but can also inform decisions about the implementation process and outcomes of the implementation process.[3]

Implementation Evaluation and Contextual Challenges[edit | edit source]

“What will affect what you implement?” To refresh your memory, please see this article to review the multiple factors that may affect the successful implementation of evidence-based rehabilitation interventions.

It is easy to become overwhelmed given the increasing quantity of information surrounding these multiple contextual challenges to implementing rehabilitation interventions. It is therefore beneficial to use a more comprehensive approach to evaluating which contextual factors have influence on a particular implementation's outcome. Box 10 provides several key implementation process evaluation questions.[3]

These implementation process evaluation questions are based upon several existing implementation science evaluation frameworks listed below.

Examples of implementation science evaluation frameworks that can identify barriers and facilitators to key implementation outcomes:

There are many other evaluation frameworks that are consistently used evaluate the effectiveness of evidence-informed rehabilitation interventions. This type of framework includes:

Implementation Evaluation and Implementation Strategies[edit | edit source]

“What will help what you implement?” To refresh your memory, please see this article to review the multiple implementation strategies which support the successful implementation of evidence-based rehabilitation interventions.[3]

The five classes of implementation strategies:

  1. Implementation process strategies
  2. Dissemination strategies
  3. Integration strategies
  4. Capacity building strategies
  5. Scale up strategies

It is easy to become overwhelmed given all these implementation strategies. It is therefore beneficial to use a more comprehensive approach to evaluating their influence on a particular implementation strategy's outcome. Box 11 provides several key implementation strategy evaluation questions.[3]

Key implementation strategy evaluation questions

Evaluate Implementation Outcomes[edit | edit source]

It is important to acknowledge that an unresolved issue in the field of implementation science is how to evaluate implementation effectiveness of evidence-based interventions. Distinguishing implementation effectiveness from intervention effectiveness is critical for transporting interventions from laboratory settings to real-world and or community settings. When such efforts fail, it is important to know if the failure occurred because the intervention was ineffective in the new setting (intervention failure), or if a good intervention was implemented incorrectly (implementation failure).[3]

Implementation Outcomes - refer to the effects of deliberate implementation strategies to adopt and embed new interventions, programs or practices into real world rehabilitation settings.[3]

Three clusters of implementation outcomes have been suggested:[11][12]

  1. Implementation outcomes - the effects of implementation strategies undertaken to implement a new intervention such as: acceptability, adoption, appropriateness, uptake, feasibility, fidelity, implementation cost, penetration, and sustainability of the evidence-based rehabilitation interventions
  2. Service system outcomes - the effects of interventions on service outcomes such as: efficiency, safety, effectiveness, equity, patient centredness, timeliness of the evidence-based rehabilitation interventions
  3. Patient outcomes - the effects of intervention on patient outcomes such as: changes in patient satisfaction, function or symptomology as a result of the evidence-based rehabilitation interventions

Measuring implementation outcomes in addition to client or service system outcomes is crucial for distinguishing effective or ineffective programmes that are well or poorly implemented. While all three clusters of implementation outcomes are key to focus on, this article will mainly focus on implementation outcomes.[3] Box 12 provides definitions of implementation outcome dimensions adapted from Proctor et al.[11]

Definitions of implementation outcome dimensions[11]

Given that there are eight implementation outcomes, once again it can be overwhelming as to how and what outcome dimension to select. There are several factors to consider when choosing which implementation outcomes to evaluate:[3]

  • The specific barriers to implementation you have observed
  • The novelty of the evidence-based practice you are trying to implement
  • The setting in which implementation is taking place
  • The resources for and quality of usual training for implementation

The stage of implementation and your unit of analysis can also have an impact. For example, acceptability may be more appropriate to study during early implementation and sustainability may be more appropriately measured later in the implementation process.[3]

Implementation Fidelity[edit | edit source]

Fidelity translates as “faithfulness”; thus, fidelity of intervention means faithful and correct implementation of the key components of a defined intervention. Unless such an evaluation is made, it cannot be determined whether a lack of impact is due to poor implementation or inadequacies inherent in the intervention in the real-world setting. Evidence-based practice also assumes that an intervention is being implemented in full accordance with its published details. This is particularly important given the greater potential for inconsistencies in implementation of an intervention in real world rather than experimental conditions. Evidence-based practice needs a means of evaluating whether the intervention is actually being implemented as the designers intended.[3]

Implementation fidelity can be described in terms of three key elements that need to be measured including:

  1. Adherence to an intervention - whether an intervention is being delivered as it was designed or written as far as content of the intervention; the exposure or dose of an intervention received by participants
  2. Intervention complexity - complex interventions have greater scope for variation in their delivery, therefore are more vulnerable to one or more components not being correctly implemented
  3. Facilitation strategies - the provision of manuals, guidelines, training, monitoring and feedback, capacity building, and incentives:
    • Quality of delivery refers to the manner in which a practitioner or administrator or volunteer delivers an intervention
    • Participant responsiveness measures how far participants respond to, or are engaged by, an intervention

It is easy to become overwhelmed given the large number of elements that may influence implementation fidelity. It is therefore beneficial to use a more comprehensive approach to evaluating the influence on the implementation fidelity outcomes. Box 13 provides several key implementation outcome (fidelity) evaluation questions.[3]

Key implementation outcome (fidelity) evaluation questions

Summary[edit | edit source]

Given the multiple challenges to implementing rehabilitation interventions, it is important to think about and assess both the implementation process and the outcomes of your implementation process. To inform future implementation facilitation efforts, it is important to know not just what worked but how and why the selected implementation strategies performed.

It is important to distinguish implementation effectiveness from intervention effectiveness. Given that there are multiple implementation effectiveness outcomes be selective about implementation outcomes to assess.

Resources[edit | edit source]

Optional Video:

Please view this approximately 20-minute video for a detailed description and comparision of implementation science frameworks.


Recommended Reading:

Additional Implementation Outcome Measurement Tools:

Given the complexity of thinking and measuring implementation outcomes, for more detailed information about instruments to measure implementation outcomes please review:

References[edit | edit source]

  1. 1.0 1.1 1.2 1.3 1.4 1.5 Naccarella, L. Implementing Evidence-Based Rehabilitation Using Implementation Science Course. Plus. 2022
  2. Bauer MS, Damschroder L, Hagedorn H, Smith J, Kilbourne AM. An introduction to implementation science for the non-specialist. BMC psychology. 2015 Dec;3(1):1-2.
  3. 3.00 3.01 3.02 3.03 3.04 3.05 3.06 3.07 3.08 3.09 3.10 Naccarella, L. Implementation Science: Evaluation Stage. Implementation Science Foundational Training Programme. Plus. 2022.  
  4. Keith RE, Crosson JC, O’Malley AS, Cromp D, Taylor EF. Using the Consolidated Framework for Implementation Research (CFIR) to produce actionable findings: a rapid-cycle evaluation approach to improving implementation. Implementation Science. 2017 Dec;12(1):1-2.
  5. Atkins L, Francis J, Islam R, O’Connor D, Patey A, Ivers N, Foy R, Duncan EM, Colquhoun H, Grimshaw JM, Lawton R. A guide to using the Theoretical Domains Framework of behaviour change to investigate implementation problems. Implementation science. 2017 Dec;12(1):1-8.
  6. Hunter SC, Kim B, Mudge A, Hall L, Young A, McRae P, Kitson AL. Experiences of using the i-PARIHS framework: a co-designed case study of four multi-site implementation projects. BMC health services research. 2020 Dec;20(1):1-4.
  7. Roberts NA, Janda M, Stover AM, Alexander KE, Wyld D, Mudge A. The utility of the implementation science framework “Integrated Promoting Action on Research Implementation in Health Services”(i-PARIHS) and the facilitator role for introducing patient-reported outcome measures (PROMs) in a medical oncology outpatient department. Quality of Life Research. 2021 Nov;30(11):3063-71.
  8. Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. American journal of public health. 1999 Sep;89(9):1322-7.
  9. Bondarenko J, Babic C, Burge AT, Holland AE. Home-based pulmonary rehabilitation: an implementation study using the RE-AIM framework. ERJ open research. 2021 Apr 1;7(2).
  10. Allen CG, Barbero C, Shantharam S, Moeti R. Is theory guiding our work? A scoping review on the use of implementation theories, frameworks, and models to bring community health workers into health care settings. Journal of public health management and practice: JPHMP. 2019 Nov;25(6):571.
  11. 11.0 11.1 11.2 11.3 Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, Griffey R, Hensley M. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Administration and policy in mental health and mental health services research. 2011 Mar;38(2):65-76.
  12. Damschroder LJ, Reardon CM, Widerquist MA, Lowery J. The updated Consolidated Framework for Implementation Research based on user feedback. Implementation Science. 2022 Dec;17(1):1-6.
  13. Youtube. Theories and Frameworks in Implementation Science | IRL. Available from: [last accessed 11/05/2022]
  14. Khadjesari Z, Boufkhed S, Vitoratou S, Schatte L, Ziemann A, Daskalopoulou C, Uglik-Marucha E, Sevdalis N, Hull L. Implementation outcome instruments for use in physical healthcare settings: a systematic review. Implementation Science. 2020 Dec;15(1):1-6.