Implementation Science: Evaluation Stage

Original Editor - Stacy Schiurring based on the course by Lucio Naccarella

Top Contributors - Stacy Schiurring, Tarina van der Stockt, Kim Jackson and Jess Bell  

An Implementation Evaluation Mindset

The term evaluation has many meanings and myths associated with it, leading to confusion, fear and resistance to its benefits.

Evaluation involves all of the following:

• gathering reliable and valid information in a systematic way from all intervention stakeholders
• attributing value to the intervention implementation process and strategies or outcomes of implementation process
• informing future rehabilitation intervention decision-making

The backbone of any implementation evaluation is having a clear evaluation purpose, and direct relevant and answerable evaluation questions that are aligned with the evaluation approaches and methods. Evaluations are typically associated with judging the effectiveness of the implementation process but can also inform decisions about the implementation process and outcomes of the implementation process.

Implementation Evaluation Purpose and Question

“What will affect what you implement?” To refresh your memory, please see this article to review the multiple factors that may affect the successful implementation of evidence-based rehabilitation interventions.

It is easy to become overwhelmed given the increasing quantity of information surrounding these multiple contextual challenges to implementing rehabilitation interventions. It is therefore beneficial to use a more comprehensive approach to evaluating which contextual factors have influence on implementation has worked.

Box 10 provides several key implementation process evaluation questions:

ADD BOX 10

These implementation process evaluation questions are based upon several existing Implementation Science evaluation frameworks, namely - the CFIR, iPARIHS, TDF and EPIS that can identify barriers and facilitators to key implementation outcomes.

For more information on these implementation evaluation frameworks please review the links below:

• CFIR – which stands for the Consolidated Framework for Implementation Research
• TDF – which stands for the Theoretical Domains Framework
• i-PARIHS- which stands for - Integrating Promoting Action on Research Implementation in Health Services
• EPIS which stands for Exploration, Preparation, Implementation and Sustainment

There are many other evaluation frameworks that are consistently used evaluate what makes evidence-informed rehabilitation interventions work or not. Please see the following examples with rehabilitation related articles:

• The RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance) framework is used to evaluate the success of implementation and impact of translating research to “real-world” conditions
• The Precede-Proceed Model of Health Program Planning & Evaluation
• Theoretical Domains Framework

Sub Heading 3

In Module 2 on “What will help what you implement?”  you were introduced to the multiple implementation strategies to support the successful implementation of evidence-based rehabilitation interventions.

We informed you about Five classes of implementation strategies: Implementation process strategies; Dissemination strategies; Integration strategies’ Capacity building strategies; and Scale up strategies. Given all these implementation strategies it is easy to become overwhelmed so you will benefit by using a more comprehensive approach to evaluating How and Why the Implementation Strategies have worked or not. Box 11 provides several key implementation strategy evaluation questions:

ADD BOX 11

more

Welcome to Section 6 that focuses on how to think about and evaluate the outcomes or effects of your implementation strategies. 

It is important to acknowledge that an unresolved issue in the field of implementation science is how to evaluate implementation effectiveness of evidence-based interventions. Distinguishing implementation effectiveness from intervention effectiveness is critical for transporting interventions from laboratory settings to real-world / community settings. When such efforts fail, as they often do, it is important to know if the failure occurred because the intervention was ineffective in the new setting (intervention failure), or if a good intervention was implemented incorrectly (implementation failure).

Thinking about and measuring implementation outcomes will advance your understanding of implementation processes.  In the Implementation Science Short Course you will recall that we introduced you and defined the term Implementation Outcomes.  Let’s recap:

Implementation Outcomes- refers to the EFFECTS of deliberate implementation strategies to adopt and embed new interventions, programs or practices into real world rehabilitation settings. Three cluster of implementation outcomes have been suggested (Proctor et al 2011):

• Implementation outcomes - the effects of implementation strategies undertaken to implement a new intervention such as: acceptability, adoption, appropriateness, uptake, feasibility, fidelity, implementation cost, penetration, and sustainability of the evidence-based rehabilitation interventions)
• Service system outcomes - the effects of interventions on service outcomes such as: efficiency, safety, effectiveness, equity, patient centredness, timeliness of the evidence-based rehabilitation interventions
• Patient Outcomes -the effects of intervention on patient outcomes such as: changes in patient satisfaction function or symptomology as a result of the evidence-based rehabilitation interventions

Measuring implementation outcomes in addition to client or service system outcomes is crucial for distinguishing effective or ineffective programs that are well or poorly implemented.   While all three clusters of implementation outcomes are key to focus on – we will mainly focus on Implementation outcomes ie: why does it or doesn’t it work here.  Given that there are many dimensions to consider, intervention acceptability, adoption, appropriateness, uptake, feasibility, fidelity, implementation cost, penetration, and sustainability – we firstly need to be sure you understand and can define these implementation outcomes dimensions.  Box 12 provides definitions of implementation outcome dimensions adapted from Proctor et al. (2011)

ADD BOX 12

Given that there are 8 implementation outcomes, once again it can be overwhelming as to how and what outcome dimension to select.  There are several factors to consider when choosing which implementation outcome(s) to evaluate including:

• the specific barriers to implementation you have observed
• the novelty of the evidence-based practice you are trying to implement
• the setting in which implementation is taking place
• the resources for and quality of usual training for implementation

The stage of implementation and your unit of analysis can also influence. For example, acceptability may be more appropriate to study during early implementation and sustainability may be more appropriately measured later in the implementation process.

Given the complexity of thinking and measuring implementation outcomes, for more detailed information about instruments to measure implementation outcomes please review:

• https://implementationoutcomerepository.org/guide-to-using-the-implementation-outcome-repository-

• Gerke et al (2017). Implementation Outcomes. [Internet]. St. Louis, MO: Washington University; 2017 July. Eight toolkits related to Dissemination and Implementation. Available from https://sites.wustl.edu/wudandi
• https://www.c4tbh.org/resources/measures-for-implementation-studies/

Implementation Fidelity

To illustrate measuring implementation outcomes, let’s focus on Implementation Fidelity, as it is only by making an appropriate evaluation of the fidelity with which an intervention has been implemented that a viable assessment can be made of its true effect. Fidelity translates as “faithfulness”; thus, fidelity of intervention means faithful and correct implementation of the key components of a defined intervention. Unless such an evaluation is made, it cannot be determined whether a lack of impact is due to poor implementation or inadequacies inherent in the intervention in the real-world setting. Evidence-based practice also assumes that an intervention is being implemented in full accordance with its published details. This is particularly important given the greater potential for inconsistencies in implementation of an intervention in real world rather than experimental conditions. Evidence-based practice needs a means of evaluating whether the intervention is actually being implemented as the designers intended. (a Key Process Evaluation Question).

Implementation fidelity can be described in terms of three key elements that need to be measured including:

1. Adherence to an intervention - this refers to whether an intervention is being delivered as it was designed or written as far as content of the intervention; the Exposure or dose- of an intervention received by participants
2. Intervention complexity - this refers to complex interventions have greater scope for variation in their delivery, and so are more vulnerable to one or more components not being implemented as they should; and
3. Facilitation strategies - this refers to the provision of manuals, guidelines, training, monitoring and feedback, capacity building, and incentives; Quality of delivery – refers to the manner in which a practitioner or administrator or volunteer delivers an intervention; Participant responsiveness - measures how far participants respond to, or are engaged by, an intervention. Given all these elements that may influence implementation fidelity it is easy to become overwhelmed so you will benefit by using a more comprehensive approach to evaluating How and why the Implementation Outcomes (Fidelity) has been achieved not. Box 13 provides several key implementation outcome (fidelity) evaluation questions:

ADD BOX 13

Summary

Key takeaways from Section 5, is that given the multiple challenges to implementing rehabilitation interventions it is important to think about and assess both the implementation process and the outcomes of your implementation process.  You need to know not just WHAT worked but HOW and WHY your Implementation Strategies have worked or not- to inform your future implementation facilitation efforts.

Key takeaways from Section 6, is that firstly, it is important to distinguishing implementation effectiveness from intervention effectiveness. Secondly, as there are multiple implementation effectiveness outcomes you need to not get overwhelmed but think carefully about which implementation outcomes to assess. We focused on implementation fidelity to illustrate a comprehensive approach to thinking about and evaluating your implementation outcomes.

Resources

• bulleted list
• x

or

1. numbered list
2. x