Implementation Science: Evaluation Stage

Original Editor - Stacy Schiurring based on the course by Lucio Naccarella

Top Contributors - Stacy Schiurring, Tarina van der Stockt, Kim Jackson and Jess Bell

Introduction

Evaluation within Implementation Science is essential for understanding whether implementation efforts succeed or fail. Building on the pre-implementation assessment of barriers and the implementation strategies deployed, evaluation systematically examines whether implementation processes and outcomes achieved their intended goals, informing future decisions.

Evaluation involves all of the following:

• Gathering reliable and valid information in a systematic way from all intervention stakeholders
• Attributing value to the intervention implementation process and strategies or outcomes of implementation process
• Informing future rehabilitation intervention decision-making

The backbone of any implementation evaluation is having a clear evaluation purpose, and direct relevant and answerable evaluation questions that are aligned with the evaluation approaches and methods. Evaluations are typically associated with judging the effectiveness of the implementation process but can also inform decisions about the implementation process and outcomes of the implementation process.

Implementation Evaluation and Contextual Challenges

“What will affect what you implement?” To refresh your memory, please see this article to review the multiple factors that may affect the successful implementation of evidence-based rehabilitation interventions.

To systematically evaluate which contextual factors influence your implementation outcomes, use the key implementation process evaluation questions in Box 10.

Box 10 - Implementation Process Evaluation Questions

• To what extent has evidence has informed the rehabilitation intervention to be designed to be adaptable, of quality and responsive to complex consumer needs? (Intervention Characteristics)
• To what extent have local organisational structures, networks and culture supported the implementation of the rehabilitation intervention (Inner Context Setting)
• To what extent have local referral pathways, external policies and incentives supported the implementation of the rehabilitation  intervention? (Outer Context Setting)
• To what extent has the implementation of the rehabilitation intervention been responsive to clients group demographics, knowledge/beliefs, and self-efficacy? (Stakeholders)
• To what extent has the rehabilitation intervention implementation engagement, planning and delivery occurred
• as per service, financial and workforce planning?  (Implementation  Process)

These implementation process evaluation questions are based upon several existing implementation science evaluation frameworks listed below.Examples of implementation science evaluation frameworks that can identify barriers and facilitators to key implementation outcomes:

• Consolidated Framework for Implementation Research (CFIR)
• Theoretical Domains Framework (TDF)
• Integrating Promoting Action on Research Implementation in Health Services (I-PARIHS)
• Exploration, Preparation, Implementation and Sustainment (EPIS)

There are many other evaluation frameworks that are consistently used evaluate the effectiveness of evidence-informed rehabilitation interventions. This type of framework includes:

• The Reach, Effectiveness, Adoption, Implementation, Maintenance (RE-AIM) framework: used to evaluate the success of implementation and impact of translating research to “real-world” conditions
• The Precede-Proceed Model of Health Program Planning & Evaluation

Implementation Evaluation and Implementation Strategies

“What will help what you implement?” To refresh your memory, please see this article to review the multiple implementation strategies which support the successful implementation of evidence-based rehabilitation interventions.

The five classes of implementation strategies:

1. Implementation process strategies
2. Dissemination strategies
3. Integration strategies
4. Capacity building strategies
5. Scale up strategies

Evaluating implementation strategies requires assessing each class systematically. Similarly, each class of implementation strategy requires specific evaluation questions. Box 11 provides key questions to guide this evaluation.

Box 11 - Implementation Strategies Evaluation Questions:

1. To what extent has stakeholder engagement & assessing practice setting context supported the successful implementation of the rehabilitation intervention (Implementation process strategies)
2. To what extent has advertising I messaging targeted to specific participants supported the successful
3. implementation of the rehabilitation intervention (Dissemination strategies)
4. To what extent has establishing reminder systems, providing supervision or coaching, reviewing existing staff roles, organisation processes and procedures supported the successful implementation of the rehabilitation intervention (Integration strategies)
5. To what extent has providing training to build capacity, provide technical assistance, tools and facilitation to support implementation processes, establishing opportunities for peer networking supported the successful implementation of the rehabilitation intervention (Capacity building strategies)
6. To what extent has providing training to build intervention specific capacity, or providing technical assistance, tools and facilitation, reviewing organisation infrastructure to support implementation of interventions supported the successful implementation of the rehabilitation intervention (Scale up strategies)

Evaluate Implementation Outcomes

It is important to acknowledge that an unresolved issue in the field of implementation science is how to evaluate implementation effectiveness of evidence-based interventions. Distinguishing implementation effectiveness from intervention effectiveness is critical for transporting interventions from laboratory settings to real-world and or community settings. When such efforts fail, it is important to know if the failure occurred because the intervention was ineffective in the new setting (intervention failure), or if a good intervention was implemented incorrectly (implementation failure).

Implementation Outcomes - refer to the effects of deliberate implementation strategies to adopt and embed new interventions, programs or practices into real world rehabilitation settings.

Three clusters of implementation outcomes have been suggested:

1. Implementation outcomes - the effects of implementation strategies undertaken to implement a new intervention such as: acceptability, adoption, appropriateness, uptake, feasibility, fidelity, implementation cost, penetration, and sustainability of the evidence-based rehabilitation interventions
2. Service system outcomes - the effects of interventions on service outcomes such as: efficiency, safety, effectiveness, equity, patient centredness, timeliness of the evidence-based rehabilitation interventions
3. Patient outcomes - the effects of intervention on patient outcomes such as: changes in patient satisfaction, function or symptomology as a result of the evidence-based rehabilitation interventions

Measuring implementation outcomes in addition to client or service system outcomes is crucial for distinguishing effective or ineffective programmes that are well or poorly implemented. While all three clusters of implementation outcomes are key to focus on, this article will mainly focus on implementation outcomes. Box 12 provides definitions of implementation outcome dimensions adapted from Proctor et al.

Box 12: Definitions of Implementation Outcome Dimensions

1. Acceptability: refers to the extent to which implementation stakeholders perceive a treatment, service, practice, or innovation to be agreeable, palatable, or satisfactory.
2. Adoption: refers to the intention, initial decision, or action to try or employ an innovation or evidence-based practice. Adoption may also be referred to as "uptake."
3. Appropriateness: refers to the perceived fit, relevance, or compatibility of the innovation or evidence-based practice for a given practice setting, provider, or consumer; and/or perceived fit of the innovation or evidence-based practice to address a particular issue or problem.
4. Cost: refers to the financial impact of an implementation effort. May include costs of treatment delivery, cost of the implementation strategy, and cost of using the service setting.
5. Feasibility: refers to the extent to which a new innovation or practice can be successfully used or carried out within a given agency or setting.
6. Fidelity: refers to the extent to degree to which an intervention or implementation strategy was delivered as prescribed in the original protocol or as intended by program developers. May include multiple dimensions such as content, process, exposure, and dosage.
7. Penetration: refers to the extent to which an innovation or practice is integrated within a service setting and its subsystems.
8. Sustainability: refers to the extent to which a recently implemented practice is maintained and/or institutionalised within a service setting's ongoing, stable operations.

Given that there are eight implementation outcomes, once again it can be overwhelming as to how and what outcome dimension to select. There are several factors to consider when choosing which implementation outcomes to evaluate:

• The specific barriers to implementation you have observed
• The novelty of the evidence-based practice you are trying to implement
• The setting in which implementation is taking place
• The resources for and quality of usual training for implementation

The stage of implementation and your unit of analysis can also have an impact. For example, acceptability may be more appropriate to study during early implementation and sustainability may be more appropriately measured later in the implementation process.

Implementation Fidelity

Fidelity translates as “faithfulness”; thus, fidelity of intervention means faithful and correct implementation of the key components of a defined intervention. Unless such an evaluation is made, it cannot be determined whether a lack of impact is due to poor implementation or inadequacies inherent in the intervention in the real-world setting. Evidence-based practice also assumes that an intervention is being implemented in full accordance with its published details. This is particularly important given the greater potential for inconsistencies in implementation of an intervention in real world rather than experimental conditions. Evidence-based practice needs a means of evaluating whether the intervention is actually being implemented as the designers intended.

Implementation fidelity can be described in terms of three key elements that need to be measured including:

1. Adherence to an intervention - whether an intervention is being delivered as it was designed or written as far as content of the intervention; the exposure or dose of an intervention received by participants
2. Intervention complexity - complex interventions have greater scope for variation in their delivery, therefore are more vulnerable to one or more components not being correctly implemented
3. Facilitation strategies - the provision of manuals, guidelines, training, monitoring and feedback, capacity building, and incentives:
   • Quality of delivery refers to the manner in which a practitioner or administrator or volunteer delivers an intervention
   • Participant responsiveness measures how far participants respond to, or are engaged by, an intervention

Assessing implementation fidelity requires examining multiple elements. Box 13 provides key questions to guide this evaluation.

Box 13 - Implementation Outcome (Fidelity Questions)

• To what extent did the patients who received the rehabilitation intervention respond to intervention as intended? (Patient responsiveness)
• To what extent was the rehabilitation intervention were the healthcare professionals who delivered the intervention engaged as intended ? (Staff responsiveness)
• To what extent did the rehabilitation intervention supporting strategies, such as, standardised written procedures, training programs and guidelines optimise delivery to be as uniform as possible and as intended? (Facilitation strategies)
• To what extent was the rehabilitation intervention was delivered in a way appropriate to achieving what was intended. (Quality of Delivery)
• To what extent was the rehabilitation intervention complexity sufficiently and clearly described. (Intervention complexity)
• To what extent did the surrounding social systems, structures and cultures of organization and concurrent events
• influence the intended implementation of the rehabilitation intervention? (Context)

Summary

Successful implementation requires evaluating both process and outcomes. Distinguish between implementation effectiveness (was it implemented correctly?) and intervention effectiveness (did the intervention work?). To inform future efforts, assess not just what worked, but how and why. Be strategic in selecting which implementation outcomes to measure based on your barriers, setting, and implementation stage.

Resources

Optional Video:

Please view this approximately 20-minute video for a detailed description and comparision of implementation science frameworks.

Recommended Reading:

• Allen CG, Barbero C, Shantharam S, Moeti R. Is theory guiding our work? A scoping review on the use of implementation theories, frameworks, and models to bring community health workers into health care settings. Journal of public health management and practice: JPHMP. 2019 Nov;25(6):571.
• Keith RE, Crosson JC, O’Malley AS, Cromp D, Taylor EF. Using the Consolidated Framework for Implementation Research (CFIR) to produce actionable findings: a rapid-cycle evaluation approach to improving implementation. Implementation Science. 2017 Dec;12(1):1-2.
• Khadjesari Z, Boufkhed S, Vitoratou S, Schatte L, Ziemann A, Daskalopoulou C, Uglik-Marucha E, Sevdalis N, Hull L. Implementation outcome instruments for use in physical healthcare settings: a systematic review. Implementation Science. 2020 Dec;15(1):1-6.
• Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, Griffey R, Hensley M. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Administration and policy in mental health and mental health services research. 2011 Mar;38(2):65-76.

Additional Implementation Outcome Measurement Tools:

Given the complexity of thinking and measuring implementation outcomes, for more detailed information about instruments to measure implementation outcomes please review:

• Washington University in St. Louis, Institute of Clinical and Translational Sciences: Implementation Outcomes Toolkit
• Implementation Outcome Repository
• Center for Technology and Behavioral Health: Measures for Implementation Research