Appraise the evidence

Original Editor - Your name will be added here if you created the original content for this page.

Top Contributors - Admin, Rachael Lowe, 127.0.0.1, Angeliki Chorti, Tony Lowe, Evan Thomas, George Prudden, WikiSysop, Alex Benham and Grace Barla  

The Appraisal

After we have searched for the evidence, we need to decide if it is both valid and important before deciding if we can apply the evidence to our individual patients. The order in which we consider validity and importance depends on individual preference.

The third step of evidence based practice, critical appraisal is the systematic evaluation of clinical research papers in order to establish:

1. Does this study address a clearly focused question?
2. Did the study use valid methods to address this question?
3. Are the valid results of this study important?
4. Are these valid, important results applicable to my patient or population?

If the answer to any of these questions is “no”, you can save yourself the trouble of reading the rest of it.

Cleland, Noteboom, Whitman and Allison (2008) addressed these questions in a two part series on selected aspects of evidence based practice relating to questions of treatment.  They discuss the following thress points which are summarised below: 

1. Are the results of the study valid?
2. What were the results?
3. Will the results help me in caring for my patients?

Are the results of the study valid?

1. Hierarchy of evidence
2. Internal and external validity
3. Randomisation and baseline homogenity of groups - was the assignment of patients to treatments randomised? Were the groups similar at the start of the trial?
4. Concealment of allocation to groups - were patients analysed in the groups to which they were randomised?
5. Blinding - were patients, health workers, and study personnel "blind" to treatment?
6. Completeness of follow up (Intention to treat principle) - was follow up complete? Were all patients who entered the trial properly accounted for and attributed at its conclusion?
7. Equivalent experience of groups apart from treatemnt of interest - aside from the experimental intervention, were the groups treated equally?
   

Hierarchy of evidence

When evaluating evidence for effectiveness of an intervention, clinicians often find it helpful to use a system to determine the level of evidence and/or grade of recommendation for a particular study.

Internal and external validity

In addition to identifying the level of evidence on the hierarchy, therapists must also consider critically appraising the study’s overall quality and the study’s internal and external validity.  Internal validity relates to elements of research design intended to exert control over extraneous variables that could potentially impact the outcomes of the study, including interactions between patient assignment, competing interventions, history, maturation, and instrumentation.  External validity refers to the generalizability of the study’s results to actual clinical practice.

Randomisation and baseline homogenity of groups 

Randomization should theoretically ensure that each group of subjects is similar at baseline so that no extraneous variables (such as known and unknown prognostic factors) compete with the intervention to explain observed outcomes. Extraneous variables that could potentially affect outcomes in studies of treatment effectiveness include patient, age, race, gender, symptom duration, condition severity, comorbidities, intellectual status, motivation, and treatment expectations. Although randomization should ideally produce observed homogenous groups at baseline, there is always a chance, particularly with small samples, that groups may be dissimilar in important known and unknown prognostic factors, which may affect group homogeneity. For this reason a reader performing a critical appraisal must independently judge the extent to which groups are similar in key prognostic factors.

Concealment of allocation to groups

Even when randomization procedures are followed, bias from investigators in- fluencing subject enrollment and group composition can threaten validity if allocation to groups is not concealed from those enrolling subjects in the study

Blinding 

In an attempt to minimize the effect of rater or subject bias, studies use various blinding schemes. There are 4 categories of study participants who should ideally be blinded to group assignment: (1) patients, (2) treating clinicians, (3) data collectors, and (4) data analysts. Although it is usually feasible to blind those from all 4 categories in a pharmaceutical study, this is usually not possible in studies of physical therapy interventions.

Completeness of follow up (Intention to treat principle)

The authors should report the reasons for any patient dropouts from the study and identify any patients who were lost to follow-up.  It is important for the clinician to know if the patient withdrew from the study due to full resolution of symptoms, for reasons unrelated to the study, or because the person experienced a worsening in status that was directly or potentially related to the examination or treatment program provided by the study protocol. When subjects are lost to follow-up, it may still be possible to include data from all subjects in the final data set using an intention-to-treat (ITT) approach, which has been used in a variety of recently published studies.

Equivalent experience of groups apart from treatemnt of interest

It is possible to introduce bias into a study of treatment if there are important between-group differences in the overall patient experience, aside from the treatment itself. For example, if one group receives more time with treating therapists or receives cointerventions in addition to the intended treatment, this disparity can present a competing explanation for any observed benefits. For this reason, investigators often try to structure study protocols to minimize any unnecessary between-group differences in overall experience during the study, other than the treatment(s) of interest.

What were the results?

• How large was the treatment effect?
• How precise was the estimate of the treatment effect?

Physiotherapists should understand statistical analyses and the presentation of quantitative results when critically appraising an article. This article is a good overview.

Will the results help me in caring for my patients?

• Can the results be applied to my patient care?
• Were all clinically important outcomes considered?
• Are the likely treatment benefits worth the potential harms and costs?

The final question in a critical appraisal of evidence involves a series of deliberate judgments about the relevance and applicability of the evidence to a specific patient in the context of a specific clinical setting. An evidencebased practitioner will need to decide whether the patient under consideration is sufficiently similar to the patients in the study or group of studies for the results to be relevant.

Next, the practitioner must integrate patient values, preferences, and expectations in shared decision making when selecting a particular treatment.  Also, the evidence will be relevant to a given patient only if outcomes measured in the clinical trial are consistent with the individual patient’s goals.

Many issues must be considered, such as anticipated frequency and duration of patient visits, cost of the treatment, possible discomfort or other adverse effects of the intervention of interest and of competing interventions (such as injections, surgery, or other noninvasive interventions), and how consistent the treatment is with patient expectations. This final question also prompts the practitioner to integrate personal clinical expertise. Some treatments require specialty skills or specific equipment that may not be currently available and may not be obtainable in a reasonable amount of time to help a particular patient.

Critical Appraisal Worksheets

These critical appraisal worksheets from the centre for evidence based medicine are very useful:

• Systematic Review Critical Appraisal Sheet
• Diagnosis Critical Appraisal Sheet
• Prognosis Critical Appraisal Sheet
• Therapy / RCT Critical Appraisal Sheet

Resources

The Pedro Tutorial is designed to help readers of clinical trials differentiate those trials which are likely to be valid from those that might not be. It also looks briefly at how therapists might use the findings of properly performed studies to make clinical decisions. The PEDro scale is a valid measure of the methodological quality of clinical trials. DeMorton (2009) suggests it is valid to sum PEDro scale item scores to obtain a total score that can be treated as interval level measurement and subjected to parametric statistical analysis.

Cleland, Noteboom, Whitman and Allison. A Primer on Selected Aspects of EvidenceBased Practice Relating to Questions of Treatment, Part 1: Asking Questions, Finding Evidence, and Determining Validity. Journal of Orthopaedic & Sports Physical Therapy, 2008, 38(8)

Cleland, Noteboom, Whitman and Allison. A Primer on Selected Aspects of EvidenceBased Practice Relating to Questions of Treatment, Part 2: Interpreting Results, Application to Clinical Practice, and Self-Evaluation. Journal of Orthopaedic & Sports Physical Therapy, 2008, 38(8)

Understand Levels and Grades of evidence with our related page

Understand simple statistics with our Test Diagnostics page

Recent Related Research (from Pubmed)

Failed to load RSS feed from http://www.ncbi.nlm.nih.gov/entrez/eutils/erss.cgi?rss_guid=1ZAz9AIS267OrH_CZsJqeJYNDpV6n4w6_plLoZo9AYhTtmggZi|charset=UTF-8|short|max=10: Error parsing XML for RSS </div>

References

References will automatically be added here, see adding references tutorial.