Appraising the Quality of Knowledge Resources

This article or area is currently under construction and may only be partially complete. Please come back soon to see the finished work! (30/11/2023)

Original Editor - User Name

Top Contributors - Wanda van Niekerk, Jess Bell and Kim Jackson  

Introduction

The third step in the evidence based practice process is appraising the quality of the resources found by creating a clinical question and locating the resources. Critically appraising a paper or study implies that you closely examine the results of the paper so that you can decide whether it is worthy of being used to inform your clinical practice. Unfortunately, just because a study is peer-reviewed and published does not necessarily mean that is of good quality. Clinical research is criticised as not being useful for reasons such as:

• it is not sufficiently pragmatic (applicable to real-life scenarios)
• it is not patient-centred, transparent or feasible

Also, with so many research papers published it is challenging to decide which studies to use or not. This is where the critical appraisal of studies is advised.

How to Deal with this Information Overload?

The vast amount of published information available creates two questions for rehabilitation professionals.

1. How do you ensure that you find what you need in the vast amount of available information?
   1. Develop a well-thought through clinical question (Evidence Based Practice and Patient Needs)
   2. Use specific search terms (Locating the Knowledge Resources in Evidence Based Practice)
2. How do you know which findings are based on sound science and is applicable to your specific case or scenario?
   1. Some useful tips to get you started:
      1. If there is a good summarised evidence based information platform available that answers your exact clinical question – use it, learn from it and apply it as you see fit with your clinical reasoning skills.
      2. Systematic reviews or clinical practice guidelines are also useful to consult.

Steps in Appraising the Quality of Knowledge Resources

Before using the results of a study to help you with clinical decision-making process, it is important to determine if the study has used sound methods. Poorly designed studies may lead to bias and may also provide you with misleading results.

Levels of Evidence

Levels of evidence (also referred to as the hierarchy of evidence) help in deciding which study type will provide the best evidence for a specific question. The heuristic ("rule of thumb") developed by the Oxford Centre for Evidence Based Medicine is a useful tool to refer to. Below are links to the different documents. Please read the introductory and background document together with the table. This will help with better understanding.

• Introductory document: Levels of Evidence: An introduction
• Background document: CEBM Background document: Explanation of the 2011 Oxford Centre for Evidence-Based Medicine (OCEBM) Levels of Evidence
• Table: Levels of evidence

If you would like to read more on levels of evidence, please see this page: Hierarchy of evidence

Study designs

Not all studies indicate the exact study design used. The methods section of a paper can help determine which study design was used, but to be able to do that clinicians need to have a good understanding of the different study designs. The list below highlights some of the main study types that will help with evidence based practice.

• Clinical practice guidelines
  • "set of healthcare recommendations developed by reviewing scientific literature and consensus from an expert panel"

• Systematic review
  • The pooling of various primary or individual studies that fit pre-specified eligibility criteria in order to answer a specific research question.
  • Systematic methods are used to reduce bias.
  • Systematic reviews should:
    • have clear objectives and pre-defined eligibility criteria
    • have reproducible methodology
    • do a systematic search to identify all eligible studies
    • assess the validity of the included studies
    • present and synthesise findings in a systematic way
  • You can read more about systematic reviews here.

"Systematic reviews are aimed at providing a comprehensive and unbiased synthesis of large numbers of relevant studies within the confines of a single document by using rigorous and transparent methods."

• Descriptive studies
  • tries to provide an idea or picture of what is going on or happening in a specific population
  • describes the problem and not examining relationships or associations
  • descriptive studies have PICO components such as Population (P) and Outcome (O)
  • can include: case reports, case series, qualitative studies and surveys (cross-sectional studies)
• Analytical studies
  • investigates the relationship between two factors - for example the effect of an intervention on the outcome
  • PICO or PECO components include Interventions (I) or Exposures (E) that are applied to different groups and compared (C)
  • can be experimental studies or observational analytic studies
  • experimental studies include randomised controlled trials
    • Randomised controlled trial - study participants are randomly assigned to either a treatment/intervention group or a control/placebo group
    • RCT are seen as the gold standard of clinical trial designto provide unbiased evidence of the effects of specific interventions
    • Participants are assigned randomly or by chance
    • The GetReal Trial Tool is a useful tool to use when designing a randomised controlled trial.
      • Read more about it: The GetReal Trial Tool: design, assess and discuss clinical drug trials in light of real world evidence generation.

• analytical observational studies include cohort studies; cross-sectional studies and case-control studies
  • Cohort studies– group of participants observed over period of time – effect of risk factors on an outcome
  • Cross-sectional study – snapshot of what’s going on in a population at a specific time
  • Case-control studies – two groups one with the outcome of interest and one without

Ways to Determine the Study Design

The Oxford Centre for Evidence Based Medicine provides a brief guide on study designs. You can have a look at it here. Also have a look at Figure 1 in this guide that shows the design tree of the different types of studies. Three questions are highlighted that will guide you in deciding which type of study was done.

• Question 1 (Q1) - "What was the aim of the study?"
  • to describe a popuplation - descriptive study
  • to investigate relationship between factors - analytic
• Question 2 (Q2) - "If the study is analytical, was there a random allocation of the intervention?"
  • yes - randomised controlled trial
  • no - observational analytic
    • For this group of observational studies, the main types of studies are dependent on the timing of the measurement of outcome
• Question 3 (Q3) - "When were the outcomes determined?"
  • some time after intervention - prospective cohort study
  • at the same time as intervention - cross-sectional study
  • before the exposure was defined or determined - case-control study (retrospective study)

You can also find a downloadable PDF by Jeremy Howick about study designs in this brief guide and read more about the advantages and disadvantages of the designs here.

https://www.physio-pedia.com/Quantitative_Research

GetReal Trial Tool

Observational analytical arm – researcher less involved than in experimental design

Historically controlled studes

Mechanism – based reasoning

Case series

Critical Appraisal

Equator Network

CASP

Resources

• bulleted list
• x

or

1. numbered list
2. x