Appraising the Quality of Knowledge Resources

Original Editor - Wanda van Niekerk

Top Contributors - Wanda van Niekerk and Jess Bell  

Introduction[edit | edit source]

The third step in the evidence-based practice process is appraising the quality of the resources found by creating a clinical question and locating the resources. Critically appraising a paper or study implies that you closely examine the results of the paper so that you can decide whether it is worthy of being used to inform your clinical practice.[1] Unfortunately, being peer-reviewed and published does not necessarily mean that a study is of good quality.[2] Clinical research is criticised as not being useful for reasons such as[2]:

  • it is not sufficiently pragmatic (applicable to real-life scenarios)
  • it is not patient-centred, transparent or feasible

Also, with so many research papers published, it is challenging to decide which studies to use or not use. This is where the critical appraisal of studies is advised.

How to Deal with this Information Overload?[edit | edit source]

The vast amount of published information available creates two questions for rehabilitation professionals.[3]

  1. How do you ensure that you find what you need in the vast amount of available information?
    1. Develop a well-thought-through clinical question (Evidence-based Practice and Patient Needs)
    2. Use specific search terms (Locating the Knowledge Resources in Evidence-Based Practice)
  2. How do you know which findings are based on sound science and apply to your specific case or scenario?
    1. Some useful tips to get you started:
      1. If there is a good, summarised, evidence-based information platform available that answers your exact clinical question, use it, learn from it and apply it as you see fit with your clinical reasoning skills.[3] You can find a list of some platforms here: Platforms with Synthesised, Evidence-Based Clinical Information.
      2. Systematic reviews or clinical practice guidelines are also useful to consult.

Steps in Appraising the Quality of Knowledge Resources[edit | edit source]

Before using the results of a study to help you with the clinical decision-making process, it is important to determine if the study has used sound methods. Poorly designed studies may lead to bias and may provide you with misleading results.[1]

Levels of Evidence[edit | edit source]

Levels of evidence (also referred to as the hierarchy of evidence) help in deciding which study type will provide the best evidence for a specific question. The heuristic ("rule of thumb") developed by the Oxford Centre for Evidence-based Medicine is a useful tool to refer to. See below for links to the different documents. For a better understanding, please read the introductory and background document together with the table.

If you would like to read more on levels of evidence, please see this page: Hierarchy of evidence.

Study Designs[edit | edit source]

Not all studies indicate the exact study design used. The methods section of a paper can help determine which study design was used, but to do that, clinicians need to have a good understanding of the different study designs. The list below highlights some of the main study types that will help with evidence-based practice.

  • Clinical practice guidelines
    • "set of healthcare recommendations developed by reviewing the scientific literature and consensus from an expert panel"[7]
  • Systematic reviews
    • the pooling of various primary or individual studies that fit pre-specified eligibility criteria to answer a specific research question.
    • systematic methods are used to reduce bias.
    • systematic reviews should[8]:
      • have clear objectives and pre-defined eligibility criteria
      • have reproducible methodology
      • do a systematic search to identify all eligible studies
      • assess the validity of the included studies
      • present and synthesise findings systematically[8]
    • You can read more about systematic reviews here.

"Systematic reviews are aimed at providing a comprehensive and unbiased synthesis of large numbers of relevant studies within the confines of a single document by using rigorous and transparent methods."[9]

  • Descriptive studies[10]
    • try to provide an idea or picture of what is going on or happening in a specific population
    • describe the problem and do not examine relationships or associations[11]
    • descriptive studies have PICO components such as Population (P) and Outcome (O)
    • can include: case reports, case series, qualitative studies and surveys (cross-sectional studies)
  • Analytical studies[10]
    • investigate the relationship between two factors - for example, the effect of an intervention on the outcome
    • PICO or PECO components include Interventions (I) or Exposures (E) that are applied to different groups and compared (C)
    • can be experimental studies or observational analytic studies
  • Experimental studies:
  • Observational analytical studies:
    • Cohort studies– a group of participants observed over a period of time – effect of risk factors on an outcome[17]
    • Cross-sectional studies – provide a snapshot of what’s going on in a population at a specific time[18]
    • Case-control studies – two groups, one with the outcome of interest and one without[19]
  • Historically controlled studies[20]
    • studies that compare a historical population, like a group of patients who received a specific treatment in the past or from whom data were collected previously, with a group of patients receiving a novel treatment in the present.
  • Mechanism-based reasoning studies
    • involves "the inference from mechanisms to claims that an intervention produced a patient-relevant outcome"[21]
  • Case series
    • "a collection of subjects (usually, patients) with common characteristics used to describe some clinical, pathophysiological, or operational aspect of a disease, treatment, exposure or diagnostic procedure"[22]
    • case series do not include a comparison group[22]
    • often use a sample of convenience[22]

Ways to Determine the Study Design[edit | edit source]

The Oxford Centre for Evidence-Based Medicine provides a brief guide on study designs. You can have a look at it here. Also, have a look at Figure 1 in this guide, which shows the design tree of the different types of studies. Three questions are highlighted that will guide you in deciding which type of study was done.[10]

  • Question 1 (Q1) - "What was the aim of the study?"
    • to describe a population - descriptive study
    • to investigate the relationship between factors - analytic
  • Question 2 (Q2) - "If the study is analytical, was there a random allocation of the intervention?"
    • yes - randomised controlled trial
    • no - observational analytic
      • for this group of observational studies, the main types of studies are dependent on the timing of the measurement of outcome
  • Question 3 (Q3) - "When were the outcomes determined?"
    • some time after the intervention - prospective cohort study
    • at the same time as the intervention - cross-sectional study
    • before the exposure was defined or determined - case-control study (retrospective study)

You can also find a downloadable PDF by Jeremy Howick about study designs in this brief guide, and read more about the advantages and disadvantages of the designs here.

Critical Appraisal[edit | edit source]

Critical appraisal is the process of evaluating the strengths and weaknesses of a published study. This process is structured and rigorous, and it assesses all the stages of research. "Critical appraisal is a method of systematically examining research studies to determine the worth of their findings."[23]

Helpful websites for critical appraisal checklists are:

You can read more on critical appraisal here: Appraise the evidence

How to Read a Paper[edit | edit source]

Papers can be read in the following order to make reading research articles less intimidating[3]:

  1. Title
  2. Abstract
  3. Figures and tables
  4. Introduction
  5. Discussion
  6. Results
  7. Methods
  8. References

If you'd like to read more, please see: How to read a paper.

References[edit | edit source]

  1. 1.0 1.1 Hoffmann T, Bennett S, Del Mar C. Evidence-based practice across the health professions. Elsevier Health Sciences; 2023.
  2. 2.0 2.1 Ioannidis JP. Why most clinical research is not useful. PLoS medicine. 2016 Jun 21;13(6):e1002049.
  3. 3.0 3.1 3.2 Olivier, B. Appraising the Quality of the Knowledge Resources Course. Plus. 2023
  4. Jeremy Howick, Iain Chalmers, Paul Glasziou, Trish Greenhalgh, Carl Heneghan, Alessandro Liberati, Ivan Moschetti, Bob Phillips, and Hazel Thornton. “The 2011 Oxford CEBM Levels of Evidence (Introductory Document)”. Oxford Centre for Evidence-Based Medicine. https://www.cebm.ox.ac.uk/resources/levels-of-evidence/ocebm-levels-of-evidence
  5. Jeremy Howick, Iain Chalmers, Paul Glasziou, Trish Greenhalgh, Carl Heneghan, Alessandro Liberati, Ivan Moschetti, Bob Phillips, and Hazel Thornton. “Explanation of the 2011 Oxford Centre for Evidence-Based Medicine (OCEBM) Levels of Evidence (Background Document)”. Oxford Centre for Evidence-Based Medicine. https://www.cebm.ox.ac.uk/resources/levels-of-evidence/ocebm-levels-of-evidence
  6. OCEBM Levels of Evidence Working Group. “The Oxford Levels of Evidence 2”. Oxford Centre for Evidence-Based Medicine. https://www.cebm.ox.ac.uk/resources/levels-of-evidence/ocebm-levels-of-evidence OCEBM Levels of Evidence Working Group = Jeremy Howick, Iain Chalmers (James Lind Library), Paul Glasziou, Trish Greenhalgh, Carl Heneghan, Alessandro Liberati, Ivan Moschetti, Bob Phillips, Hazel Thornton, Olive Goddard and Mary Hodgkinson
  7. Conley B, Bunzli S, Bullen J, O'Brien P, Persaud J, Gunatillake T, Dowsey MM, Choong PF, Lin I. Core recommendations for osteoarthritis care: a systematic review of clinical practice guidelines. Arthritis care & research. 2023 Feb 10.
  8. 8.0 8.1 Krnic Martinic M, Pieper D, Glatt A, Puljak L. Definition of a systematic review used in overviews of systematic reviews, meta-epidemiological studies and textbooks. BMC medical research methodology. 2019 Dec;19:1-2.
  9. Santos WM, Secoli SR, Püschel VA. The Joanna Briggs Institute approach for systematic reviews. Revista latino-americana de enfermagem. 2018 Nov 14;26:e3074.
  10. 10.0 10.1 10.2 Centre for Evidence-Based Medicine. Study Designs. Available from https://www.cebm.ox.ac.uk/resources/ebm-tools/study-designs (last accessed 29 November 2023)
  11. Siedlecki SL. Understanding descriptive research designs and methods. Clinical Nurse Specialist. 2020 Jan 1;34(1):8-12.
  12. 12.0 12.1 12.2 Zabor EC, Kaizer AM, Hobbs BP. Randomized controlled trials. Chest. 2020 Jul 1;158(1):S79-87.
  13. Granholm A, Alhazzani W, Derde LP, Angus DC, Zampieri FG, Hammond NE, Sweeney RM, Myatra SN, Azoulay E, Rowan K, Young PJ. Randomised clinical trials in critical care: past, present and future. Intensive care medicine. 2022 Feb 1:1-5.
  14. Zuidgeest MG, Goetz I, Meinecke AK, Boateng D, Irving EA, van Thiel GJ, Welsing PM, Oude-Rengerink K, Grobbee DE. The GetReal Trial Tool: design, assess and discuss clinical drug trials in light of real world evidence generation. Journal of Clinical Epidemiology. 2022 Sep 1;149:244-53.
  15. IMI GetReal. GetReal Trial Tool - Motiongraphic. Available from: https://www.youtube.com/watch?v=TKxf73RIH0w&t=112s [last accessed 30/11/2023]
  16. IMI GetReal. GetReal Trial Tool tutorial. Available from: https://www.youtube.com/watch?v=tu1sE06wfDY&t=6s [last accessed 30/11/2023]
  17. Wang X, Kattan MW. Cohort studies: design, analysis, and reporting. Chest. 2020 Jul 1;158(1):S72-8.
  18. Savitz DA, Wellenius GA. Can cross-sectional studies contribute to causal inference? It depends. American Journal of Epidemiology. 2023 Apr;192(4):514-6.
  19. Dey T, Mukherjee A, Chakraborty S. A practical overview of case-control studies in clinical practice. Chest. 2020 Jul 1;158(1):S57-64.
  20. Ghadessi M, Tang R, Zhou J, Liu R, Wang C, Toyoizumi K, Mei C, Zhang L, Deng CQ, Beckman RA. A roadmap to using historical controls in clinical trials–by Drug Information Association Adaptive Design Scientific Working Group (DIA-ADSWG). Orphanet Journal of Rare Diseases. 2020 Dec;15(1):1-9.
  21. Howick J, Glasziou P, Aronson JK. Evidence-based mechanistic reasoning. Journal of the Royal Society of Medicine. 2010 Nov 1;103(11):433-41.
  22. 22.0 22.1 22.2 Porta M, editor. A dictionary of epidemiology. Oxford university press; 2014 May 23.
  23. Crombie IK. The pocket guide to critical appraisal. John Wiley & Sons; 2022 Jun 20.