Chapter 6. How to Use a Systematic Review and Meta-Analysis
Alonso Carrasco-Labra, D.D.S., M.Sc., Ph.D.; Romina Brignardello-Petersen, D.D.S., M.Sc., Ph.D.; Michael Glick, D.M.D.; Gordon H. Guyatt, M.D., M.Sc.; and Amir Azarpazhooh, D.D.S., M.Sc., Ph.D.
In This Chapter:
Knowledge Synthesis and Translation
What Is the Difference Between Narrative and Systematic Reviews?
Why Are Systematic Reviews Considered to Be a Study Design?
Critically Appraising Systematic Reviews to Inform Clinical Decisions
• How Serious Is the Risk of Bias?
Introduction
In the previous chapters in this book, we introduced the process of evidence-based dentistry (EBD)1 and explained how to search for evidence to inform clinical practice2 and how to use articles about therapy,3 harm,4 and diagnosis.5 In this chapter, we explain how to use a systematic review to answer a clinical question, introduce and describe the basic concepts for understanding the design of a systematic review, and explain how to use these concepts to critically appraise such studies. In addition, readers will learn how to interpret and use the results presented in the systematic review to inform clinical decisions. In a subsequent chapter, we describe how to use evidence-based clinical practice guidelines.
Box 6.1. Clinical Scenario
While working at a clinic in a remote rural location, you meet with a 55-year-old woman who has been referred to you for treatment of open caries on tooth #20, which the referring clinician has diagnosed as having irreversible pulpitis. After conducting a clinical examination, you recommend that the tooth be extracted. The patient asks if she will need to attend many follow-up appointments, because traveling from her home to your clinic is a burden. Among other measures, you decide to minimize your patient’s risk of having alveolar osteitis (dry socket) as a postoperative complication. You have been in the habit of prescribing chlorhexidine in cases such as this one; however, you wonder if this medication is actually of any use in decreasing the patient’s risk of experiencing dry socket. You consult the Cochrane Library and find a recently published systematic review that addresses this issue.6
Knowledge Synthesis and Translation
Keeping updated in knowledge related to any clinical discipline is challenging. According to the Medline (PubMed) Trend Database,7 more than 30,000 randomized controlled trials (RCTs) were published in 2012. Of these, approximately 1,500 were dental trials. This overwhelming amount of information can be impossible to manage for any health care provider, including dentists. Systematic reviews represent an efficient way to learn about the available evidence for an intervention or exposure.
According to the Cochrane Collaboration,8 a systematic review is defined as “a review of a clearly formulated question that uses systematic and explicit methods to identify, select, and critically appraise relevant research, and to collect and analyze data from the studies that are included in the review.” Like primary observational studies and randomized controlled trials, systematic reviews represent a type of study that requires implementation in a manner that minimizes the risk of bias. If these bias-reducing strategies (that is, the systematic review process) are in place, then readers can trust that the appropriate analysis will provide the best available estimate of the effect of an intervention or exposure.8
What Is the Difference Between Narrative and Systematic Reviews?
Authors of many articles published in the dental literature refer to their work as “reviews”; only a small number of these articles, however, are systematic reviews. For this reason, clinicians interested in using review articles must be able to distinguish between narrative and systematic reviews. In narrative reviews, authors discuss one or more aspects related to a particular condition or disease (that is, etiology, diagnosis, prognosis, or therapy and management). Although this type of review may be useful for dental students (and sometimes dentists) who are interested in background information, the failure of authors to systematically collect and process the data makes the conclusions that arise from such articles potentially misleading.9,10 In contrast, authors of trustworthy reviews use systematic, transparent, and comprehensive methods to retrieve, select, critically appraise, and summarize all the available evidence regarding the effectiveness of an intervention, prognosis, and diagnosis questions. Although it is possible to have systematic reviews that focus on prognosis and diagnosis (and harm) issues, in this chapter, we will focus on therapy (and harm) issues. Table 6.111 presents additional details regarding the difference between narrative and systematic reviews.
One of the key limitations of systematic reviews and meta-analyses is that the summary estimates produced are only as trustworthy as the results of the primary studies that inform the review.12 Thus, the authors of rigorously conducted systematic reviews still may report low-quality evidence that warrants only weak inferences.
Table 6.1. Differences Between Narrative and Systematic Reviews*
|
Characteristic |
Narrative Reviews |
Systematic Reviews |
|
Clinical Question of the Review |
Seldom reported, or addressed several broad questions |
Focused question specifying population, intervention or exposure, and outcome |
|
Search for Primary Studies |
Seldom reported; if reported, not comprehensive |
Comprehensive search of databases of evidence resources |
|
Selection of Primary Studies |
Seldom reported; if reported, often biased sample of studies |
Explicit selection criteria for primary studies |
|
Assessment of the Risk of Bias of Primary Studies |
Seldom reported; if reported, not usually systematic |
Risk of bias of primary studies assessed |
|
Methods to Summarize the Included Studies’ Results |
Usually qualitative nonsystematic summary |
Synthesis is systematic (qualitative or quantitative) |
* Reprinted with permission of JAMA and The JAMA Network from Guyatt and colleagues.11
Why Are Systematic Reviews Considered to Be a Study Design?
Systematic reviews and evidence-based clinical practice guidelines represent the most valuable documents that can inform clinical decision-making.12 Review authors start by defining a clear and focused research clinical question in a way that is similar to that described in a previous chapter in this book, which addressed the framing of a question using the Population, Intervention, Comparison, Outcomes (PICO) approach (see Chapter 2).2 The components of this PICO question correspond with the eligibility criteria outlined by the authors of the studies that are included in the review. For example, if the author specifies the population of interest as patients after tooth extraction, eligible studies must enroll such patients, and only such patients. Authors of systematic reviews should present clearly the population or type of patients, clinical intervention or exposure of interest, the comparator for such an intervention, and the selection of patient-important outcomes.
After specifying all these criteria, review authors should describe their methods in a protocol that, ideally, should be published or made available to users. Many reasons support the authors’ explicit and transparent declaration of selection criteria and methods in systematic reviews. Investigators have described important discrepancies between the selection of outcomes and methods when comparing them before the review starts with after the final manuscript is published.13 Ideally, reviewers will describe the comprehensive search for all the available (published and unpublished) evidence they have conducted. As a result of this process, researchers should screen a set of references at the level of title and abstract first, and complete study reports later, to establish whether the articles retrieved meet the selection criteria (Figure 6.111). Having identified all the relevant studies, reviewers abstract data that are related to the studies’ characteristics, risk of bias assessment, and results. Finally, they summarize the evidence and assess the quality of the body of evidence14 (Figure 6.111).
Figure 6.1. Process of Conducting a Systematic Review
Adapted with permission of JAMA and The JAMA Network from Guyatt and colleagues.11
Box 6.2. The Study You Found
You consult the Cochrane Library and find a recently published systematic review whose authors aimed to assess the effects of local interventions on the prevention and treatment of alveolar osteitis (dry socket) after tooth extraction.6 Reviewers identified 21 trials, and among these, the authors of 18 trials, which included more than 2,370 participants, reported results related to the prevention of dry socket. Although the authors of the systematic review reported more than 10 intrasocket interventions, they found limited evidence for each of these interventions.
Critically Appraising Systematic Reviews to Inform Clinical Decisions
As described in previous chapters, the process of using an article from the dental literature involves assessing the risk of bias, assessing the results, and assessing the applicability of the results.14
How Serious Is the Risk of Bias?
The probability that the results of a systematic review are correct depends on whether reviewers identified, retrieved, selected, critically appraised, and summarized all the relevant studies.15 Table 6.216–21 presents examples of the aspects to consider when assessing the risk of bias of a systematic review.
Table 6.2. Examples Illustrating Critical Appraisal of the Risk of Bias of a Systematic Review
|
Aspect |
Example |
Explanation |
|
How Serious Is the Risk of Bias? |
||
|
Did the review include explicit and appropriate eligibility criteria? |
“Population: orthodontics patients of either gender, any age and any type of malocclusion (Class I, II, or III) and crowding treated with fixed multibrackets on both arches with first molars included over the course of at least 12 months. Intervention: only in vivo studies on human participants involving different oral health motivation strategies, and oral and dental hygiene techniques and procedures. Comparison: no treatment or usual care (the gold standard), or inactive control. Outcome: as a primary outcome, the following data were evaluated: plaque index (PI) and gingival index (GI). The secondary outcomes considered were carious lesions and the presence/absence of white spot.”16 |
The reviewers aimed to answer the question “whether it is clinically possible to avoid plaque increase and prevent permanent teeth lesions in orthodontics patients, and in particular, whether prophylactic procedures performed by the dental hygienist are efficacious in reducing the risk of demineralization in orthodontics patients fitted with multibracket appliances.”16 |
|
Was the search for relevant studies detailed and exhaustive? |
“Relevant articles were identified in any language by searching MEDLINE (from 1950 to September 2011), the entire Cochrane Library (from 1990 to September 2011), CINAHL Nursing database (from 1980 to September 2011), and the University of Michigan School of Dentistry ‘Dentistry and Oral Sciences’ database (EBSCO host) (from 1990 to September 2011). The final search update was performed on September 19, 2011. To locate potentially relevant studies in MEDLINE, exploded MeSH terms and key words were used to generate sets for the following themes: 1) periodontitis; 2) preterm birth; and 3) scaling treatment. We then found the intersection of these terms using the Boolean term ‘AND.’ This basic approach was modified as necessary to search each electronic database (see Supplementary Appendix 3 in the online Journal of Periodontology). No limitations in the search were used.”17 “To locate unpublished trials, ClinicalTrials.gov and abstracts of scientific conferences were searched (National Academy for State Health Policy’s 20th Annual State Health Policy Conference, Denver, Colorado, October 15, 2007; 34th National Conference of Indian Society of Periodontology, Dharwad, India, December 3-5, 2009).”17 |
Study authors described every electronic database consulted during the review searching process along with the time frame searched. They also reported the complete search strategy in an online supplementary appendix. Additionally, they searched for “grey literature” in clinical trials registries and conference abstracts. One potential limitation in this review is that the authors did not include EMBASE among the included databases for searching. Some reports show that systematic reviews including one or a few databases have a higher risk of missing relevant studies compared with reviews including all of the most important databases.18 The Cochrane Handbook suggests that at least MEDLINE, EMBASE, and CENTRAL should be consulted in every systematic review search.19 The possibility that this review missed some relevant primary studies cannot be discarded. |
|
“We performed the risk-of-bias assessment for the included trials by using the Cochrane Collaboration’s risk-of-bias assessment tool, which incorporated six domains: random sequence generation, allocation concealment, masking, completeness of outcome data, risk of selective outcome reporting and risk of other bias.”20 “Using the predetermined six domains for risk-of-bias assessment, we determined three of the five RCTs* to have a high risk of bias, whereas we judged two to have an unclear risk of bias and none to have a low risk of bias.”20 |
The authors presented a table with a detailed description of the risk of bias for each of the included studies. This analysis showed that most of the studies were judged as having a high risk of bias. The rest of the studies were classified as unclear risk of bias, which raises concerns about the review results. |
|
|
Were assessments of studies reproducible? |
“Two review authors (Anneli Ahovuo-Saloranta [AAS] and Helenab Forss [HF]) independently carried out the selection of papers on the basis of the title, keywords and abstract, and the decisions about eligibility. The full text of every study considered for inclusion was obtained. If the information relevant to the inclusion criteria was not available in the abstract or if the title was relevant but the abstract was not available, the full text of the report was obtained.”21 “Data were extracted independently and in duplicate by two review authors (AAS, HF) using a previously prepared data extraction form. The extraction form was pilot-tested independently by two review authors in the previous review version (AAS, Anne Hiiri [AH]) with a sample of studies to be included.”21 |
Authors provided an exhaustive explanation about all the methodological steps involved to minimize the risk of mistakes and arbitrary judgments during the review process. Critical steps were conducted independently and in duplicate. |
* RCT = randomized controlled trial.
Did the review present explicit and appropriate eligibility criteria?
Consider a systematic review whose author intended to pool the effects of all antibiotics for treating all types of maxillofacial infections. Now, consider a review whose author intended to pool the results of the effects of fluoride varnishes for reducing the incidence of carious lesions in children and adolescents. Clearly, the first review would be excessively broad in its scope; equally clearly, the second review would be satisfactorily narrow.
What makes a systematic review question too broad? There are pathophysiological and microbiological aspects of maxillofacial infections and of the mechanism of the action of antibiotics that suggest that treatment effects vary across different types of patients (different types of infections) and interventions (different types of antibiotics).11 A single pooled estimate summarizing all these data would not be applicable to any specific group of patients or any specific antibiotic, and therefore, it would be of no use to the clinician. On the other hand, results from the more focused question are likely to be similar for children and adolescents, for the available fluoride varnishes, and for the available approaches to identifying carious lesions. This similarity of effect across patients, interventions, and outcomes is what legitimizes the single pooled estimate of effect.22
Was the search for relevant studies exhaustive?
Secure estimates of treatment effect require a comprehensive search for eligible studies. Unfortunately, even a search that includes all the relevant electronic databases may be insufficient. If the authors of a primary study selectively report and publish research results according to the effect (that is, studies favoring the intervention get published; studies that are negative do not), then the results will be a systematic overestimate of effect (that is, publication bias).23,24 One way to mitigate this issue is to search and include “gray literature,” which includes documents such as dissertations, conference abstracts, personal correspondence, records of studies’ methods and results found in investigators’ file drawers or on their hard drives, and policy documents.25,26
When assessing whether the search strategy was comprehensive, users should focus on which databases the authors consulted, and whether the systematic reviewers restricted their search to resources published in one particular language or to only published reports.19 To learn more about specific databases and their characteristics, please refer to Chapter 2.
Did the primary studies have a low risk of bias?
The credibility of the results of a systematic review is as only as great as the credibility of the primary studies included. Assessing credibility requires addressing the risk of bias associated with the primary studies that are included in a systematic review. Studies assessed as having a high risk of bias may overestimate treatment effects by up to 150%.27
Thus, when the risk of bias across the included studies is low, credibility increases.11 Different primary study designs are associated with specific sets of potential biases and, consequently, require reviewers to follow specific checklists to assess the risk of bias. For issues of therapy addressed in RCTs, key considerations include concealment of randomization, blinding, and minimizing loss to follow-up.5,28
Were the selection and assessment of primary studies reproducible?
All of the steps that investigators perform when conducting a systematic review (Figure 6.111) are susceptible to error.29 For example, including and excluding studies from the review requires making judgments. Likewise, errors may occur during data abstraction: for example, investigators found that 20 of 34 reviews conducted by the Cochrane Cystic Fibrosis and Genetic Disorders Group included errors such as miscalculations and misinterpretation of data from the primary studies.30 Independent, duplicate eligibility review and data extraction with resolution of discrepancies can minimize such errors and other unintentional or subconscious bias.31 Hence, users of systematic reviews should check whether reviewers conducted screening and data extraction in duplicate.
Box 6.3. Your Assessment of the Risk of Bias of the Systematic Review You Identified
The authors of the systematic review6 defined a sufficiently narrow range of patients, interventions, comparisons, and outcomes, and they conducted a comprehensive search that included reviewing several databases and making an effort to retrieve unpublished data. The reporting by authors of the included primary studies was poor, and the result was that risks of bias ratings were often assessed as unclear risk of bias. Finally, the authors of the systematic review reported that they selected the studies in duplicate and independently. Thus, you determine that this systematic review has a low to moderate risk of bias, and you proceed to read and interpret the results.
Stay updated, free dental videos. Join our Telegram channel
VIDEdental - Online dental courses
