Meta-analysis: Critical Appraisal


Department of Neurology Neurosciences Centre, and Clinical Epidemiology Unit, All India Institute of Medical Sciences, New Delhi Delhi, India
This chapter deals with the critical appraisal of a meta-analysis of treatment studies that are randomised.

This chapter deals with the critical appraisal of a meta-analysis of treatment studies that are randomised.


When checking the relevance, you ask whether the population, intervention, comparison and outcome of the meta-analysis are matching those of your clinical question or not.

Validity Assessment

The following questions need to be asked and their answers examined while critically appraising a meta-analysis or systematic review:

  • Q1. Did the review (meta-analysis) address a focused clinical question?
  • Q2. Was the search for relevant studies comprehensive and well documented?
  • Q3. Were the included studies of high methodological quality?
  • Q4. Was there a good agreement between reviewers in selection and assessment of studies?
  • Q5. Were the combined results combinable (similar from study to study)?

Q.1. Did the Review (Meta-analysis) Address a Focused Clinical Question?

The strength of a meta-analysis comes from the ability to combine the results across studies. However, one has to be cautious not to combine apples and arranges. If the effect is similar across studies, the results support combining; if not, they raise the questions like combining apples and oranges.
1A. Why do we ask this question?
The question is important from two (the internal and external validity) points of view. First, if the effects are dissimilar across the studies and if the underlying biology suggests that the effects would be dissimilar – yet the reviewers combine the study results to get a summary – then the summary is likely to be misleading (a threat to internal validity). Such a summary would not be applicable to patients.


Internal validity here would refer to the extent to which the summary result reflects the ‘true’ effects across the studies in the meta-analysis.

External validity refers to the extent to which the summary results would apply to patients (and/or interventions) outside those in the studies’ (Lack of focus in the question may pose a threat to both internal as well as external validity).
1B. How do we answer the question?
In the methods section look for precise statement of what range of patients, interventions and outcomes the reviewer has considered for the meta-analysis, then consider whether it is plausible that the effects across the range of patients, interventions and outcomes could be similar. If the answer is yes, then the review is addressing a focused clinical question, otherwise not.
Here, some caution is necessary in answering the question. No matter how similar are patients, interventions and outcomes, there would still be some differences across the included studies. You should not reject the review without looking at the results. Our knowledge of biology is usually so incomplete that results may differ from our expectation. Also, a narrowly focused review may have the limitations of a subgroup analysis that often yield false-positive conclusions. Therefore, a broadly focused review is preferable to a narrowly focused review. For example, the role of antiplatelet events (aspirin, dipyridamole, etc.) in arterial thrombosis (stroke, myocardial infarction, etc.) may be a reasonable topic for meta-analysis, but effects of chemotherapy on all cancers are not, because chemotherapy is effective for some cancers but not for others.
1C. How do we interpret the answer?
If the question is reasonably focused, you proceed without hesitation. Even if the question appears too broad, it is advisable to proceed further and revisit this question while evaluating similarity of results across the studies.

Q.2. Was the Search for Relevant Studies Comprehensive and Well Documented?

A comprehensive search would require searching for both published and unpublished studies. The search should be documented so that if you (or someone) had doubt about its comprehensives or wanted to replicate the meta-analysis, you could replicate the search and check the yield.
2A. Why do we ask this question?
The question is important to ensure that the review does not suffer from publication bias (positive studies are published more easily than negative ones). There are examples of meta-analyses, which predicted very strong effects that were not observed in a subsequent clinical trial. For example, a meta-analysis predicted that magnesium sulphate would be highly effective in reducing mortality following acute myocardial infarction. A subsequent clinical trial did not find such effects. The meta-analysis has been alleged to have ‘publication bias’ [1].
2B. How do we answer the question?
Read the methods section to find how the reviewers searched for studies. A good search may include:


Electronic databases, such as MEDLINE, EMBASE and the Cochrane Controlled Trials Register

Textbooks and monographs

Reference lists of retrieved articles

Contact with experts

Abstracts of conferences/meetings

Contact with pharmaceutical companies

Hand searching
Experts and pharmaceutical companies help to provide ‘in press’ and co-published studies. (There are statistical methods to explore possibility of publication bias, but they are beyond the scope of this book.) Non-English language studies also need to be included in the review.
2C. How do we interpret the answer?
If the search is not comprehensive (e.g. limited only to English language), then findings of the review may be considered only tentative.

Q.3. Were the Included Studies of High Methodological Quality?

Poor design or conduct of the included studies may compromise the validity of the review. The quality is relevant in two of the steps of meta-analysis or SR:


While selecting studies for inclusion into the review

While assessing the quality of the included studies
3A. How do we answer the question?
Read the methods section of the review to find whether reviewers used a certain selection criteria for including only good-quality studies. Usually, a study design is used as selection criteria, e.g. randomised controlled trials.
Reviewers should also assess whether the studies have been conducted properly. Thus, even for randomised trials, quality needs to be critically appraised using the criteria and questions given in the therapy section of this book.
3B. How do we interpret the answer?
If methodological quality was not assessed, you should doubt the results of the meta-analysis. If this was done, you may like to see if there was good agreement between two reviewers. A kappa of around 0.6 or more would be acceptable. The next thing is to see whether the reviewers have used the quality score in excluding, or weighting or prioritising the studies while combining them.

Q.4. Was There a Good Agreement Between Reviewers in Selection and Assessment of Studies?

Only gold members can continue reading. Log In or Register to continue

Oct 18, 2015 | Posted by in General Dentistry | Comments Off on Meta-analysis: Critical Appraisal
Premium Wordpress Themes by UFO Themes