Department of Neurology Neurosciences Centre, and Clinical Epidemiology Unit, All India Institute of Medical Sciences, New Delhi Delhi, India
The term ‘meta-analysis’ was first used by Gene Glass in his presidential address to the American Educational Research Association in 1976. He distinguished it from primary analysis (analysis of original data in a research study, usually carried out under the direction of those who designed the study) and secondary analysis (reanalysis of the data with better statistical techniques usually carried out by someone not involved in the design of the original study). Meta-analysis, according to Glass, is a statistical analysis of summary results of individual studies for the purpose of integrating the findings. It may be noted, however, that the idea of combining results from separate studies can be identified as early as 1904 when Pearson first used data pooling. In 1932, Sir Ronald Fisher reported on combining P values. Gene Glass advanced the field by making original and groundbreaking contributions.
Origin of the Term ‘Meta-analysis’
The term ‘meta-analysis’ was first used by Gene Glass in his presidential address to the American Educational Research Association in 1976. He distinguished it from primary analysis (analysis of original data in a research study, usually carried out under the direction of those who designed the study) and secondary analysis (reanalysis of the data with better statistical techniques usually carried out by someone not involved in the design of the original study). Meta-analysis, according to Glass, is a statistical analysis of summary results of individual studies for the purpose of integrating the findings. It may be noted, however, that the idea of combining results from separate studies can be traced to 1904 when Pearson first used data pooling. In 1932, Sir Ronald Fisher reported on combining P values. Gene Glass advanced the field by making original and groundbreaking contributions.
Scope of Meta-analysis
In recent years, the term ‘meta-analysis’ has been used in a broad sense, and its scope has also enlarged. Meta-analysis is being used in almost all branches of science including meteorology, agriculture, psychology, nuclear physics and, of course, medicine.
In medicine, meta-analyses may be done on surveys, diagnostic, prognostic or a etiologic studies, but most of meta-analyses to date evaluate treatment studies. The following description applies largely to meta-analysis of treatment studies.
What Is Meta-analysis?
Simply said, meta-analysis is a study of studies. However, different people use the term to mean different things. Most experts use the term to refer to the statistical (quantitative) combination of results from two or more studies addressing the same research question. Others use it to refer to systematic reviews. To understand the strengths and limitations of meta-analysis, we need to know how it differs from ‘traditional’ and ‘systematic’ reviews.
Need for Reviews
We are witnessing unprecedented information explosion. More than 1,000,000 articles are published each year in more than 20,000 journals. It is humanly impossible to read through all the articles published in any given field. Therefore, we need concise summaries of the literature, after separating the insignificant and unsound from salient and crucial. Such summaries are found in review articles in journals.
Traditional (Narrative) Reviews: What Is It?
Students, residents as well as experts often look for review articles. When residents have to present seminars or journal clubs, when experts have to deliver lectures or talks or write a chapter in a book and when practitioners want to update their knowledge and learn about new treatments or diagnostic tests, they go through the recent issues of some reputed journals in the field and look for a relevant review article. Most journals publish review articles regularly.
How are these articles written? Most often, an expert on the topic is invited or motivated to write a review article. He collects as many articles on the topic as he can find. He goes through them with varying interest. Then, he summarises the findings in his own writing style. More often than not, he has his own preconceptions about the topic (based on his own publications) that have varying influence on what he writes. The articles undergo editorial review before being published in the journals. Such review articles are sometimes referred to as ‘traditional reviews’. They may be defined as review articles written by one or more experts based on a convenience sample of studies with no description of the underlying methodology.
Strengths of Traditional Reviews
They cover a wide range of issues on a topic in a concise manner. Often, articles on a clinical issue provide a succinct summary of basic sciences in relation to the topic. Articles on a disease like hepatitis might cover epidemiology, aetiology, pathogenesis, pathology, natural history and treatment. The treatment part may cover all the available drug or surgical treatment.
Limitations of Traditional Reviews
The traditional review articles (also called ‘narrative’ reviews) have served the summarising function, but they usually have one or more of the following limitations.
Lack of explicit methodology:
Traditional review articles usually do not carry any ‘methods’ section. The reader is left uncertain whether a comprehensive search of relevant papers has been done, whether the inclusion of papers for reference has been unbiased and whether steps have been taken to limit errors in the reviewing process and that due to the primary studies.
Often, writers of the traditional reviews resort to vote-counting. They count the number of studies in favour of an intervention and the number against. Then, they may support the side having the greater number of studies. This is fallible. They count even ‘incompetent’ votes. Votes are incompetent when they have errors. Errors are because of inadequacy in the quality or quantity of data. Without adequate data, authors conclude against an intervention. This is like saying, ‘Absence of proof (of benefit) is the proof of absence (of benefit)’, but this is not true.
One ought to see the result of a study in view of its limitations. A small single-centre study should not be counted equal to a large multicentric one. An open nonrandomised study should not be given the same weight as a randomised placebo controlled one. In other words, reviewers ought to give weight according to the quality and quantity of the data in the studies. Most traditional reviews do not do this.
Over-reliance on intermediate outcomes (end points):
It has been observed that many traditional reviews give undue importance to the intermediate patient outcomes. This may be misleading and, at times, dangerous. A dramatic example is the ‘trial of encainide and flecainide’ [1 2]. These drugs were shown to suppress abnormal ventricular depolarisations, but in the randomised trial, the mortality was substantially higher in the patients receiving the antiarrhythmic drugs. Reviewers relying on only arrhythmia as the outcome could have erroneously concluded in favour of the drugs. The error would occur because of over-reliance on intermediate outcomes. The intermediate outcomes must be closely and strongly related to the clinically important outcomes for reviewers to consider recommending on the basis of those. Such relationships are few and far between.
‘Missed opportunity’ to combine the results:
Traditional reviews often look at the individual studies separately but do not combine their results, even if they are combinable. Thus, they may miss an opportunity to arrive at clearer conclusions, which may be possible after combining the results.
‘Unity is strength’ is a famous adage. By uniting (combining) the results of the individual studies, reviewers may gain strength (power), which in turn may permit clear-cut conclusions even if individual studies were inconclusive.
Systematic Reviews (SRs) Versus Meta-analysis
A systematic review may be defined as a review addressing a specific research question (on treatment, diagnosis, prognosis or aetiology) using explicit methodology of collecting, selecting and appraising studies and, whenever appropriate, synthesising their results quantitatively.
The definition indicates that the systematic review may have only qualitative or both qualitative and quantitative components. The qualitative component consists of the assessment of quality of individual studies, whereas the quantitative component is the meta-analysis.
The National Library of Medicine (NLM) of the USA has included in MEDLINE the term ‘meta-analysis’ as subject heading in 1989 and as publication type in 1993. It does not have systematic review as a medical subject heading or publication type. This may be one reason why the term meta-analysis is more popular.
Systematic Review but not Meta-analysis
Some experts would not use the term ‘meta-analysis’ for such systematic reviews, which do not contain statistical combination of the results of the studies. The combination may not be required if there is only one study, it may not be possible if the studies report different outcomes or it may not be desirable if the studies have widely varying results due to reasons such as differences in study population or intervention. Sometimes, a systematic review may conclude that there is no worthwhile study on the topic. Under such circumstances, it may not be acceptable to use the term meta-analysis but such circumstances are rare.
Meta-analysis but not Systematic Review
Sometimes, authors may not use a systematic process to search, select and appraise the studies on a given topic, yet find some convenient sample of studies and combine the results of the studies statistically and call it a meta-analysis. Technically it may be a meta-analysis, but it cannot be called a systematic review, because the process of finding, selecting or appraising the studies has not been systematic. Such meta-analysis may lead to erroneous results. They are poorly conducted meta-analyses.
Fortunately, most meta-analyses are not so poorly conducted, and also most systematic reviews do combine the results of the studies statistically. Hence, for practical purposes, most systematic reviews contain a meta-analysis, and hence the common usage of titles like ‘a systematic review and meta-analysis’.
Strengths of Systematic Reviews/Meta-analysis
Good systematic reviews (SRs) have several strengths, for example:
Comprehensive search strategy:
Good systematic reviews usually scan multiple sources of information. For example, Cochrane reviews use electronic databases, hand searching, experts, specialised registers of trials and personal contacts to retrieve relevant studies.