Meta-analyses have been accused of oversimplifying the results of a research area by focusing mainly on overall effects while downplaying mediating or interaction effects, and they been criticized that sometimes they are mixing studies that measure “apples” with those that measure “oranges” [32]; ultimately, no meaningful results can be obtained. Some researchers support the idea that MAs ignore the possible impact of study quality and risk of bias on the results of a review. By assessing the quality and risk of bias of the source studies included in the MA, this problem could be eliminated. When conducting a MA, there could be some mistakes in the classification of studies or in calculating the effect sizes because of the complicated coding system used, as well as because many studies do not provide all the ­necessary information for inclusion in the analyses. A further weakness of MAs can arise from the situations when in some cases the available studies for a particular research area are few in number or they are of low quality, their data are heterogeneous [33], and thus the corresponding results should be interpreted with great caution.

The procedures employed in a MA permit quantitative synthesis of the literature that addresses specific research areas. A MA is likely to be more objective than a traditional extensive literature review, and therefore its use is more efficient to summarize large bodies of literature [34]. By means of MA, it is possible to reach stronger conclusions because more studies can be analyzed objectively with specific statistical methods as opposed to traditional narrative literature reviews. Furthermore, it should be mentioned that a MA is also very helpful in examining lack of evidence in a specific research field, providing insight into new directions for research, as well as finding mediating or interactional relationships or trends that they are either too minimal to be observed or they cannot be hypothesized and tested in individual studies [34 35].

The definition of the response variables that are intended to be examined is significant, because most of the times different definitions of the same variables (as they are described in the original studies included in the analysis), are not suitable for combining. Furthermore, there is more need to present the minimum difference in the response variable that is considered to be clinically important and not only the statistically significant differences. This way, bias in interpreting the results of the analysis could be avoided, and research is mainly focused on clinical importance.

First of all, the period of time covered by the literature search during which the desired studies have taken place must be mentioned, because this places the MA in perspective with developments in medicine that may precede, coincide with, or follow the performed study, and allow other researchers to replicate the study if needed. Nevertheless, the most important issue is a detailed description of the information sources and search strategies that were used to locate the studies to be included in the MA. In MAs, the “material” to be studied consists of individual source studies of identical or similar research fields. It is very important to identify as many of these studies as possible, so that the “sample” used in the MA will be as large and as representative as possible. An incomplete search can lead to “selection bias” by failing to identify important studies. In order to avoid selection bias when conducting a MA, the literature should be searched thoroughly and systematically. It is recommended that these search methods should include: (a) Keyword (index terms) searches of computerized bibliographic databases (such as MEDLINE), which will be described later in this chapter [37]. All additional information about these searches should also be presented, such as the databases searched, the dates covered by the search, and whether or not the search was conducted by a professional medical librarian. (b) Cross-checking citations to appropriate studies, either through reviewing the bibliographies of published articles that have been already identified or through indexing and citation services, such as the Science Citation Index. (c) Checking investigators, granting agencies, and industries or pharmaceutical companies for information on published or unpublished studies. (d) Searching trial registries of pertinent studies, such as The Cochrane Library Databases and the Oxford Database of Perinatal Trials. (e) Journal hand-searching.

In order to identify all possible studies to be included in a MA, not only one but several search strategies should be performed, and computer literature searches should not be the only strategy used to identify studies [20 38 39]. Another question that needs to be answered is if non-English language studies should be included in the analysis [38]. Unfortunately, MAs published in the English language journals restrict their search to papers which were also published in English, although they usually claim a very thorough search to retrieve every paper dealing with the topic to be investigated, which is rather questionable.

The tendency for increased publication rates among studies that show a statistically significant effect of a specific treatment has been already documented [40–42]. Publication bias is correlated to the fact that studies with statistically significant results are more likely to be published than those without statistically significant results [43], and therefore measures should be taken to identify and reduce publication bias.

The inclusion of unpublished data in SRs or MAs is still controversial [20 21 44 45]. However, before conducting a search for a MA, a decision should be made whether unpublished data that have not undergone a formal peer review process, including dissertations or conference presentations, should be included [21 39 42 46].

Finally, publication and selection biases in MAs are more likely to affect small studies, which also tend to be of lower methodological quality. This may lead to “small-study effects,” where the smaller studies in a MA show larger treatment effects. Small-study effects may also arise because of between-trial heterogeneity. Thus, it is generally considered that studies with large sample sizes are less likely to be affected by publication bias than studies with smaller sample sizes, because larger studies are more likely to be reported even when they present negative or not-significant results [38 39].

A very useful graphic method that can be utilized to identify and present possible publication bias in MAs in an association between treatment effect and study size is the “funnel plot” [20 38 39]. This is a scatter plot of the treatment effects calculated from the individual studies on the horizontal axis, against study size or standard error on the vertical axis [47]. Funnel plots take advantage of the well-known “law of large numbers,” i.e., the larger the sample size, the more probable it is that the sample mean is a better estimate of the population mean. This suggests that the precision that results from estimating the underlying treatment effect will increase as the sample size of the component studies increases. Thus, large studies appear at the top of the graph and tend to cluster near the mean effect size. Smaller studies appear towards the bottom of the graph and tend to be dispersed across a range of values.

In the absence of bias the studies are expected to be symmetrically distributed about the pooled effect size (Fig.6.1), and thus visual inspection of a funnel plot may give a clear indication of publication bias. In the presence of bias, the plot’s lower part can be expected to show a higher study concentration on one side of the plot than the other. In case of funnel plot asymmetry, additional statistical tests can be used to further assess the publication bias, such as the Begg & Mazumdar’s rank correlation test [48], the Egger’s test of the intercept [18], the Failsafe N or “file-drawer number” approach [49 50], and the Duval and Tweedie’s “trim and fill” method [51].

Selection bias is a systematic error that results from the way the subjects are selected into the study or because there are selective losses of subjects before data analysis. Selection bias can occur in any kind of epidemiological study [52] and is one of the main reasons for divergent results among MAs, [39]. Thus, there is significant need to define the specific inclusion and exclusion criteria used for the studies to be included in the MA. Another aspect of selection bias is the populations utilized in the papers to be included in the MA. As MAs combine results of many different studies, the populations involved in these studies may be many and diverse, and there is a need to describe these populations and to which the results are to be generalized. The variability in the populations included in the MA may make interpreting the results difficult.

The inclusion and exclusion criteria of the source studies should be as specific as possible in order to be able to compare only compatible and relevant studies of suitable quality. According to Kassirer [21], the studies included in a MA should meet the following criteria: (a) The included studies should test the same hypothesis [46] and should therefore have the same outcome or end point. (b) These studies should compare similar patients or similar interventions [38 46]. For example, a study testing a drug against a placebo should not be compared with one that tests it against a competing drug. (c) The source studies should present some characteristics of scientific quality, such as adequate sample size, random assignment between treatment and control groups, masking of patients and examiners, quality controls on data collection and management, and finally formal statistical analyses. At least two reviewers should individually perform the selection of studies to be included in the analysis in a blind procedure, and then the inter-reviewer agreement should be objectively evaluated (i.e., by calculating the Cohen’s kappa).

Currently the term “risk of bias” assessment is widely used for evaluating each included study in a SR or a MA, while formerly, the authors tended to use the term “quality” analysis. When carrying out a SR or a MA it is important to distinguish between ­quality and risk of bias and to focus on evaluating and reporting the latter. Assessing the risk of bias should be part of the conduct and reporting of any SR or a MA. In all situations, systematic reviewers are encouraged to think ahead carefully about what risks of bias (methodological and clinical) may have a bearing on the results of their SRs. There are three main ways to assess risk of bias: (a) scales, (b) checklists, (c) and individual component approaches. Scales that are commonly used for the assessment of risk of bias include: the Downs-Blacks scale [53], the Jadad scale [54], and the Newcastle-Ottawa Scale (NOS) as suggested by the Cochrane Handbook for Systematic Reviews of Interventions [55]. Checklists are less frequently used and potentially have the same problems as scales. The most commonly used is the PRISMA checklist [56]. The use of a component approach and one that is based on domains for which there is good empirical evidence and perhaps strong clinical grounds is advocated. The new Cochrane Risk of Bias tool is one such component approach [57].

Characterization of the quality of the studies allows researchers to use more specific inclusion or exclusion criteria and sometimes to assign weights to studies of different quality when combining them in the analysis. At least two reviewers should individually evaluate the quality of each study in a blind procedure, and then the differences among them should be objectively evaluated [38 39] (i.e., by calculating the Cohen’s kappa).

The criteria used to extract data from various studies included in the MA should be reported and must be specified in advance. In other words, the “who, what, and how” of the extraction process should be precisely described. When the extraction criteria are too general, data extraction could be a subjective procedure, which may introduce bias into the process. Similar to quality assessment of the source studies, data extraction should be performed by two different researchers, and the inter-examiner reliability to establish the consistency of extraction should be evaluated by comparing the results for agreement (i.e., by calculating the Cohen’s kappa).

The effect of statistically pooling the results through a meta-analytic procedure is to increase the sample size. Consequently, the statistical power of the MA is stronger than that of the individual studies. Differences in statistical methods can result in different results, and therefore careful attention should be directed to choosing the statistical procedures for conducting MAs.

There are some basic differences in the statistical procedures used in the source studies and those used in MAs. The most significant difference is the unit of analysis, which in the individual source studies is the subject (e.g., patients, students, and observational entities), while in a MA, it is the results of a study.

Another important difference is the application of appropriate statistical techniques. Some meta-analysts simply apply the usual techniques used for primary analysis also in MAs [58]. As the hypothesis tested in a MA is based on differing sample sizes, these statistical tests will have different sampling variances and will probably violate the assumption of the homogeneity of these variances. Therefore, statistical procedures that are specially developed for MAs seem to be more appropriate for the quantification of the results of the individual studies [59–62].

Despite the existence of these specially designed procedures, some simple statistical methods for combining the results of individual source studies include various counting or summation procedures, such as the “head counting” or “vote counting”approach [32 35 63 64] or simply combining significance tests of the source studies [35]. Researchers should be very careful in choosing these simple procedures for statistical analysis, which are actually not recommended. In the first procedure, the highest number of positive or negative studies determines the results of the analysis. In the second procedure, the results in a MA are summarized by mathematically combining thep-values of each of the studies into a singlep-value. The latter approach is thus based entirely onp-values and because, as mentioned above, studies with nonsignificant results are published less often, this method will be associated with publication bias.