1

Introduction

Recent years has seen an increase in publication of systematic reviews (SRs) and meta-analyses in biomedical research as they are considered by healthcare professionals the most effective and transparent approach to provide a synthesis of research and clinical evidence and guide decision making in practice . Notwithstanding this, these have attracted criticism both related to excessive publication of overlapping and barren reviews, as well as concerns in relation to the quality and yield associated with many of these .

A meta-analysis uses a mathematical approach to combine the results from individual studies with regard to a research question of interest in order to obtain an overall estimate of the treatment effect . However, due to the fact that individual studies may differ in terms of settings, populations, interventions, outcomes or design, it may be possible that this lack of homogeneity or heterogeneity may render the findings inconsistent. Inconsistency refers to the presence of variation across the results of individual studies greater than that anticipated by chance . Presence of unexplained inconsistency is an important reason for downgrading the quality of the evidence from meta-analyses .

Assessment of heterogeneity involves both clinical and statistical aspects. Clinical heterogeneity refers to a priori identification of differences in population settings, interventions or variations in treatment outcomes and may be detected upon examination of the eligibility criteria for the population under study . Statistical heterogeneity can be inspected either visually or through statistical tests. Visual inspection involves examination of the forest plots and assessment of variation in the point estimates of the included studies as well as the degree of overlap of the associated confidence intervals . Statistical heterogeneity may be confirmed through statistical tests, the most known of which are the Q-statistic also referred to as the chi-squared test and the I ² statistic . A statistically significant chi-squared test (i.e. p < 0.1) may indicate important variation in individual study estimates; however, false negative results cannot be excluded especially when few studies are included in the meta-analysis . The I ² statistic is considered a more straightforward measure to quantify heterogeneity describing the percentage variability in the point estimates that are attributed to heterogeneity due to between study differences .

The Cochrane Collaboration has suggested the use of four categories when interpreting heterogeneity based on I ² as follows: 0–40% unimportant heterogeneity, 30–60% moderate, 50–90% substantial and 75–100% considerable heterogeneity. The degree of overlap between these categories represents the subjectivity in interpreting the magnitude of heterogeneity . Empirical evidence based on Medline and Cochrane databases has indicated that moderate to substantial statistical heterogeneity has been identified in up to 39% of meta-analyses in medicine, depending on the type of outcome assessed, while I ² has been shown to increase as the number of studies included in the meta-analysis increases for continuous outcomes .

The degree of statistical heterogeneity might constitute an indicator of inappropriate synthesis of conceptually different studies. Statistical heterogeneity may arise as a result of clinical or methodological differences between studies. This pertains to differences related to the participants, interventions and outcomes, or differences in study design or risk of bias within studies. Where the variation in treatment effect is greater than that anticipated to arise due to chance, the presence of increased levels of heterogeneity is suspected . This in turn may compromise conclusions drawn from meta-analysis of heterogeneous studies, with attendant effects on the translation of the findings to clinical practice in terms of clinical decision making, or the generalizability of the results . As such, careful consideration of study characteristics prior to possible inclusion within a planned meta-analysis is important with clear delineation of eligibility criteria based ideally on a pre-planned protocol.

To our knowledge, there has been no similar attempt to quantify the magnitude of heterogeneity within meta-analyses in dentistry. The primary objective of the present study was therefore to record and quantify the extent of statistical heterogeneity (based on I-squared; I ² ) in Cochrane and non- Cochrane meta-analyses in the major dental specialty of orthodontics. In addition, we aimed to identify possible associations between heterogeneity and a number of study characteristics including year of publication, funding, type and significance of outcome, type of mathematical analysis followed and number of studies included within the syntheses.

2

Materials and methods

The contents of five major orthodontic journals, namely the American Journal of Orthodontics and Dentofacial Orthopedics (AJODO), Angle Orthodontist (Angle), European Journal of Orthodontics (EJO), Journal of Orthodontics (JO), and Orthodontics and Craniofacial Research (OCR), as well as orthodontic topics covered in the Cochrane Database of Systematic Reviews (CDSR) were electronically searched from January 2000 to December 2017. The selection of journals was based on the relative frequency of systematic reviews (SRs) published within them. On initial screening, articles were considered potentially eligible if they included the terms “systematic review” or “meta-analysis” in the title or the abstract. Relevant articles (SRs) were finally included after full text evaluation, when at least one mathematical synthesis was undertaken with associated report of the degree of heterogeneity based on I ² statistic.

The selection of articles was carried out by two authors (DK, IM) and data was extracted on pre-specified standardized forms by one author (DK) covering the following information:

• –

  Article level: journal, type of review (interventional/epidemiologic), continent of authorship, number of authors, year of publication, involvement of a statistician/methodologist, number of syntheses per study and funding.

• –

  Meta-analysis (synthesis) level: magnitude of heterogeneity (I ² %), point estimate along with confidence interval (95% CIs), design of included studies, number of studies within syntheses, type of outcome (binary/continuous), type of analysis (random effects, fixed effects), significance of meta-analysis outcome (significant, non-significant). Standard errors (SE) were calculated for each summary outcome from point estimates and 95%CIs. Heterogeneity (I ² %) was further classified to four non-overlapping categories: 0%, 1–29%, 30–59%, 60–100% based on previous research . Consultation with one author (NP) was performed when uncertainty existed.

Descriptive statistics in relation to the characteristics of the included studies were undertaken at review level and based on publication type (Cochrane, non-Cochrane). Cross-tabulations and associations between level of heterogeneity and synthesis type were applied. Chi-squared and Fisher’s exact tests were performed where appropriate. Furthermore, a scatterplot was used to graphically present the relationship between number of studies included within the synthesis and I ² . Mixed effects univariable and multivariable ordinal logistic regression was used to identify significant predictors for the presence of statistical heterogeneity (0%, 1–29%, 30–59%, 60–100%) both at review level (including review type, involvement of a statistician, number of syntheses) and at synthesis level (including number of studies within syntheses, type of outcome, type of analysis). The random effects parameter was the variable study (ie, SR), which represents the between cluster variability, as more than one meta-analyses within a SR were considered. Non-significant predictors at p = 0.10 in the univariable analysis were excluded from the multivariable model. The presence of significant interaction between type of outcome (binary/continuous) and type of analysis (random/fixed effect) was also assessed. The level of statistical significance was pre-specified at p < 0.05. Statistical analyses were performed with STATA version 15.1 software (Stata Corporation, College Station, Tex, USA).

2

Materials and methods

The contents of five major orthodontic journals, namely the American Journal of Orthodontics and Dentofacial Orthopedics (AJODO), Angle Orthodontist (Angle), European Journal of Orthodontics (EJO), Journal of Orthodontics (JO), and Orthodontics and Craniofacial Research (OCR), as well as orthodontic topics covered in the Cochrane Database of Systematic Reviews (CDSR) were electronically searched from January 2000 to December 2017. The selection of journals was based on the relative frequency of systematic reviews (SRs) published within them. On initial screening, articles were considered potentially eligible if they included the terms “systematic review” or “meta-analysis” in the title or the abstract. Relevant articles (SRs) were finally included after full text evaluation, when at least one mathematical synthesis was undertaken with associated report of the degree of heterogeneity based on I ² statistic.

The selection of articles was carried out by two authors (DK, IM) and data was extracted on pre-specified standardized forms by one author (DK) covering the following information:

• –

  Article level: journal, type of review (interventional/epidemiologic), continent of authorship, number of authors, year of publication, involvement of a statistician/methodologist, number of syntheses per study and funding.

• –

  Meta-analysis (synthesis) level: magnitude of heterogeneity (I ² %), point estimate along with confidence interval (95% CIs), design of included studies, number of studies within syntheses, type of outcome (binary/continuous), type of analysis (random effects, fixed effects), significance of meta-analysis outcome (significant, non-significant). Standard errors (SE) were calculated for each summary outcome from point estimates and 95%CIs. Heterogeneity (I ² %) was further classified to four non-overlapping categories: 0%, 1–29%, 30–59%, 60–100% based on previous research . Consultation with one author (NP) was performed when uncertainty existed.

Descriptive statistics in relation to the characteristics of the included studies were undertaken at review level and based on publication type (Cochrane, non-Cochrane). Cross-tabulations and associations between level of heterogeneity and synthesis type were applied. Chi-squared and Fisher’s exact tests were performed where appropriate. Furthermore, a scatterplot was used to graphically present the relationship between number of studies included within the synthesis and I ² . Mixed effects univariable and multivariable ordinal logistic regression was used to identify significant predictors for the presence of statistical heterogeneity (0%, 1–29%, 30–59%, 60–100%) both at review level (including review type, involvement of a statistician, number of syntheses) and at synthesis level (including number of studies within syntheses, type of outcome, type of analysis). The random effects parameter was the variable study (ie, SR), which represents the between cluster variability, as more than one meta-analyses within a SR were considered. Non-significant predictors at p = 0.10 in the univariable analysis were excluded from the multivariable model. The presence of significant interaction between type of outcome (binary/continuous) and type of analysis (random/fixed effect) was also assessed. The level of statistical significance was pre-specified at p < 0.05. Statistical analyses were performed with STATA version 15.1 software (Stata Corporation, College Station, Tex, USA).

3

Results

A total of 84 systematic reviews including at least one meta-analysis were initially identified, while 72 remained after applying the eligibility criteria ( Fig. 1 ). A quarter of studies were Cochrane reviews (n = 18, 25%), with 54 being published in orthodontic journals (75%). The majority of the reviews were authored by European investigators (45/72, 63%) and were co-authored by 4–6 researchers (44/72, 61%). Cochrane reviews were more likely to involve a methodologist or statistician than non-Cochrane (18/18, 100%, p < 0.001), and also to receive funding (16/18, 89%).

Flow diagram of systematic review selection.

Overall, most reviews included 3–5 meta-analyses (n = 24, 33%), while there was a considerable increase in the publication of systematic reviews with at least one synthesis in recent years i.e. 2014–2017 (n = 49, 68%) ( Table 1 , Fig. 2 ). The mean review heterogeneity (I ² %, recorded as the average heterogeneity across all syntheses within a review) indicated moderate to considerable I ² values both within interventional (45/60, 75%) and epidemiologic (10/12, 84%) reviews. A similar finding was confirmed for both Cochrane (11/18, 61%) and non-Cochrane reviews (44/54, 81%) ( Table 2 ).

Table 1

Distribution of study characteristics by journal type (n = 72).

	Non-Cochrane	Cochrane	Total	p-value
	N (%)	N (%)
Continent				0.10 a
America	6 (85)	1 (15)	7 (100)
Europe	30 (67)	15 (33)	45 (100)
Asio/other	18 (90)	2 (10)	20 (100)
No. of authors				0.26 a
1–3	16 (89)	2 (11)	18 (100)
4–6	30 (68)	14 (32)	44 (100)
0ver 6	8 (80)	2 (10)	10 (100)
Statistician Involved				<0.001 a
No	51 (100)	0 (0)	51 (100)
Yes	3 (14)	18 (86)	21 (100)
Year of Publication				0.23 a
2000–2010	10 (63)	6 (37)	16 (100)
2011–2013	15 (79)	4 (21)	19 (100)
2014–2017	41 (84)	8 (16)	49 (100)
Number of syntheses per review				0.72 a
1	11 (73)	4 (27)	15 (100)
2	9 (64)	5 (36)	14 (100)
3–5	19 (79)	5 (21)	24 (100)
6–10	8 (89)	1 (11)	9 (100)
Over 10	7 (70)	3 (30)	10 (100)
Funding				<0.001 a
No	45 (96)	2 (4)	47 (100)
Yes	9 (36)	16 (64)	25 (100)
Total	54 (75)	18 (25)	72 (100)

Only gold members can continue reading. Log In or Register to continue

Share this:

• Click to share on Twitter (Opens in new window)
• Click to share on Facebook (Opens in new window)

Related

Related posts:

1. Do traditional techniques produce better conventional complete dentures than simplified techniques? A 10-year follow-up of a randomized clinical trial
2. Efficacy of sealing occlusal caries with a flowable composite in primary molars: A 2-year randomized controlled clinical trial
3. Quality improvement in randomized controlled trial abstracts in prosthodontics since the publication of CONSORT guideline for abstracts: A systematic review
4. Water-associated attributes in the contemporary dentin bonding milieu
5. Are there associations between sleep bruxism, chronic stress, and sleep quality?
6. Trend-analysis of dental hard-tissue conditions as function of tooth age

Stay updated, free dental videos. Join our Telegram channel

Join