I was interested to read the article by Dindaroğlu et al in the May 2016 issue of the Journal . The authors used 3dMD Flex (3dMD, Atlanta, Ga) to obtain 60 stereophotogrammetric images from the same participant to evaluate the reproducibility of the lip position at rest in 3 dimensions with reverse engineering software and stereophotogrammetric images. Segmentation of the upper and lower lips and the 3-dimensional deviation analysis were performed with Geomagic Control (3D Systems, Rock Hill, SC) software. The Shapiro-Wilk test, paired sample t test, Bland-Altman plots, and Wilcoxon signed rank test were used for statistical analysis.

Reliability (repeatability or reproducibility) was assessed by different statistical tests, such as paired t test; this is a common mistake in reliability analysis. Briefly, for quantitative variable intraclass correlation coefficients and for qualitative variables, weighted kappa should be used with caution because simple kappa has its own limitations.

The authors also reported that 30 images were obtained in 3 sessions on the same day, and the procedure was repeated 6 weeks later for 30 more images. They found no significant difference between the 2 time points; this is another mistake. Regarding reliability, it is crucial to know that an individual-based approach instead of a group-based (mean) approach should be considered. Therefore, intraclass correlation coefficient single measures, instead of average measures, should be reported to correctly assess the reliability. In other words, the possibility of getting no significant difference (mean) of a variable between 2 time points with no reliability is high. Therefore, reporting a Bland-Altman plot and coefficient of variance may also be questionable.

Moreover, reporting significant differences between 2 time points is a completely different methodologic issue from the clinical importance of the mentioned difference. In reliability analysis, depending on sample size, an important clinical difference can be statistically nonsignificant.

The authors concluded that the rest position can be reproduced within a small range, both on the same day and between the sessions. Such a conclusion can be a misleading message.

As a take-home message, for reliability analysis, appropriate tests should be applied by researchers. Otherwise, misdiagnosis and mismanagement cannot be avoided.