We read a recently published study by Lee et al in the October, 2019, issue of the Journal (Lee RJ, Ko J, Park J, Pi S, Devgon D, Nelson G, et al. Accuracy and reliability of the expected root position setup on clinical decision making of root position at midtreatment. Am J Orthod Dentofacial Orthop 2019;156:566-73). The authors aimed to study the accuracy and reliability of clinical decisions made on root position using the expected root position (ERP) setup. Four examiners assessed both the cone-beam computed tomography (CBCT) scan and ERP setup and made clinical decisions regarding the root position with each method. Cohen’s kappa was determined to assess intraoperator and intermethod reliability. Sensitivity, specificity, positive predictive value, and negative predictive value were calculated to determine the accuracy of the ERP setup. They showed that the kappa values for intraoperator reliability for both the CBCT scan and ERP setup fell within the 0.61-0.80 range. The kappa values for intermethod reliability between the CBCT scan and ERP setup fell within the 0.61-0.80 range for all tooth groups.

Although this article has provided valuable information, reconsidering some substantial points might help the clarity of the method and an accurate interpretation of the study. ^, Regarding reliability, for qualitative variables with more than 2 categories, using simple kappa is a common mistake because kappa has its own limitations. First, it depends on the prevalence in each category. Second, it also depends on the number of categories. Finally, the physician’s opinion and the patient’s clinical status should not be neglected. The weighted kappa is a more appropriate statistical test and should have been used to assess the reliability in this paper.

For assessing reliability for a qualitative variable, relying on the kappa coefficient can lead to a misleading message.