Introduction
Studies of computer-aided learning (CAL) in orthodontics have documented both objective and subjective outcomes; however, to date, no studies have attempted to correlate these 2 outcome measures.
Methods
The main objective outcome measured was performance on a written test covering material in the orthodontic diagnosis electronic tutorial (ODET) administered to 92 fourth-year undergraduate dental students. The main subjective outcome measured was a 12-statement questionnaire to elicit students’ perception of the ODET and CAL as teaching modalities.
Results
In the male and female subgroups, a statistically significant difference in mean lecture test scores favoring women (72.46%) over men (67.08%) was observed ( P = 0.05). This difference was not observed for mean ODET test scores ( P = 0.52). Although responses to the questionnaire were mostly positive, the students are not prepared to replace lectures with CAL tutorials. Responses showed that male students preferred self-instruction as a mode of learning more than female students did ( P = 0.05). When linking objective and subjective outcomes, the mean ODET test score had a statistically significant ( P = 0.025), but weak, positive correlation (r = 0.243) with self-reported time spent reviewing the ODET but not with any other statement in the questionnaire.
Conclusions
Despite a difference in lecture test scores between male and female students, there was no difference in mean ODET test scores between the 2 subgroups. This might be explained by sex differences: male students preferred self-instruction more than female students did. Improved performance on the ODET test was noted for students who reported longer times spent reviewing the tutorial. Because students are not prepared to replace lectures with CAL tutorials, from their perspective, the ODET should continue to be used with traditional modes of learning.
Computer-aided learning (CAL) in dental education has become increasingly popular. By 2006, approximately 25% of North American dental schools had already implemented mandatory laptop programs for students. CAL might no longer be considered a novel method for learning but a necessary tool in the arsenal of education.
A CAL program that is at least as effective as other methods of learning has several potential value-added advantages (depending on how the program is designed and the students’ ease of access to the CAL modules). Using a well designed CAL system, a student can learn at his or her own pace, review lessons numerous times, and access the system in a convenient, distraction-free environment.
A systematic review of CAL in orthodontics concluded that there is insufficient evidence to support the complete replacement of conventional teaching with CAL programs, supporting the notion that CAL should be used as an adjunct to conventional teaching or as a mode of self-instruction. If CAL can elicit positive responses from students, it might motivate them to learn the orthodontic curriculum. With the documented shortage of orthodontic faculty in North America, used effectively, CAL could help overcome diminishing resources.
Review of the literature
By following the same protocol outlined in our previous systematic review of CAL in orthodontics and updating a search of the literature to include articles up to the fourth week of August 2007, the objectives of this review were to build on our previously published systematic review and provide contemporary information regarding CAL in orthodontics. In this study, the updated review was not limited to the analysis of randomized controlled trials. Rigid quantitative scientific methods of controlled experimentation are not always valid in an environment such as a medical or dental school class, where the variables in the subject pool are almost as great as or greater than the subject pool itself.
It is most effective when evaluating the 10 studies included in this review to separate the quantitative and objective findings from the qualitative and subjective findings ( Table I ). As summarized in Table I , the following conclusions can be made.
- 1.
Studies of CAL in orthodontics that met our inclusion criteria were split between no difference (4 of 7 studies), significant advantage of CAL over conventional teaching (2 studies), and significant advantage of conventional tutorial over CAL in terms of knowledge gained (1 study).
- 2.
Studies measuring the amount of time spent on CAL by students showed no correlation between time spent using the computer program and improvements in test scores.
- 3.
CAL programs in orthodontics, for the most part (5 of the 7 studies), elicit positive responses and attitudes from students toward learning.
- 4.
The most complete evaluation studies of CAL as a teaching tool include both objective and subjective outcome measures, allowing for the evaluation of both effectiveness of the study and the student’s satisfaction with the learning program. These findings might contribute to student motivation.
- 5.
There is insufficient evidence for the replacement of conventional teaching with CAL programs in orthodontics (only 2 studies). CAL is better used as an adjunct to conventional teaching or a mode of self-instruction in orthodontics.
- 6.
CAL’s advantage is that it can present and preserve orthodontic study models, radiographs, and other visual diagnostic aids. Students can learn at an individualized pace when they are most alert. CAL allows immediate feedback on the presented material.
- 7.
No studies have evaluated the correlation between subjective and objective outcome measures. This type of evaluation might provide insight regarding the role of motivation and learning style preference in modulating the effectiveness of CAL in orthodontics.
| Quantitative/objective findings | Qualitative/subjective findings | |||||
|---|---|---|---|---|---|---|
| Study | Subject | Favor CAL | No difference | Favor traditional learning | Favor CAL | Mixed responses |
| Clark et al (1997) | Cephalometric measurements | X | X | |||
| Hobson et al (1998) | Diagnosis and treatment planning | X | X | |||
| Irvine and Moore (1986) | Mixed dentition analysis | X | X | |||
| Turner and Weerakone (1993) | Cephalometric measurements | X | X | |||
| Aly et al (2004) | Orthodontic undergraduate curriculum | X | NA | |||
| Lowe et al (2001) | IOTN | X | NA | |||
| Luffingham (1984) | Orthodontic principles | X | NA | |||
| Aly et al (2003) | Orthodontic diagnosis and treatment | NA | X | |||
| Nurko and Proffit (2005) | Advanced clinical orthodontics | NA | X | |||
| Stephens and Dowell (1983) | Case assessment and treatment planning | NA | X | |||
| Total, 10 studies | 2/7 | 4/7 | 1/7 | 5/7 | 2/7 | |
We found that evaluation studies of CAL in dental education focused on the measurement of objective outcomes such as test performance, with up to 17 well-designed randomized controlled trials available as of September 2007. Several of these studies also documented subjective outcome measures in the form of students’ responses to questionnaires eliciting their perceptions of the CAL program. There does not appear to be a study that correlates objective with subjective outcomes to elucidate a possible link between students’ intrinsic feelings and motivations toward the educational unit and their performance in formal evaluations.
The orthodontic diagnosis electronic tutorial (ODET), developed and used as part of the undergraduate orthodontic curriculum at McGill University in Montreal, Quebec, Canada, is being considered for similar use at the University of Toronto. As part of the formal evaluation before its implementation, an evaluation of the program through objective and subjective measurement tools was undertaken with additional insight into the undergraduate students’ attitudes and perceptions toward the ODET and CAL.
The objectives of this study were (1) to elicit and assess students’ perceptions of the ODET; (2) to assess whether there is a correlation between objective (test scores) outcome measures and subjective (questionnaire responses) outcome measures; and (3) to assess whether there are any differences in outcomes (test scores and responses to the questionnaire) between the subgroups of male vs female, and doctor of dental surgery (DDS) vs international dentist advanced placement program (IDAPP) undergraduate dental students.
Material and methods
Ninety-two fourth-year undergraduate dental students (38 men, 54 women) participated in this study. The fourth-year class (designated 0T8 for the year of graduation) consisted of 67 DDS students and 25 IDAPP students.
The ODET was developed to give undergraduate dental students the skills to recognize, quantify, and properly diagnose a developing malocclusion. It was designed to introduce students to the practice of orthodontics as it applies to general dentistry and currently forms the basis for the orthodontic curriculum taught at McGill University. The objectives of the tutorial are for students to be able to recognize orthodontic patient problems including developing malocclusions, recognize nonorthodontic patient problems, identify the differences between standard norms and deviations in development, effectively diagnose a malocclusion using various diagnostic tools, and identify and apply the appropriate diagnostic tools using the outside-in approach (extraoral and soft-tissue findings are considered before intraoral and radiographic findings) to patient screening.
The electronic tutorial is focused on the orthodontic examination comprising facial, functional, and dental examinations; cephalometric and panoramic analyses; and classifications of malocclusions. The content of each section is described through text and figures. Words explaining key concepts are highlighted in blue and act as hyperlinks for animation and other visual teaching tools that appear on the screen ( Fig 1 ). At the end of each section, students can evaluate their understanding of the material by completing exercises providing immediate feedback. As they progress through the exercises, the students can easily return to the content sections by using a drop-down menu on the upper left side of the screen. Navigation through the tutorial can be done nonlinearly, giving the user full control of the learning (learner-controlled program). Students can rely on their personal learning preferences without having to comply with the structure of the electronic tutorial.
Objective outcome measures
The orthodontic lecture term test (OLTT), developed by the undergraduate orthodontic course coordinator (J.P.), includes several short-answer questions to assess the students’ ability to diagnose and analyze an orthodontic patient by answering several knowledge and application questions from lecture material. An answer key for the test was prepared before administration of the test and used to mark all tests. If an answer was provided by students that could have been correct but was not included in the answer key, consensus was reached between the 2 principal researchers (H.R. and J.P.) regarding its inclusion in the answer key and considered when marking all tests.
The ODET was developed by the undergraduate coordinator (J.P.) and the principal researcher (H.R.); this test matched the lecture test in level of difficulty while encompassing the topics included in the ODET. Questions in the test can be categorized as “knowledge” questions (testing students’ familiarity and understanding of the material) and “application” questions (testing students’ ability to apply concepts presented in the ODET). Both the ODET test and OLTT were administered one after the other. The assessment was administered in a manner similar to that of the OLTT. Questions were discarded if they yielded a successful response percentage of less than 25% for the entire class for both tests, eliminating any potential source of ambiguity.
Subjective outcome measures
A 12-statement questionnaire adapted from a previous study was prepared to assess dental students’ perceptions and impressions of the electronic tutorial ( Fig 2 ). Response options for the statements were “strongly disagree,” “disagree,” “uncertain,” “agree,” or “strongly agree.” The 92 students responded using the following scoring for the Likert scale: strongly agree (5), agree (4), uncertain (3), disagree (2), and strongly disagree (1). Statements in the questionnaire were meant to reflect students’ perception of the (1) usability and acceptability of the tutorial (statements 1 and 2), (2) educational quality (statements 3-5), (3) acceptability of CAL (statements 6 and 7), and (4) students’ learning style preferences (ie, preference of CAL over traditional learning methods; statements 8-10). The last 2 statements were meant to assess students’ motivation to use the electronic tutorial (statement 11) and students’ self-reported time spent on the tutorial (statement 12). For statement 12, students were asked to estimate the amount of time that they spent reviewing the tutorial by circling one of the following: <1 hour, 1-3 hours, 4-6 hours, 7-10 hours, or >10 hours. A higher frequency of responses agreeing with statements 1 through 11 indicated positive student perceptions of the ODET and CAL. Students were invited to provide input regarding features of the electronic tutorial that they liked, disliked, or would improve. Once completed, questionnaires were coded to be able to correlate the responses (subjective outcome measures) with ODET and OLTT scores (objective measures).
Statistical analysis
All statistical analyses were performed by using SPSS for Windows (version 15.1, SPSS, Chicago, Ill). Differences in OLTT vs ODET mean test scores were analyzed by using paired-samples t tests. Differences in test scores for the various subgroups were analyzed by using independent-samples t tests and univariate analysis of variance (ANOVA). Statements in the questionnaire regarding learning style preferences (questions 8-10) were trichotomized into “disagree,” “uncertain,” and “agree.” Differences between male and female responses were assessed by using independent-samples t tests. Correlations between mean ODET test scores and mean level of agreement on the questionnaire for each response were calculated by using the Spearman rank correlation coefficient.
Material and methods
Ninety-two fourth-year undergraduate dental students (38 men, 54 women) participated in this study. The fourth-year class (designated 0T8 for the year of graduation) consisted of 67 DDS students and 25 IDAPP students.
The ODET was developed to give undergraduate dental students the skills to recognize, quantify, and properly diagnose a developing malocclusion. It was designed to introduce students to the practice of orthodontics as it applies to general dentistry and currently forms the basis for the orthodontic curriculum taught at McGill University. The objectives of the tutorial are for students to be able to recognize orthodontic patient problems including developing malocclusions, recognize nonorthodontic patient problems, identify the differences between standard norms and deviations in development, effectively diagnose a malocclusion using various diagnostic tools, and identify and apply the appropriate diagnostic tools using the outside-in approach (extraoral and soft-tissue findings are considered before intraoral and radiographic findings) to patient screening.
The electronic tutorial is focused on the orthodontic examination comprising facial, functional, and dental examinations; cephalometric and panoramic analyses; and classifications of malocclusions. The content of each section is described through text and figures. Words explaining key concepts are highlighted in blue and act as hyperlinks for animation and other visual teaching tools that appear on the screen ( Fig 1 ). At the end of each section, students can evaluate their understanding of the material by completing exercises providing immediate feedback. As they progress through the exercises, the students can easily return to the content sections by using a drop-down menu on the upper left side of the screen. Navigation through the tutorial can be done nonlinearly, giving the user full control of the learning (learner-controlled program). Students can rely on their personal learning preferences without having to comply with the structure of the electronic tutorial.
