Introduction
The importance that prospective patients place on practice characteristics when choosing an orthodontic practice has not been extensively reported. The objective of this research was to develop a valid and reliable questionnaire to address the relative importance of orthodontic office and doctor characteristics for prospective patients or parents of child patients during the initial orthodontic office consultation.
Methods
An initial questionnaire, based on published literature, was field-tested on 16 subjects to assess its validity. Based on the field test, the questionnaire was modified and tested for reliability by using a test-retest method. The questionnaire covered the following areas: doctor, office, staff, and finances. The reliability study included 2 groups of subjects: 12 consecutive prospective adult patients and 41 consecutive parents of prospective child patients. The questionnaires consisted of 43 and 50 questions for the adult patients and the parents of patients, respectively. The subjects rated the importance of practice characteristics in their selection of an orthodontic practice using a 100-mm visual analog scale anchored at “not important at all” and “most important.”
Results
Reliability was analyzed by using the intraclass correlation coefficient (ICC). Summary scores of all 53 subjects showed excellent reliability (ICC, 0.88; range, 0.61-1.0). Summary scores of all 50 questions showed acceptable reliability (ICC, 0.70; range, 0.45-0.88). Twenty-one questions had excellent reliability (ICC, >.75), and 29 questions had fair-to-good reliability (ICC, 0.41-0.75). No questions showed poor reliability (ICC, <0.4).
Conclusions
The pilot study data indicated that the overall reliability of the questionnaire is acceptable.
The relative importance of practice factors to prospective adult patients and the parents of prospective adolescent patients when they are selecting an orthodontic practice has not been extensively reported. About 50% of the parents of adolescent patients have been referred to their orthodontist by their general dentist, and 25% to 30% of parents received recommendations from a friend or relative. However, there are indications that a significant number do not rely solely on the referral. For example, Walley et al reported that 30% of parents visit more than 1 orthodontic office. In addition, Edwards et al reported that, even though patients learn about their orthodontic office from a referral, the main reason they chose their orthodontist was the orthodontist’s caring attitude and good reputation. Therefore, it is important for orthodontists to know what consumers are evaluating when they visit the office for the initial consultation.
Although previous studies presented evidence about several factors that orthodontic patients prefer, there are few quantitative data about the importance that prospective patients and decision makers place on these attributes and what influence they have on their selection of an orthodontist. Most significantly, those studies primarily evaluated patients retrospectively, after they either had chosen an orthodontic practice and were in treatment, or had already completed treatment. Furthermore, there were no reports of validity or reliability of the methods used. Therefore, it is important to reexamine the issue and question prospective patients when they are making their practice choices. Furthermore, several aspects of orthodontic office characteristics such as services, amenities, incorporation of current digital technology, influence of the demeanors of the doctor and staff, and importance of board certification have not been addressed in previous studies. So, it would be desirable to include these factors to determine their importance to prospective adult patients and parental decision makers when they choose an orthodontic practice.
The purpose of this study was to develop a valid and reproducible questionnaire with which to measure the relative importance of office and doctor characteristics to prospective orthodontic patients and parents.
Material and methods
This paper-based survey comprised a series of questions that were administered to prospective adult patients and the parents of prospective child patients visiting an orthodontic office for an examination. They were asked to quantify the importance of factors relative to selecting an orthodontic practice by recording their responses on a 100-mm visual analog scale (VAS). A VAS is used to quantify an attitude or a perception across a range of subjective values and has been shown to be a valid and reliable method of measurement. A VAS is frequently used for assessing a variety of variables including pain, patient preferences, attitudes, and feelings. The subjects—prospective adult patients and the parents of prospective adolescent patients—were asked to quantify how important the attributes of the orthodontist, the office, the staff, and the financial arrangements were to them in selecting an orthodontic office.
The format of the questions was as follows. The subjects were given an office characteristic and asked to place a mark on a 100-mm line that corresponded to the level of importance they placed on that characteristic when selecting an orthodontist. The scale was anchored on the left side (0 mm) with “not important at all” and on the right side (100 mm) with “most important.”
This pilot study consisted of 3 components: the initial development of a questionnaire with an evaluation of questions by 7 laypersons, a field test to assess validity on 16 prospective orthodontic patients, and a reliability test of the questionnaire by using a test-retest method with 53 prospective orthodontic patients.
Validity addresses whether the questionnaire actually measures what it intends to measure. This study implemented methods to assess and improve construct, face, and content validity.
Construct validity was addressed by including factors from previous studies in the questionnaire and comparing the results from the field test to those from the previous studies. For a questionnaire to have face validity, the questions must appear relevant to the subjects whose responses are being assessed. During the field test, the subjects were asked whether the questions were relevant, and their suggestions were solicited to improve the questionnaire. Content validity refers to whether the questionnaire appears relevant to experts in the field. In this study, the questionnaires were generated by a panel of orthodontists both in private practice and with university affiliations.
This study was approved by the Ohio State University Institutional Review Board. Inclusion criteria for the subjects in the field test and the reliability test were the same. Subjects visiting an orthodontic office (18 years or older) who had just received an orthodontic examination for either themselves or their child were invited to participate in the study. All who agreed to participate signed an informed consent form. The subjects were placed in 1 of 2 categories: adults seeking treatment for themselves or parents seeking treatment for their child. Subjects were excluded from participating in the study if they (the adult patient) or their child had previously received orthodontic treatment or were currently in orthodontic treatment. Children did not complete the questionnaire because it is unclear whether they play a central role in selecting the office or the orthodontist. For example, in 1 study, parents reported that only 5% of the children had selected their orthodontist.
The development of the questionnaires was a 4-step process. (1) The relevant published literature on the topic was reviewed, and aspects of orthodontic practices presented in these publications were incorporated into the questionnaire. Incorporating questions when responses from previous studies are known was done to evaluate construct validity. (2) Additional questions were added after discussions with practicing orthodontists who were faculty at The Ohio State University. Incorporating factors that are relevant to orthodontists ensures content validity, since they are considered experts in the field. (3) This draft of the questionnaire was administered to 7 laypersons to discover any problems with readability and comprehension as part of an intelligibility test for face validity. The persons taking the intelligibility test were instructed to write any concerns or suggestions directly on the draft. (4) After minor modifications were made to the draft questionnaire, the revised version was field-tested on prospective orthodontic patients.
The second stage of the pilot study was a field test on prospective patients visiting an orthodontic office for an examination. The field-test study had 16 subjects (11 parents and 5 adults) from 2 private practice offices, 1 located in suburban Columbus, Ohio (8 subjects), and 1 in rural Ohio (8 subjects). Eleven subjects were female, and 5 were male. Six questions that had been found to be important to patients in previous studies were incorporated into the questionnaire to examine construct validity.
Subjects enrolled in the field test were informed verbally that the main objective was to critique the questionnaire so that it could be improved. They were asked to make suggestions by writing directly on the questionnaire. After completing the questionnaire, the subjects participated in a brief interview in which they were asked open-ended questions regarding how the questions were written, the length of the questionnaire, and its relevancy to establish face validity. They were also specifically asked how they would improve the instructions if they had designed the study themselves. Data were collected until a convenience sample of 16 completed the questionnaire. One investigator (B.L.S.) recorded the responses during the interview.
The responses from the field test were compiled, and changes to the draft questionnaire were made by consensus of the panel of orthodontists. The adult questionnaire was revised from 47 to 43 questions. The parent questionnaire was revised from 54 to 50 questions. The modified questionnaires were the versions used in the subsequent reliability study.
Reliability is a measure of the reproducibility of a questionnaire. We used a test-retest method to measure reliability. Although it is customary to wait approximately 2 weeks in the test-retest method, the subjects were instructed to fill out the same questionnaire the next day. This approach was intended to minimize any change in attitude that could occur over time: eg, if a subject visited a different office in the next few days. Because the goal was to assess attitudes based on the subjects’ first impressions when they visited the office, the concern was that a change in attitude might occur if the second questionnaire was administered 2 weeks later because of an intervening experience.
About 8 to 12 minutes were needed to complete each questionnaire. The first questionnaire was completed in the office immediately after an examination by an orthodontist. The subjects were given another copy of the survey and asked to complete and return it by mail in a prepaid envelope the next day.
In the reliability study, there were 53 subjects (41 parents, 12 adults) from 5 private practice offices; 4 were located in suburban Columbus, Ohio, and 1 in rural Ohio. The subjects of the reliability study had not previously participated in the field test but did meet the same inclusion and exclusion criteria. The parent questionnaire contained 50 questions ( Fig 1 ), and the adult questionnaire contained 43 questions ( Fig 2 ). The parent questionnaire contained all of the adult questions plus an additional 7 questions targeted specifically to parents.
Statistical analysis
Reliability was assessed by calculating with software (SAS Institute, Cary, NC) the intraclass correlation coefficient (ICC). Each question and subject were scored independently and separately for adults and parents. Then summary scores were calculated for all 53 subjects and all 50 questions. All data were measured by 1 person and entered into a spreadsheet program (Excel, Microsoft, Redmond, Wash). After the adult and parent questionnaires were analyzed separately, the questions from both questionnaires were matched accordingly and then pooled together for a combined analysis. For this study, an ICC above 0.75 indicated excellent reliability, an ICC between 0.4 and 0.75 indicated fair-to-good reliability, and an ICC below 0.4 indicated poor reliability.
Results
Face validity and content validity were deemed acceptable after the field test because of few comments on wording, relevancy, and comprehension. In addition, the recommendations made by the subjects were used to modify the questionnaire and incorporated into its final form. Six characteristics that had been reported to be important to patients in previous studies were placed in the questionnaire to examine construct validity. The mean VAS scores for the 6 questions evaluating factors identified in previous investigations were relatively high, indicating good agreement with other investigators and good construct validity.
The mean VAS scores from all 54 respondents in this study for the factors identified in previous studies were a doctor with a caring attitude (95), an office that accepts my insurance (91), an office with a good payment plan (89), the doctor’s overall reputation (83), a referral from family or a friend (71), and a referral from my dentist (69).
The number of subjects who participated in this study was sufficient to establish validity and reliability. However, it was not intended to be a sample large or representative enough to allow us to generalize the results to the population at large. Nonetheless, there is some value in presenting a summary of the results.
Both prospective adult patients and parents of prospective adolescent patients rated the doctor’s age, sex, ethnic background, religious beliefs, and use of a white coat among the least important factors in choosing a practice (mean VAS, <40 mm). Both groups also rated a play area for small children and refreshments as less important. Prospective adult patients indicated that television in the waiting area was not an important factor in selecting a practice. The parents of prospective adolescent patients responded that pickup from and return to school, parties for children, and prizes for cooperation were also among the least important factors in selecting a practice.
Both prospective adult patients and the parents of prospective adolescent patients noted (mean VAS, >85 mm) a clean office, an office with the latest technology and equipment, a doctor with a caring attitude, a doctor who gives thorough and easy-to-understand explanations, a treatment plan that makes sense, a doctor who is board certified, the doctor’s overall reputation, and an office with a good payment plan as important factors. Parents of prospective adolescent patients also rated whether the child liked the office and whether the office accepts their insurance as additional important factors in choosing a practice.
The number of subjects who declined to participate was not recorded for the reliability study. There were 7 subjects (3 adults, 4 parents) who completed the first questionnaire but did not return the second questionnaire. Unpaired questionnaires were not included in the statistical analysis. Completion rate for the subjects who agreed to participate and returned both questionnaires was calculated as 56 subjects of a possible 63, or 89%. Three respondents who completed and returned both questionnaires were eliminated from the analysis because of failure to follow directions. In total, 53 pairs of questionnaires were considered valid and complete.
The overall reliability of the questionnaire was acceptable (ICC, 0.70) ( Table I ), based on summary scores for all questions. For the reliability of each question ( Table II ), 21 questions had excellent reliability, 29 questions had fair-to-good reliability, and no question had poor reliability.
