Abstract
Parents of children with a cleft lip and/or palate (CL/P) and patients with CL/P actively search for online information about CL/P. The quality of this information has not been sufficiently evaluated. The aim of this study was to define quality criteria for online information about CL/P and assess the quality of frequently accessed websites. Patients, parents, and professionals were equally involved in all stages of this study. A literature review was performed to obtain known quality criteria for online information. These criteria were prioritized by patients, parents, and professionals. The most important criteria were used to rate the quality of four websites on CL/P. Forty-two quality items were extracted from the literature. Patients, parents, and professionals agreed on the importance of 16 of these items. New groups of patients, parents, and professionals assessed four websites on CL/P. Although the groups were like-minded in their overall assessment of the quality of the websites, distinct differences emerged between the groups in relation to certain items. This study shows the importance of patient participation in healthcare research, as well as a feasible approach to do so. Involving patients in composing online health information will set different priorities, which is necessary in establishing high quality information.
The Internet has become a valuable source of health information. For example, 59% of European citizens used the Internet to search health information in 2014, and similar numbers have been reported for the USA. As a consequence, physicians are increasingly facing patients who use the Internet in general and also more specific online forums to seek medical information. Unfortunately, the quality of online patient information varies widely, although the variation is similar to that seen in other media.
Professionals might be reluctant regarding the Internet-acquired knowledge that patients bring to the clinical encounter. Since the quality of this online information varies widely, it may cause a feeling of professional devaluation on the one hand, it may challenge the professional on the other hand, and in the end it could make the consultation less time-efficient. However, patients expect their physicians to acknowledge the information obtained online that they bring to the consultation, and want the professionals to discuss, explain, and contextualize it. Such an interaction can improve the physician–patient relationship.
To maximize the benefits of online information, the quality of the information is key. There are numerous quality assessment checklists that define the quality of online information, such as the DISCERN quality criteria, the JAMA ( Journal of the American Medical Association ) benchmarks, and the Health on the Net Foundation (HON) principles. Remarkably, patient perspectives on the definition of quality did not form part of the developmental process for these checklists. Since patients are the main users of online patient information, it seems wrong to consider only the professionals’ perspective.
The first aim of this study was to define quality criteria for online information on cleft lip and/or palate (CL/P). A systematic process of developing and validating a quality assessment checklist for online patient information about CL/P was applied, in which patients with CL/P and the parents of children with CL/P, as well as professionals in CL/P care, were equally involved in all phases. Secondly, the quality of online patient information on CL/P was assessed. Finally, potential differences in perspective on the quality of online CL/P information between the three stakeholders were mapped and the value of involving patients and parents in this study was evaluated.
Materials and methods
This study consisted of four phases ( Fig. 1 ). Three groups of participants were involved in all phases: patients with CL/P (>16 years of age), parents of children with CL/P, and professionals in cleft care from different specialties (ENT surgeons, geneticists, maxillofacial surgeons, orthodontists, plastic surgeons, psychologists, specialist nurses, and speech therapists).
Phase I
The aim of phase I was to construct a draft questionnaire with all known quality items for online information. A literature review was performed to determine which quality items on online information are known in the literature. The search was performed in August 2012, in PubMed, using the following search terms: health [tiab] information [tiab] internet [tiab] (quality [tiab] OR assessment [tiab]) NOT “quality of life” [tiab].
Papers were selected by two reviewers in a three-stage approach: title, abstract, and full text. In the case of discrepancies between the reviewers in the selection of full texts, a third reviewer facilitated the discussion until consensus was reached. The inclusion criteria are summarized in Table 1 . Quality items and checklists used, such as HON and DISCERN, were obtained, and the quality items were categorized and listed.
| 1 | Published between 2000 and 2012. |
| 2 | Publication written in English or Dutch. |
| 3 | Publication has to evaluate health-related information (with no restriction on the type of information). |
| 4 | Publication has to evaluate online health-related information, meant for patients. |
| 5 | The quality of information has to be evaluated with a list of quality criteria. This can be a pre-existing and validated instrument or a self-selected list of quality criteria. |
| 6 | The sources and methods used to produce the quality criteria have to be recorded in the paper. |
| 7 | The applied list with quality criteria has to be published in the article. |
| 8 | More than one website has to be evaluated with the applied quality criteria list. |
| 9 | The search strategy for online patient information and reasons why specific websites were evaluated has to be recorded. |
Phase II
The aim of phase II was to construct a final questionnaire with relevant quality items, which the three groups could use to evaluate the quality of online patient information on CL/P in phase IV of the study. Patients, parents, and professionals were asked to prioritize the items in the draft questionnaire constructed in phase I. For this questionnaire, the participants first had to indicate to what extent the quality items were of importance to them when looking for online information about CL/P. A four-point Likert scale was used: very important, important, less important, and not important. Next, the participants were asked to prioritize the quality items by selecting their top five most important quality items.
A summary importance score was calculated per item per group: the percentage top five rankings per item were multiplied by the cumulative percentage of the ‘very important’ and ‘important’ scores per item. An example of two items scored by the professionals and how this leads to the summary importance score is presented in Table 2 .
| Quality item | Importance score % a | Top 5 ranking % b | Summary score | ||||
|---|---|---|---|---|---|---|---|
| Very important 1 | Important 2 | Less important 3 | Not important 4 | Cumulative % 1 + 2 | |||
| No use of complex words | 44.4 | 55.6 | 0 | 0 | 100 | 33.3 | 33.3 |
| No use of medical terms | 22.2 | 33.3 | 44.4 | 0 | 55.5 | 11.1 | 6.6 |
a Importance score %: percentage of professionals who scored the quality items as very important, important, less important, or not important.
b Top 5 ranking %: percentage of professionals who ranked the quality item in their top 5 most important quality items.
Quality items were selected if the summary importance score was ≥5 for all three groups. Items for which at least one of the three groups scored <5 were excluded. Hence, only quality items on which all three groups agreed to be of importance were selected.
Phase III
The aim of phase III was to select four frequently used websites with patient information on CL/P. These websites were used in phase IV to evaluate the quality of the online information.
For the selection of frequently used patient websites, parents and patients were asked to surf the Internet at home for information about CL/P. Respondents were asked to write down the full URL of the websites viewed. Websites were selected for phase IV if they were visited more than once and originated from academic or non-academic hospitals, or other CL/P-related organizations.
Phase IV
The aim of phase IV was to evaluate the quality of four websites with patient information on CL/P. Three new groups of respondents evaluated the four websites selected in phase III. Fifteen patients, 15 parents, and 15 professionals from two different academic Dutch CL/P centres used the questionnaire produced in phase II to evaluate the websites. The respondents marked to what extent they agreed that a website met the different quality items, again on a four-point Likert scale: strongly agree, agree, disagree, and strongly disagree.
Two comparisons were made: (1) A comparison of the results between websites by group: the cumulative frequency agreement was calculated by summing the frequencies of scores ‘strongly agree’ and ‘agree’. An overall mean score by website was obtained, allowing for a comparison between websites. This was done separately for the three groups. (2) A comparison of the results by quality item between groups: the distribution across the answer categories was calculated for all items. Agreement was defined as ≥60% of participants agreed (strongly agree + agree) that the websites met a quality item. A χ 2 test analysis was performed to assess the differences in agreement between the three groups. A P -value of <0.01 was considered significant, as correction for multiple testing was performed with 16 quality items. IBM SPSS Statistics version 20.0 software (IBM Corp., Armonk, NY, USA) was used for all analyses.
All patients and parents included in phases II, III, and IV were recruited consecutively from routine outpatient cleft clinics of the university medical centres until the required numbers were reached. For each patient, one of the parents involved was invited to participate in the study.
All professionals included in phases II and IV were invited to participate by for phase II the authors selected professionals from different cleft centres with different specialty backgrounds. Professionals from the two participating academic cleft centres were invited to participate in phase IV.
Patients, parents, and professionals could only participate in one phase of the study. Patients and parents were given a token of gratitude (€15.00) for taking part in phases II, III, and IV of the study. This was funded by research funds of the Department of Oral and Maxillofacial Surgery of Radboud University Medical Centre Nijmegen.
Results
Phase I
The literature search resulted in 1096 articles, which were screened on title. Subsequently 449 articles were screened on abstract and 307 screened on full text. Finally 108 articles were included. Forty-two quality items were extracted from these articles, which were divided into 14 categories in accordance with the categories found in the literature ( Table 3 ). The most used quality evaluation tools were HON, DISCERN, LIDA, and JAMA, as well as readability indexes.
| Category | Quality item |
|---|---|
| Authoritative | Origin of the information on the website: the source is a private clinic |
| Origin of the information on the website: the source is a community-based hospital | |
| Origin of the information on the website: the source is a non-profit website, such as Wikipedia | |
| Origin of the information on the website: the source is a news site | |
| Origin of the information on the website: the source is a blog/discussion forum of peers | |
| Origin of the information on the website: the source is a university medical centre a | |
| Latest update | The website provides information about the frequency of updates |
| The website provides the most recent date of update of the website | |
| Design | Consistent use of colours and logos on the website |
| The website has a clear structure a | |
| The website has a logical organization | |
| The website has a professional look and feel | |
| Financial disclosure | Transparency on the website about sponsorship |
| The website provides the names of non-commercial organizations supporting the website | |
| Advertising | The website states clearly that advertising is a source of funding |
| A description of the advertising policy is provided on the website | |
| The website clearly distinguishes between editorial and advertising content a | |
| Usability | The amount of information found on the website a |
| It is easy to find the required information on the website a | |
| The type of information that can be found on the website a | |
| The website contains an internal search engine | |
| Justifiability | Any claims on the website relating to the benefits of treatments, commercial products, or services are supported by evidence a |
| The website provides evidence about the benefits of a specific treatment, commercial product, or service a , b | |
| Readability | No use of complex words on the website a |
| No use of too many words per sentence on the website | |
| No use of medical terms on the website a , b | |
| Reliability | A statement is included on the website whenever medical/health advice is offered by medical professionals a |
| Provision of contact information of the author/editor of the website | |
| The website identifies who produced the website | |
| The website identifies the background of the author | |
| Complementary | A statement is included on the website that the information provided is only meant to complement and not replace any doctor–patient encounter a |
| Attribution | Sources of information are clearly labelled on the website a , b |
| The website cites the source and dates of medical information, where possible includes specific HTML links a , b | |
| Accessibility | The website is easily accessible a |
| The website mentions contact information | |
| The information is accessible without the need to download software | |
| Interactivity | The website provides the opportunity to leave comments |
| The website has a forum | |
| Privacy | The website displays a statement that the producers adhere to privacy legislation a |
| The website describes the privacy policy | |
| The website informs visitors whether or not their data will be stored in a database | |
| The website reports who can access the database containing the logging information of visitors |
Phase II
In this phase of the study, 25 patients, 27 parents, and nine professionals participated, with participation rates of 76%, 84%, and 90%, respectively.
Consensus between the three groups was reached on 31 out of the 42 quality criteria; all three groups prioritized 16 items as being important for the quality of online information ( Table 3 and Fig. 2 ). The five quality items with the highest overall summary scores were: (1) sources of information are clearly labelled on the website; (2) the website is easily accessible; (3) origin of the information on the website: the source is a university medical centre; (4) it is easy to find the required information on the website; and (5) any claims on the website relating to the benefits of treatments, commercial products, or services are supported by evidence.
