Dental training in evidence-based dentistry: A replication study

Background: In the area of education research, it is well-known that studies of a defined question are seldom replicated. Furthermore, e-learning resources with evidence-based content in dentistry have received relatively little attention from researchers.

The Context and Purpose of the Study: The aim of this clinical study was to evaluate how dentistry students from two consecutive cohorts in their first clinical semester rate a long-standing evidence-based dentistry (EbD) resource in an e-learning environment using a questionnaire of 43 specific items on 1) general questions regarding user-friendliness and acceptability, as well as 2) specific questions on content and functional range (A), handling and technical aspects (B), and didactics and educational value (C) based on a Likert scale from 0 = ‘strongly disagree’ to 3 = ‘strongly agree’, and how this compares to a primary study in which the resource was addressed as a novelty. The data were analyzed statistically using a one-way ANOVA followed by a Kruskal-Wallis multiple-comparison Z-test.

Results: A response rate of 100% was achieved. The majority of the users thought the topic of EbD to be important. The e-learning resource was rated with a score of 2.40 ± 0.66 (on a Likert scale from 1-6 where 1 = “very good” and 6 = “insufficient”). 86.15% of the students stated that they consider the resource beneficial for their study in clinical simulation and in patient treatment courses. The results averaged for A: 1.92 (±0.57; median: 1.928), B: 1.48 (±0.60), and C: 2.27 (±0.67). The obtained results in the replication study showed no statistical significant differences to the primary study.

Conclusions: The e-learning resource with dentistry vignettes cases and learning components on evidence-based principles was consistently rated positively by the students. Owing to their agreement with the data of the primary study, the results of the present study point to the remarkable validity of the method of evaluation. This should be addressed in future studies with larger cohorts.

Introduction

In the area of education research, it is well-known that studies of a defined question are seldom replicated [1]. Such replication, however, is extremely valuable in assessing the validity of a method, which may be limited by the well-known Hawthorne effect, for example [2-4]. Even in cases in which randomization is not appropriate to the question replication is indicated as a matter of principle [5].

A recent study analyzed the complete publication history of the current top 100 education journals ranked by 5-year impact factor and found that only 0.13% of education articles were replications [1]. The authors emphasize the “importance of direct replications in helping education research improve its ability to shape education policy and practice” [1]. Such replication studies may be little publicized for two reasons, namely, that the original data from the primary study are not available or that numerous other parameters are intentionally or unintentionally altered in the replication. Concluding their recent evaluation, Gerhardt-Szep and colleagues came to recommend the e-learning environment in an e-learning resource on evidence-based dentistry in a cohort of dental students [6]. A modified method of evaluation formed the basis of this commendation [7,8]. On behalf of the German Association of Medical Informatics, Biometry and Epidemiology (GMDS), the authors have coordinated the development of a catalogue of “Quality Criteria for Electronic Publications in Medicine” (QCEPM) [7]. This catalogue is subdivided into categories including contents, technical aspects, and didactics. According to these items, typical faults and deficiencies of medical electronic software and programmes are elucidated and possible solutions are given. These criteria are intended to support the formative evaluation during the development of each kind of electronic publication, including eLearning medical programmes, and to provide a basis for their summative evaluation [7]. The catalogue is available in German, English, Portuguese and Spanish, at: https://www.imbi.uni-freiburg.de/medinf/gmdsqc/ [7]. A modified method of evaluation has already been applied in a dental context [9, 10]. The reliability values determined here for the individual factors studies were designated “good” with a score of 0.83-0.89 and for the entire questionnaire “excellent” with a score of 0.92. It is important to stress that all the studies which used this method of evaluation addressed without exception newly introduced implementations of e-learning programmes. To what extent the internal validity of each is influenced by, for example, the well-known Hawthorne effect is difficult to predict and thereby represents a serious limitation of the results. The fact that the e-learning resource was newly introduced and that participants may therefore have paid especial regard to it could possibly have played an important role [4]. To reduce the impact of this limitation and thereby make the results of the evaluation more generalizable, it would be sensible to re-evaluate the resource at a later date, for example, at one year after the initial implementation. Following from this, the focus of the present identical replication study was the following: how do dental students receive and respond to the as yet little researched teaching of EbD with multimedia components in comparison to a prior study in a different population? [6]. The research focus was the method of evaluation developed on the basis of the QCEPM. This was to be evaluated descriptively by direct comparison of the primary and replication studies, focusing on two concrete research questions:

1) How do students in their first clinical semester of conservative dentistry rate the EbD e-learning resource in general and with respect to defined parameters?

2) Are there differences in individual results between the present replication study and the results of the primary study?

Material and Methods

The e-learning resource

The e-learning resource forms one part of the so-called Frankfurt Dentistry Initiative (FranZi, Frankfurter Zahnmedizinische Initiative). The component presented in the present work (“Toothache Clinic - Focus”) includes several interactive case vignettes. The resource was developed using an open-source authoring tool (WebKit Freiburg, Version 3.1 / beta) and is accessible without limitation online (https://elearning.med.uni-frankfurt.de/spielwiese-fokus/). Many functions were incorporated in the 56-page resource (multiple choice questions drag and drop drop-down menus, draw functions, video clips, feedback and tips). EbD content was incorporated into over 57% of the total resource. EbD content was integrated with respect to the stipulation of the evidence-based medicine workgroup of the Frankfurt University which refers to newcomers to EbD. The initial implementation (primary study) of the online course “Toothache Clinic - Focus” was four semesters, i.e. two years, prior.

Student group and examination intervals

Dental students from two consecutive cohorts (summer semester 2014 and winter semester 2014/15) in their first clinical semester (sixth academic semester) after using the voluntary e-learning resource were surveyed using a written questionnaire. In the first meeting (c. 30 minutes long), they were informed about the use of the resources and told that they have been available for several years. None of the students had previous received educational content on evidence-based learning. A questionnaire conceived on the basis of the Ulm Quality Criteria Catalogue for Medical Learning Programmes served as the method of evaluation [8,9]. This has previously been used in connection with FranZi [9]. The first part of the questionnaire concerned general details of semester number, sex, age, previous marks on major state exams (Abitur, preliminary scientific and medical examinations) as well as work experience and familiarity with electronic media (Likert scale from 1 = “very good” to 5 = “poor”). Furthermore, special questions also addressed the utility of the content in patient treatment and the openness to implementation of EbD. The overall rating for the module was elicited using a scale from 1 (very good) to 6 (insufficient) according to the German school marking system. Part two of the questionnaire concerned the evaluation of the resource in the dimensions of content and functional range (A), handling and technical aspects (B), and didactics and educational value (C) on a four-level Likert scale from 0 = “strongly disagree” to 3 = “strongly agree”. Students could use the resource for 14 days and were asked to return the questionnaires at a second meeting (c. 30 minutes long).

Statistics

The analysis was performed using PASS and NCSS (Version 6.0.2.1. Kaysville, Utah), using using a one-way analysis of variance. Significant differences were determined using the nonparametric Kruskal-Wallis multiple-comparison Z-value test according to Bonferroni. Means, standard deviations (SD) or frequencies were also evaluated.

After consultation with the university’s ethics committee, it was decided that an ethics committee vote was not necessary since the evaluation was anonymous.

Results

The descriptive results of the study are shown in tables 1-5. The dental students gave the e-learning module in the present study an average score of 2.40 ± 0.66 (primary study: 2.26 ± 0.64). For the questions on the learning programme in the present study, dimension A received an overall average score of 1.90 ± 0.57 (primary study: 1.90 ± 0.63), dimension B 1.48 ± 1.60 (primary study: 1.55 ± 1.93) and dimension C 2.27 ± 0.67 (primary study 2.23 ± 0.79). All obtained results in the replication study showed no statistical significant differences to the primary study.

Discussion

The present replication study has shown that an e-learning resource with dental case vignettes and evidence-based dentistry content, which has been available for several years, is on average rated positively by students of dentistry in their first clinical semester of conservative dentistry and that the users would appreciate inclusion of EbD in their course of study.

The agreement of the results with those of the primary study points to the remarkable validity of the chosen method of evaluation. Currently, there exist few replication studies in the field of e-learning. Specifically, in the context of dentistry and with EbD content, no such publications are known to the authors [3,11].

Analogous to the present study, similar research in the field of medicine focussing on online-learning of evidence-based medicine (EbM) has shown that participants judge the courses positively and that the e-learning environment is covariant with clinically relevant increase in knowledge [12-17]. It must be noted that the foundation for the use of EbD content is as a rule formed from real clinical situations and that novices are usually involved voluntarily [18,14-16]. In a similar manner, the e-learning resource investigated in the present replication study concerned real patient cases and combined the basic knowledge of EbD with orientation towards patients, resulting in a kind of practical knowledge. According to a 2013 survey on courses in EbM, two of the five steps of EbM, namely integration of results with patients and the evaluation of particular merit, fell significantly behind the others and are thus addressed less frequently [15,16]. All steps were integrated into the present resource.

Taken as a whole, the methodology of evidence-based dentistry (EbD) takes centre stage in the endeavour to place dentistry on a scientific foundation [19]. In Germany in particular, however, EbD is not firmly established in dentistry curricula. This is in stark contrast to the USA, for example, where the inclusion of EbD in dentistry curricula has already been publicised several times [20-23]. This present problem in Germany, that dentistry graduates show minimal competence in EbD at the beginning of their working life as practicing dentists, should in the future be solved by the medical licensing act for dentists currently being planned in Germany and the National Catalogue of Competence-based Learning Objectives in Dentistry containing operationalised EbM learning objectives which has already been adopted [24,25]. Moreover, a 2016 study reported that five defined parameters should be considered before a forthcoming implementation of EbM in curricula, namely 1) integrating EbM with other courses and content, 2) incorporating clinical content into EbM training, 3) EbM faculty development, 4) EbM whole-task exercises, and 5) longitudinal integration of EbM [26]. Based on these considerations, especial care was taken in the present study to integrate EbD content into the course of conservative dentistry particularly in the case of diagnosis and treatment of toothache. In addition, EbD content was related to realistic clinical cases. Faculty development was likewise apparent, since creation of the resource content in co-operation with the local EbM workgroup clearly required it. A longitudinal integration into several semesters of the dentistry study has not yet been realised, but it should be addressed in future studies.

According to the literature, teaching of EbM content can be successfully achieved in various settings (teacher-centred, e-learning, blended learning) [13,27,28]. In the present study, the resource was defined as an optional means for students’ self-study. In the future, however, it will also be used in the form of blended learning. Comparable studies that used an identical evaluation protocol with the dimensions described here, albeit in other e-learning tutorials (“Toothache Clinic – Basic”), came to comparable conclusions with respect to the dimension of “didactics and educational value” (2.29 ± 0.38) [8,9]. The two other dimensions were rated more highly, however (content and functional range: 2.32 ± 0.47 and handling and technical aspects: 2.47 ± 0.36) [8,9]. Here, it may be worth noting that these previous studies were conducted in 2009 when the e-learning resource was completely new and the students were unfamiliar with it [8,9]. Three years later, at the time of the primary study of this investigation, the field of electronic media and resources seems to have developed greatly so that, understandably, the users gave lower ratings in questions of function and technical aspects. A period of 1.5 years lay between the data of the primary and replication studies with EbD content which led at least to more conspicuous judgements for several items in the dimension “didactics and educational value” of the programme. For example, in the primary study the students gave a higher rating in regard to the appropriateness of the learning unit for self-study or group study. The users tended to take the view that the resource is well suited to self-study. None of the students had experience using the studied resource in a group or in a blended learning scenario, however, so that an according implementation is quite conceivable.

In the study at hand, no data concerning increases in knowledge were analysed, though MC questions integrated into the resource could be consulted in future studies to assess increase in the users’ knowledge. The present study is thus concerned with a purely subjective course evaluation on the part of the students, to record one limitation. Likewise, international alignment is conceivable owing to the open sourcing of the resources, though they are at present only available in German. Plans for the translation of all FranZi resources into English are already underway.

Conclusion

In summary, the results of this clinical replication study are in agreement with the primary study and show that e-learning resources with dentistry case vignettes and integrated EbD content is rated positively by students of dentistry in their first clinical semester on conservative dentistry.

1. Makel MC, Plucker JA (2014) Facts Are More Important Than Novelty: Replication in the Education Sciences. Educ Res 43: 304–316. Link: https://goo.gl/HHWGWS
2. Smith CE, Dauz ER, Clements F, Puno FN, Cook D, et al. (2006) Telehealth services to improve nonadherence: A placebo-controlled study. Telemed J E Health 12: 289-96. Link: https://goo.gl/zyuWuj
3. Brandt MJ, IJzerman H, Dijksterhuis A, Farachc FJ, Geller J (2014) The Replication Recipe: What makes for a convincing replication? J Exp Soc Psychol 50: 217–224. Link: https://goo.gl/r4CPED
4. Paradis E, Sutkin G (2017) Beyond a good story: from Hawthorne Effect to reactivity in health professions education research. Med Educ 51 :31-39. Link: https://goo.gl/7YSo6n
5. Fraenkel JR, Wallen NE (2009) How to design and evaluate research in education. 7th edition. Edited by Ryan M, McGraw-Hill, USA 103.
6. Gerhardt-Szep S, Dreher S, Rüttermann S, Weberschock T (2017) Conception and implementation of a novel E-learning module with EbM-learning contents in operative dentistry. Z Evid Fortbild Qual Gesundhwes (accepted).
7. Schulz S, Auhuber T, Schrader U, Klar R (1998) Quality criteria for electronic publications in medicine. Stud Health Technol Inform 51: 217-226. Link: https://goo.gl/ScG1n1
8. Scholz W, Fassnacht U, Öchsner W, Stracke S, Waldmann UM, et al. (2006) Quality criteria catalog for medical learning programs ( German: Qualitätskriterienkatalog für medizinische Lernprogramme). GMS J Med Educ 23: 1-12. Link: https://goo.gl/8Ty3qW
9. Gerhardt-Szep S, Lagan A, Schwalm B, Kandsperger L, Möltner A, et al. (2011) A Questionnaire to Evaluate Computer-Assisted Learning Programs.In Proceedings of the Second Conference Research in Medical Education (RIME),  25-27 May, Tübingen, Germany, Edited by Lammerding-Köppel, Competence Center for University in Medicine Baden-Württemberg 29. Link: https://goo.gl/iqc8En
10. Gerhardt-Szep S (2009) Conception, implementation and evaluation of the interactive, interdisciplinary and case-oriented eLearning course "Zahnschmerzambulanz. MME thesis, Project work, Heidelberg University, Medical Faculty. Link: https://goo.gl/RR7b4A
11. McCambridge J, Witton J, Elbourne DR (2014) Systematic review of the Hawthorne effect: new concepts are needed to study research participation effects. J Clin Epidemiol 67: 267-277. Link: https://goo.gl/1MecfP
12. Kotur PF (2012) Introduction of evidence-based medicine in undergraduate medical curriculum for development of professional competencies in medical students. Curr Opin Anaesthesiol 25: 719-723. Link: https://goo.gl/h77yzK
13. Ilic D, Nordin RB, Glasziou P, Tilson JK, Villanueva E (2015) A randomised controlled trial of a blended learning education intervention for teaching evidence-based medicine. BMC Med Educ 15: 39. Link: https://goo.gl/JN9J27
14. Bergold M, Strametz R, Weinbrenner S, Khan KS, Zamora J et al (2013) Evidence-based Medicine online for young doctors - a randomised controlled trial. Z Evid Fortbild Qual Gesundhwes 107: 36-43. Link: https://goo.gl/w3EnsH
15. Weberschock T, Dörr J, Valipour A, Strametz R, Meyer G et al. (2013) Evidence-based medicine teaching activities in the German-speaking area: a survey. Z Evid Fortbild Qual Gesundhwes 107: 5-12. Link: https://goo.gl/fuhHch
16. Weberschock T, Sorinola O, Thangaratinam S, Oude Rengerink K, Arvanitis TN et al. (2013) How to confidently teach EBM on foot: development and evaluation of a web-based e-learning course. Evid Based Med 18: 170-2. Link: https://goo.gl/3XC2Tn
17. Khader YS1, Batayha W, Al-Omari M (2011) The effect of evidence-based medicine (EBM) training seminars on the knowledge and attitudes of medical students towards EBM. J Eval Clin Pract 17: 640-3. Link: https://goo.gl/ixWDZy
18. Lallier TE (2014) Introducing evidence-based dentistry to dental students using histology.  J Dent Educ 78: 380-8. Link: https://goo.gl/8reR7X
19. Türp JC, Röhrich C, Antes G (2005) Evidence-based dentistry in German-language dental journals 1997-2003. Evid Based Dent 6: 86-7. Link: https://goo.gl/UqPsp7
20. Teich ST, Demko CA, Lang LA (2013) Evidence-based dentistry and clinical implementation by third-year dental students. J Dent Educ 77: 1286-99. Link: https://goo.gl/ewUL1R
21. Marshall TA, Straub-Morarend CL, Guzman-Armstrong S, McKernan SC, Marchini L et al (2017) An approach on defining competency in evidence-based dentistry. Eur J Dent Educ 2017 Mar 6. doi: 10.1111/eje.12267. Link: https://goo.gl/uTUakD
22. Marshall TA, Straub-Morarend CL, Handoo N, Solow CM, Cunningham-Ford MA, Finkelstein MW (2014) Integrating critical thinking and evidence-based dentistry across a four-year dental curriculum: a model for independent learning. J Dent Educ 78: 359-67. Link: https://goo.gl/kirxyr
23. Marshall TA, Straub-Morarend CL, Qian F, Finkelstein MW (2013) Perceptions and practices of dental school faculty regarding evidence-based dentistry. J Dent Educ 77: 146-51. Link: https://goo.gl/VZsgcN
24. Mühlhauser I, Gerhardt-Szep S, Schwendicke F, Türp JC (2016) Evidence-based medicine must be more than a voluntary offer in the new regulations for dental training (ZÄPrO)!  Statement of the German Network for Evidence-based Medicine. Berlin. 22.11.2016. Link:  https://goo.gl/LXApbb
25. Fischer MR, Bauer D, Mohn K, NKLM-Projektgruppe (2015) Finally finished! National Competence Based Catalogues of Learning Objectives for Undergraduate Medical Education (NKLM) and Dental Education (NKLZ) ready for trial. GMS Z Med Ausbild 32: Doc35. Link: https://goo.gl/4kaZZ3
26. Maggio LA, ten Cate O, Chen HC, Irby DM, O'Brien BC (2016) Challenges to Learning Evidence-Based Medicine and Educational Approaches to Meet These Challenges: A Qualitative Study of Selected EBM Curricula in U.S. and Canadian Medical Schools. Acad Med 91: 101-6. Link: https://goo.gl/RPzu2N
27. Maggio LA (2016) Educating physicians in evidence based medicine: current practices and curricular strategies. Perspect Med Educ. 2016 Oct 25. Link: https://goo.gl/7RnBdY
28. Ilic D, Maloney S (2014) Methods of teaching medical trainees evidence-based medicine: a systematic review. Med Educ 48: 124-35.  Link: https://goo.gl/7U59SX