BCMJ, Vol. 50, No. 7, September 2008, page(s) 380-383—Article
Chris Y. Lovato, PhD, Caroline C. Murphy, MSc
Comparability of student performance and experiences in UBC’s distributed MD undergraduate program: The first 2 years
Exam scores and survey responses indicate students are benefiting from comparable educational experiences at all three British Columbia medical school sites.
Comparability of educational experiences is essential to the success of a distributed medical education program. Evidence of comparability is needed to inform both accreditation and decision making. To assess the comparability of student performance and experiences during the first 2 years of UBC’s distributed MD undergraduate program, student exam scores and student responses to surveys regarding courses, tutor performance, and information technology were analyzed. Student performance and perceptions were found to be similar across the three geographically separate sites, and the information technology used was viewed by students as acceptable and accessible. A larger evaluation study is now underway at all sites to examine short-term, intermediate, and long-term outcomes such as choice of speciality training and practice location.
With the introduction of distributed medical education in BC and across Canada, medical schools are addressing new questions about the comparability of programs at different sites. The Liaison Committee on Medical Education (LCME) accreditation standards state, “There must be comparable educational experiences and equivalent methods of evaluation across all alternative instructional sites within a given discipline” (ED-8).[1] Evaluation is also essential for ongoing program improvement and decision making.
A comparability study of preclinical training in UBC’s distributed MD undergraduate program was undertaken after the first class entered in 2004. Evaluation questions focused on student performance, perceptions of family practice tutors, and satisfaction with videoconference technology.
Methods
All students enrolled in the program during the first 2 years of implementation (2004/05 and 2005/06) were included in the evaluation. To determine if there were significant differences in curriculum delivery and student performance at the three program sites in Victoria, Prince George, and Vancouver, we compared two key courses:
• Foundations of Medicine (FMED). This course in basic science begins in the second term of year 1 and continues through the end of year 2. The course integrates problem-based learning, lectures, laboratories, and self-directed learning experiences and is taught in 12 systems-based blocks (e.g., cardiovascular, pulmonary, integumentary).
• Family Practice Continuum (FPC). This course is designed to introduce students to the principles of family medicine. The course begins in the first term of year 1 and continues through the end of year 2; it employs lectures, small group tutorials, workshops, and clinic visits during which students have an opportunity to apply their learning in a family practitioner’s office.
Data collected from students at all three sites included:
• Exam scores for years 1 and 2 FMED blocks and FPC.
• Student responses to surveys assessing performance of years 1 and 2 FPC tutors.
• Student responses to surveys assessing the acceptability and impact of technology.
Survey items were compared across sites using 1-way ANOVA. A statistically significant finding in exam scores (P=.05) was considered to be of programmatic significance if there was a difference of more than five percentage points between each pair of sites; a statistically significant difference of more than one point on the five-point Likert scale used in student surveys was considered to be of programmatic significance.
Results
Student performance
Exam scores from all students enrolled during 2004/05 and 2005/06 were included (see Table 1). Mean scores for the four blocks of FMED taken during year 1 ranged from 77.5% to 89.3% in 2004/05, and from 77.9% to 83.8% in 2005/06. Mean exam scores for the eight blocks taken during year 2 FMED in 2005/06 ranged from 74.7% to 84.0%.
Comparison of FMED exam scores using 1-way ANOVA indicated that in the first 2 years of implementation there were no statistically significant differences across sites for any of the year 1 FMED blocks.
In 2005/06, there were statistically significant differences across sites in year 2 FMED scores for two of the eight blocks. In one program site, average scores were 5.6 and 5.0 percentage points lower in Endocrine and Metabolism and Brain and Behaviour than in one of the other program sites. Upon closer examination of the data, we identified two students from this site who received scores below the mean score obtained by all other students at the same site.
The mean score across all sites for the year 1 FPC was 89.3% in 2004/05. There were no significant differences in scores across program sites. In 2005/06, the mean score for year 1 FPC was 86%. There was a statistically significant difference across program sites of 2.2 percentage points, but this is not considered to be of programmatic significance.
Family practice tutors
At the end of both terms students completed surveys regarding their FPC physician tutors. Students responded to questions that addressed overall quality of teaching (global tutor performance) and the ability of tutors to help students achieve stated learning objectives (effectiveness of tutor) using a five-point Likert scale (1=unacceptable to 5=outstanding) (see Table 2).
Students also used a five-point Likert scale (1=strongly disagree to 5=strongly agree) when responding to a statement about the general rules taught by tutors (see Table 3). The response rate for the tutor surveys was 94%.
Results from 1-way ANOVA indicated statistically significant differences on two items in year 1, 2004/05, and one item in year 1, 2005/06; however, in all three cases the difference was less than 1 point. There were no statistically significant differences in the year 2 assessment of FPC tutors across sites.
For the year 2 curriculum in 2005/06 all sites averaged at least 3.69 points or better on all three items. As Table 2 shows, approximately 80% of year 1 and 2 students rated their FPC tutor’s general performance as above average or outstanding. And as Table 3 shows, more than 90% of year 1 and 2 students agreed or strongly agreed that they were taught some “pearls” that will be useful in future patient care by their FPC tutors.
Responses to open-ended questions indicated that students at all sites favor the Family Practice Continuum course for its practical application. In the words of one student, “I really loved this course because it gave us a chance to get out to the clinic right away and see how our learning was applicable to practising medicine.” Student suggestions for improving the course were similar across sites and related to further clarification of learning objectives and student roles/responsibilities.
Technology
At the end of the 2005/06 academic year, students completed a survey about the technology used to deliver the curriculum (see Table 4). Students responded to questions about videoconference picture and sound quality using a five-point Likert scale (1=strongly disagree to 5=strongly agree). The response rate was 66%.
Each site averaged at least 3.73 points or better on the items reported. Results from ANOVA indicated a statistically significant difference in responses about videoconference picture clarity for year 1 students and in sound quality for year 1 and 2 students. In each case the difference was 0.52 or less. The majority of students agreed or strongly agreed that video and sound quality met their expectations.
Conclusions
This evaluation study provides evidence of comparable student performance and comparable student perceptions across the three geographically separate sites of the preclinical UBC MD undergraduate program. Survey responses concerning the quality of curriculum delivery by videoconference indicate that students at all sites had a comparable educational experience.
As noted earlier, we interpreted a difference of more than five percentage points on exam scores and a difference of more than one point on the five-point Likert scale survey results to be of programmatic significance. Several statistical differences were observed; however, most fell below the predetermined level of programmatic significance. For example, in the exam scores for the FPC block, the greatest difference was between 89.0% and 86.8%—a difference of only 2.2 percentage points.
The two FMED blocks in which we found a statistical difference (Endocrine and Metabolism and Brain and Behaviour) included the scores of two students from one site who had performed below the class average. Both students were successfully remediated and these two blocks will continue to be monitored closely for any possible trend in programmatically significant differences.
There were no differences of programmatic significance across sites regarding the ability of tutors to provide constructive feedback or the effectiveness of tutors in enabling students to achieve FPC stated learning objectives. Anecdotal evidence and data regarding FPC experiences highlight some differences in learning opportunities across sites that reflect the unique characteristics of the three learning environments.
For example, 2005/06 data indicate that 70% of family practice tutors in Prince George delivered babies compared with 25% of Vancouver tutors and 32% of Victoria tutors. As this cohort progresses into the clinical years it will be important to document these differences to inform program planning and expansion.
Results from this evaluation also suggest that the technology UBC has implemented is producing positive results both in terms of student performance and student acceptance. Videoconferencing is viewed by students as an acceptable and accessible method of learning new information. In a separate focus group study, one student summarized his experience of moving from the main UBC campus to one of the other program sites by saying, “The school experience hasn’t changed.”
This comparability study is part of a comprehensive evaluation of UBC’s MD undergraduate program. The larger study will eventually address process and implementation questions regarding all aspects of the program (e.g., admissions, curriculum, student affairs), as well as evaluation of short-term, intermediate, and long-term outcomes such as choice of speciality training and practice location.
Competing interests
None declared.
References
- 1. Liaison Committee on Medical Education. Functions and structure of a medical school. 2007. www.lcme.org/functionslist.htm (accessed 18 June 2008).
Dr Lovato is the director of the Evaluation Studies Unit, Faculty of Medicine, UBC, and associate professor in the Department of Health Care and Epidemiology at UBC. Ms Murphy is the assistant director of the Evaluation Studies Unit, Faculty of Medicine, UBC.
