Background
This semester (January – April 2015) my colleagues and I at Learning Services in the Sauder School of Business have helped professors create and deploy over six thousand online exams for students bringing to the classroom their own devices connected to UBC’s secure wi-fi and logged into “Connect” – UBC’s instance of Blackboard Learn‘s LMS. We have developed very specific guidelines in collaboration with professors in order to ensure students are technically ready for the exam, including an online technical pretest and instructions displayed in their Connect LMS course, as well as online exam instructions shown on digital projector screens during the exam.
A very large group of students
We had a single simultaneous deployment of an online exam for 1272 students enrolled in COMM291 – Application of Statistics in Business. All students brought their laptops to class and started the exam at approximately the same time using the BlackBoard Learn Test tool. We repeated this twice, both for the midterm and the final exam. 15 exam rooms were located in two buildings on the UBC Vancouver campus.
Midterm Performance Blues: Lag time, saving, image display issues
For the midterm we noticed many students experiencing lag times with when initially accessing the exam, some with more than 30 seconds of lag time. This was most likely the result of 1. limited staggered start time (only 15 second intervals between launching largest rooms) and 2. the fact that the LMS database was having to process 1200+ instances of the entire online exam displaying all questions at once. Several students lost some of their responses due to saving issues, while others had issues seeing displayed images properly.
Final Success: Logistics and Exam Build Improvements
Thanks in large part to the eagerness and technical savviness of COMM291 Prof. Greg Werker, we were able to work collaboratively with UBC IT Learning Applications to optimize certain parameters of the online exam deployment to attain much better performance results:
- Increased staggered release time to 1 minute (60 second) intervals of approximately 100-200 students at a time going into the exam
- Placed several on standby mode in the event lagging occurred in other rooms, so as to provide a sufficient buffer for database load issues
- Reduced picture sizes displayed during the exam to less than 100 kilobytes
- Placed mandatory questions in the exam prompting students to ensure their questions are saving.
Performance Comparisons of Midterm and Final Exams
UBC IT supported us on the database side and took performance measurements and comparisons for both the midterm and the final exams. Here are the launch statistics (performance during the first 25 minutes):
With the staggered release, we were seeing more “green”, i.e. quicker access times for students logging into the assessment tool (the online exam link). For “Save Attempt” performance, i.e. when answers were automatically saved or when students attempted to save their own answers, we noticed marked improvements on launch and submission times, with only very slight latency issues that were hardly noticeable by students:
Finally with submission performance we also saw significant improvements:
Because the launch was staggered by a minute per chunk of rooms, submission times were also staggered for those students who chose to use the full allotted time (the majority of students). Therefore the LMS database did much better, with reduced latency issues. Jeff Longland from UBC IT Learning Applications mentioned that “Looking at this data, I think the staggered start helped to mitigate some of the issues that were experienced with the midterm. There doesn’t appear to be as much congestion around launch and submission events for the final exam.”
Kudos
A big thanks goes out to the UBC IT Learning Applications team, Infrastructure support, Networking and a host of other IT departments all working collaboratively with the goal of improving LMS performance on our campus.
Barish Golland 