Before I give a more detailed description of the study I want to caution that this is still a working paper. This means that it has not gone through the peer review process. The peer-review process is in place to assure that research that is published is high quality. As the name suggests, this process involves a review by peers in your scientific community, wherein potential issues are critiqued and addressed before publication. We therefore tend to lend more weight to research findings that come from peer-reviewed journals and are cautious about results from non-peer-reviewed sources. That said, this paper is a summary of a very large body of work and sub-sections of that work have been published elsewhere in peer-reviewed journals (3, 4). Furthermore, while peer-review is incredibly important to the scientific method, it is also notoriously fickle and time consuming. It could be months, or even years, until this work gets published in the form of a peer-reviewed journal. So while I offer a word of caution that the final product isn’t 100% up to snuff, I still think it’s an important contribution that’s worth discussion.
In this study researchers partnered with professors of a first-year course in economics. Economics is a popular first-year course that is taken by roughly 5,000 students each year at the University of Toronto, making it an ideal course to implement a study intervention in. As a small part of their grade, students were asked to fill out an online survey. After filling out this survey students were randomly assigned to either a control group (which involved taking a personality survey) or a study treatment condition. Over the course of five years, researchers investigated six study treatments:
Goal Setting. Students were asked to think about their future goals and reflect on how their current behaviors might or might not help them achieve those goals.
Mindest. Students were told about the benefits of adapting a positive and resilient mindset towards academic challenges.
Online Coaching. Students completed online exercises where they were encouraged to think about the future they wanted and what steps they could take to achieve it.
Online Coaching with One-Way Text Messaging. Students completed online exercises as above, but received follow-up messages throughout the school year that offered advice, encouragement, and information about resources.
Online Coaching with Two-Way Text Messaging. Here the online coaching and texting were designed to emphasize study time. In this treatment group upper-year students who were academically qualified and had previous experience mentoring or tutoring were selected to act as coaches and follow up with students via text.
Face-to-Face Coaching. Students were offered personal coaches to meet with once a week during the school year. Again, coaches were upper-year students. Coaches were encouraged to follow up with students between meetings and were available via Skype or text messaging between meetings.
There were a number of measures of interest in this study, but for the purposes of this summary I’ll go over three: study time, student grades and persistence, and well-being.
The researchers note that university students spend less time studying than they should. Many colleges and universities recommend studying two to three hours outside of class for every one hour spent in class. A typical course load is 15 hours of coursework a week, which translates to 20-45 hours of studying outside of class. Most students spend less than 15 hours a week studying outside of class (5). Therefore, low study time is often blamed for low graduation rates. Student simply aren’t devoting enough time to their studies.
Of the six treatments in this study the Online and Two-Way Text Messaging groups showed some improvement in study time. On average they studied about 1.3 hours more a week. None of the other treatments produced any positive effect on study time.
Student Grades and Persistence
None of the treatments affected student grades. Regardless of treatment condition, final grades were the same as the control group. Extra coaching, weekly meetings, focusing on goals – none of it impacted grades.
Similarly, none of the treatments affected persistence. Students in the control group were just as likely to continue on to their second and third year as students in the treatment conditions. A student’s decision to stay and persist towards completing their degree was not impacted by any of the treatments.
Here we finally get to some good news! The coaching conditions, particularly the Face-to-Face treatment, had positive effects on student well-being. Students who met regularly with a coach had better well-being, including an improved sense of belonging and university support. In the current environment with increased student mental-health issues, this finding is very important.
It is notable that the Face-to-Face treatment was by far the most expensive and time-consuming of all the treatments done in this study. The researchers set out with the express goal of finding a less resource demanding intervention to improve student outcomes. Which is perhaps why this intervention seems to have been developed last and was done with a substantially smaller sample than the rest of the treatment conditions (only 90 students were selected to receive Face-to-Face coaching).