Introduction

Prescription drug misuse (PDM) among youth aged 12–18 has been labeled a national public health emergency (Hargan, 2017) as the United States witnessed a 500% increase in opioid overdose deaths among 15–24 year old adolescents between 1999 and 2020 (CDC Wonder, 2021). Data from the National Survey on Drug Use and Health (NSDUH) between 2015 and 2017 suggest that approximately 20% of youth aged 12–17 use prescription opioids medically or non-medically and approximately 3.5% of youth misuse prescription opioids (Carmona et al., 2020). Although the primary focus of the prescription drug abuse epidemic in public health and the media has been opioids (Austin & Short, 2020), non-medical use of prescription depressants (i.e., tranquilizers or sedatives) and stimulants among youth are becoming increasingly problematic as well. Approximately 8% of youth reported lifetime use of sedatives, with about a third of those reporting misuse (Schepis and Hakes, 2013). A 2016 study (McCabe et al., 2017a, 2017b) found that 16.7% of high school students misused prescription stimulants at least once by age 18 with misuse reported to be particularly high (43%) among adolescents with medical prescriptions. Surveillance data from emergency departments between 2016 and 2019 showed that visits for stimulant overdoses increased across all pediatric ages (Hadland et. al., 2021).

The sequelae of PDM in adolescence are pervasive and long term. Adolescents who report PDM face an immediate increased risk of academic failure, engaging in risky behaviors, and psychopathology (Hammond et al., 2007; Schepis & Hakes, 2013). Longitudinal studies suggest that the negative effects of adolescent PDM persist into adulthood. Prescription stimulant misuse during adolescence is associated with a greater likelihood of substance use disorder and lower likelihood of college graduation (McCabe et al., 2017a, 2017b). Adolescents who misuse prescription opioids are at higher risk for major depressive episodes (Edlund et al., 2015; Havens et al., 2011) and prescription and illicit drug dependence (Young et al., 2012). Adolescents who report a combination of both medical and non-medical use of prescription depressants are two to three times more likely to experience substance abuse disorders as adults (McCabe et al., 2017a, 2017b).

In summary, although a significant amount of media and public health attention has centered on prescription opioid abuse among adolescents, adolescent misuse of prescription depressants and stimulants are also a concern. Increasingly, public health experts and medical providers have been calling for effective, evidence-based prescription drug misuse prevention and intervention programming that reaches large numbers of youth who may be exposed to opportunities to engage in non-medical prescription drug use (U.S. Dept. of Health and Human Services, 2013). Fortunately, initial evidence suggests that programs aimed at promoting prescription safety education, limiting prescription distribution, and reducing overdoses are shown to be effective for this age group (e.g., Compton et. al. 2019). Such programming presents a window of opportunity to educate youth regarding the risks of prescription drug misuse while simultaneously promoting positive prescription drug safety skills.

The Landscape of Drug Misuse Prevention Programming

According to the National Association of State Boards of Education (NASBE, 2022; https://statepolicies.nasbe.org/), nearly all states and territories in the U.S. have policies regarding alcohol and drug use/abuse instruction, either as a stand-alone topic or as part of health education. Myriad programs address drug misuse and abuse prevention among children and teens, with school-based programming being the most commonly evaluated intervention type (Flay, 2000; see Sandler et al., 2014 for a summary of intervention meta-analyses conducted between 2000 and 2013). The effectiveness of these programs vary widely, and the financial investments required for them are among the primary barriers to implementation (Forman et al., 2008). Nevertheless, most states adopt evidence-based standards to help guide and influence which curricula schools will select. These standards address both content (e.g., programs must assess knowledge of biological effects of substance misuse) and delivery method (e.g., interactive instruction; Bruckner et al., 2014).

One program with promising outcomes in randomized control trials is the widely-studied Strengthening Families Program (SFP; Kumpfer & Magalhães, 2018). SFP involves both youth and their caregivers in 2-h weekly skills classes, including drug refusal skills. The dosage ranges from 7 to 14 weeks depending on the risk level of the population, with lower dosage recommended for lower-risk families or communities. The program has been shown to reduce substance misuse, anxiety and depression, and child maltreatment. SFP ranges in cost from around several hundred dollars for the program, to several thousand dollars for optional program facilitator training (https://strengtheningfamiliesprogram.org/).

In addition to school- and community-based programs such as SFP, technology-based interventions (i.e., web-delivered and mobile-friendly) are emerging as effective, lower-cost tools in the prevention landscape, particularly among youth (Marsch & Brodovsky, 2017). According to Marsch and colleagues (2021) technology-based interventions—as compared to traditional school- and community-based programs—are more appealing to youth, more cost effective, and allow for simple, standardized implementation across a variety of platforms with little to no training required. In addition, technology-based programs can have a more expansive reach, enrolling large numbers of youth with relative ease. In the present study, we evaluate the effectiveness of the EVERFI Prescription Drug Safety (PDS) curriculum, a free, technology-based PDM prevention program designed for youth in grades 8–12. As a technology-based intervention, the program is flexible enough to be administered in school, community settings, and at home. In the present study, we recruited teachers to administer the EVERFI program to their students in schools.

Challenging Misperceptions to Prevent Prescription Drug Misuse

Regardless of the context in which the program is delivered, there are some shared features of effective PDM interventions that meet evidence-based standards (Bruckner et al., 2014). These include fostering better understanding of the norms and behaviors regarding substance misuse, refusal and peer-resistance skills, and knowledge of the effects of PDM (Bruckner et al., 2014).

Rooted in Social Norms Theory, which proposes that individuals’ behaviors align with their beliefs about the prevalence and acceptability of those behaviors in their social contexts (Berkowitz, ​​2003), PDM programs that adopt a social norms approach are designed to shift the normative behavior of a group by correcting misperceptions about the prevalence and social acceptance of the targeted unhealthy behavior (Dempsey et al., 2018). Normative beliefs have been linked to intentions to initiate alcohol, cigarette, and marijuana use (Olds et al., 2005). As teens’ overall perceptions of PDM tend to be significantly higher than actual misuse (Sanders et al., 2014), shifting what is accepted as “normal'' or “typical” prescription drug use behavior in a school setting involves challenging students’ perceived versus actual peer actions and attitudes about prescription drug use (Flay, 2000).

Effective PDM prevention programs also promote knowledge of safe prescription drug use, challenging the misperceptions of prescription drug abuse as safer, less addictive, and less risky than using illicit drugs. These misperceptions are a contributing factor to subsequent abuse (Twombly & Holtz, 2008). A nationally representative survey of youth in grades 9–12 found that about one-third believe that prescription drugs are okay to take without a prescription, for example to use as study aids or to deal with injury or pain (Partnership for Drug-Free Kids, 2013). Accordingly, the U.S. Department of Health and Human Services finds that educating young people about prescription drug safety and helping them develop the intentions and skills to make healthy choices are two critical components for intervention success (U.S. Dept. of Health and Human Services, 2013). While family- and community-based programming may be effective to reduce the availability of prescription drugs and address PDM etiology (Spoth et al., 2013), interactive technology-based interventions may be particularly useful to correct misconceptions about prescription drugs as safe, to establish accurate peer social norms regarding PDM, and to educate youth on the legitimate medical use of prescription drugs (Marsch et al., 2021). The EVERFI drug safety promotion framework (Everfi, 2020a) leverages web-based technology and uses social norms theory to challenge myths about prescription drug misuse, and to provide youth with PDM knowledge and refusal skills.

Moderators of Program Effectiveness

Each student and teacher who engages with a PDM program has their own unique background, skill sets, and life experiences that may impact program effectiveness. Even a well-documented, uniform program will be administered and received differently based on individual and contextual factors. For example, a student who has a family member impacted by prescription drug misuse may perceive and engage with the PDS curriculum differently than a student who has no prior exposure to prescription drug misuse. A teacher who has prior experience teaching the PDS curriculum may be more effective than a teacher who has never taught the curriculum before. Given these endogenous factors, it is important to understand whether student or teacher characteristics moderate program effectiveness.

Implementation characteristics are one such variable hypothesized to moderate the impact of prevention programs. In a meta-analysis of substance abuse prevention program evaluations, the authors found significant effects for programs identified as interactive—defined as providing contact among participants and opportunities to exchange ideas, and which taught refusal skills and interpersonal skills, as compared to programs that use didactic methods (Sandler et al., 2014).

Other implementation characteristics that may moderate the effectiveness of a school-based PDM prevention program are the facilitator’s comfort and confidence with the material, prior experience implementing the program, and resources and support to deliver the program as intended. Durlak and DuPre (2008) provide guidance about implementation factors that may moderate the effectiveness of universal school-based interventions on student outcomes. These factors include dosage (i.e., degree of exposure to the curriculum content); quality of administration (e.g., to what extent the teachers feel prepared, confident, and enthusiastic about their delivery of the program); and participant engagement (i.e., the degree to which students are engaged with the material).

Student demographic characteristics are likewise hypothesized to moderate the effectiveness of intervention programs to promote healthy behaviors (Durlak et al., 2011). Gender differences in substance use and abuse behaviors are well-documented: both males and females initiate at similar rates, with males increasing faster and demonstrating greater likelihood than females to abuse addictive drugs in adulthood (Kuhn, 2015). With regard to moderating the effectiveness of interventions, gender and race may be crude but statistically meaningful proxies for underlying cultural norms, societal expectations, and socialization processes that impact the ways in which students experience a school-based program (Rowe & Trickett, 2018). Evidence suggests gender moderates the effectiveness of community-based preventive interventions to reduce delinquent behaviors, including substance use, although there are no clear moderation patterns across programs (Fagan & Lindsey, 2014). Likewise, a meta-analysis of demographic characteristics moderating the impact of school-based social-emotional learning programs found no consistent direction of effect for gender or race across programs (Rowe & Trickett, 2018), but did identify “within-study” variation. Prevention scientists call for evaluators to test relevant demographic moderators of intervention program effectiveness, to appropriately qualify the generalizability of intervention programs and meet standards for evidence-based prevention (Flay et al., 2005).

EVERFI’s Prescription Drug Safety Program

The EVERFI Prescription Drug Safety (PDS) curriculum was created for youth in middle and high school to empower students with knowledge about safe prescription drug use and provide tools to support their peers in avoiding or seeking help with prescription drug misuse. The PDS curriculum is a technology-based digital learning experience that incorporates interactive, true-to-life scenarios that reinforce key learning objectives and utilizes social norms theory to engage the healthy majority and challenge misperceptions of PDM.

The PDS curriculum has been implemented in schools across the United States since 2018. Individual teachers and school districts elect to use the PDS curriculum to teach prescription drug safety in regularly scheduled classes, or have students complete the program at home. Students complete six modules regarding prescription drug safety skills: “The Basics,” “Science of Addiction,” “Understanding Prescriptions,” “Safe Use,” “Refusal Skills,” and “Supporting a Friend” (see Fig. 1). The PDS curriculum has built-in self-assessments for tracking student progress through the program. There is a pre-survey and post-survey as well as twelve quizzes that are administered throughout the course to assess learning for each module.

Fig. 1
figure 1

PDS curriculum course structure includes a pre-survey (“Survey 1”), a post-survey (“Survey 2”), and 6 modules with quizzes to assess learning at each module

Full size image

The present study is an evaluation of the EVERFI PDS curriculum. The program seeks to promote the following key drug safety outcomes:

  1. 1.

    Personal responsibility: Sense of responsibility for helping others in their community who may be at risk for abusing or misusing prescription drugs.

  2. 2.

    Social norms: Perceptions of the prevalence of PDM and peer engagement in preventing PDM.

  3. 3.

    Knowledge: Understanding of addiction and safe use of prescriptions.

  4. 4.

    Future actions: Intentions to misuse prescription drugs in the future.

  5. 5.

    Refusal skills: Avoiding prescription drug misuse/abuse.

Preliminary pre-/post-assessments suggest that the PDS curriculum improves knowledge of appropriate prescription drug use and safety and empowers students to take action in their community to refuse and prevent prescription drug misuse (Everfi, 2020b). For example, Everfi’s (2020b) Prescription Drug Safety Network Impact Report found a 16% increase in students reporting “It is my responsibility to help prevent prescription drug use at my school” from pre- to post- survey, and a 22% increase in students reporting that they can identify the signs of prescription drug abuse and misuse. However, these data were collected from students who completed the EVERFI PDS curriculum only—there was no control group comparison to ensure that the observed changes in attitudes and behavior were a result of the PDS curriculum and not simply developmentally normative changes.

The present study is a quasi-experimental evaluation of the PDS curriculum. Focusing on the five key outcomes described above (Personal Responsibility, Social Norms, Knowledge, Future Actions, and Refusal Skills), this evaluation addresses four research questions. First, we examined whether students demonstrated growth in PDM safety skills between the pre-survey and post-survey (Research Question (RQ) 1), and the moderating effects of student demographic characteristics such as gender, race/ethnicity, or student prescription drug history (RQ2). Next, we aimed to better understand curriculum implementation and outcomes for students who completed the curriculum (i.e., the experimental condition) by examining whether observed growth in the five key outcomes was maintained one month after completion of the PDS curriculum (RQ3). The purpose of these first three research questions is to examine whether EVERFI’s no-cost PDS curriculum is an effective tool for improving PDM outcomes among large groups of students. Our final research question focused on whether implementation characteristics moderated variation in student growth in the five key outcomes (RQ4). Implementation characteristics included program dosage, quality of administration, student engagement, and teachers’ experience with the PDS curriculum. Understanding how these aspects of implementation impact youth outcomes will help us to make recommendations to improve the PDS curriculum content or implementation protocol.

Method

EVERFI (www.everfi.com) is a digital curriculum provider that offers teachers and schools online programming addressing a variety of social issues at no cost. Using a quasi-experimental design, we evaluated the relationship between EVERFI’s Prescription Drug Safety (PDS) curriculum (https://everfi.com/courses/k-12/prescription-drug-safety-high-school/) and changes in five key prescription drug safety skills (Personal Responsibility, Knowledge, Social Norms, Future Actions, and Refusal Skills) in a sample of students in grades 8–12. In addition, we explored whether student characteristics and implementation characteristics were related to variation in student growth in the five key outcomes. The study protocol was approved by the Biomedical Research Alliance of New York, a fully accredited independent Institutional Review Board, and included a waiver of consent (as the research involved no more than minimal risk to participants).

Recruitment

We conducted an a priori power analysis to guide recruitment goals. Results suggested we had sufficient power (0.80 or higher) to detect differences in means as small as 0.11 with samples of 1000 students in the experimental and control groups. Teachers were recruited to participate in the study using EVERFI’s database of teachers for the 2021–2022 school year. In the Fall of 2021, EVERFI sent emails to departments of education and individual schools and teachers describing the study. Teachers were asked to indicate interest in study participation via an electronic form linked in the email. Teachers were eligible for participation if they served 10 or more students and were administering the PDS curriculum between September 2021 and December 2021 (experimental condition only) or planned to delay the PDS curriculum until after January 2022 (control condition only).

The unit being assigned to either the experimental or control conditions were the teachers. Teachers who expressed interest in participating in the study, but had already started administering the PDS curriculum, were assigned to the experimental group (along with their students). For the remaining teachers who had not started the curriculum, they (and their students) were randomly assigned to either the experimental or control group. Figure 2 illustrates the process of enrollment, allocation, and follow-up. In all, 142 teachers received a group assignment. Of these, a total of 46 teachers and 1251 students successfully completed both the pre- and post- surveys for the control group. In the experimental group, 48 teachers and 1074 students successfully completed the pre- and post- surveys, and their data were used to address RQ1 and RQ2; 48 teachers and 1257 students completed a time 3 survey and a pre or post survey, and their data was used to address RQ3 and RQ4. Blinding was not used in the random assignment. As an incentive for participation, teachers received a $50 e-gift card.

Fig. 2
figure 2

Participant flow chart and study design

Full size image

Procedure

To assess student growth in the five key outcomes, we employed a pre-post design, using two student surveys that took approximately 10 min each to complete. Students in the experimental group completed a “pre-survey” prior to starting the PDS curriculum and a “post-survey” immediately after completing the curriculum. Students in the control group completed both the pre- and post-surveys prior to receiving the PDS curriculum. Because the EVERFI curriculum is designed to allow teachers flexibility in terms of how quickly (or slowly) they move through the material, we did not place restrictions on the length of time required between the pre- and post-surveys. On average, students completed the surveys two weeks apart (M = 14.34 days, SD = 12.73). In the experimental group, two additional surveys were administered—students completed a “time 3 survey,” approximately one month (M = 27.58 days, SD = 13.14) after completing the post-survey and teachers completed a 10-min Teacher Survey, which asked a series of questions regarding their implementation of the PDS curriculum.

Pre- and post- student surveys were collected via the EVERFI PDS program’s embedded online surveys (experimental group) or through Survey Monkey (control group). Data for the time 3 follow-up student survey and the teacher survey were collected via Survey Monkey.

Intervention

The intervention was delivered to the experimental condition. As described above (see Fig. 1), students in the experimental condition completed six modules regarding prescription drug safety skills at their own pace. The PDS curriculum is web-based and students are directed by their teachers to log into the EVERFI PDS website as part of class instruction (for example, during their regular health class). The curriculum uses a pre-survey (administered before students complete the curriculum) and post-survey (administered after students complete the curriculum) to assess changes in the five key outcomes. In addition to the pre- and post-surveys, students complete a brief quiz after each of the six modules to assess whether they understood the module’s content. Students must attain a score of 80% or higher to move on to the next module. The PDS curriculum can be delivered over the course of one day or several class instruction periods—the pace of the course is entirely up to the teacher’s discretion. In the experimental group, the average time students took to complete the curriculum was M = 3.65 days (SD = 7.23).

There was high fidelity to the intervention protocol. All students in the experimental condition received the same curriculum in each of the six modules. Fidelity is maintained by requiring that students attain a score of 80% on the quizzes that follow each module before moving on to the next module. Under these requirements, the curriculum ensures that all students who completed a post-survey also completed all six modules and demonstrated a reasonable level of understanding (as demonstrated by attaining a score of 80% on each quiz).

Participants

The baseline demographic characteristics of participants and relevant differences by study condition are described below.

Teachers

Ninety-four teachers participated in the study (46 control group; 48 experimental group). Geographically, the teachers were located throughout the United States with 30% of teachers located in the Southern census region of the United States, 30% in the Midwest, 21% in the Northeast, and 18% in the West. One teacher (1%) was located in Canada. Twenty-three percent of teachers reported teaching one class, 22% reported teaching 2 classes, 21% reported administering the EVERFI PDS Curriculum in 3 classes, 9% reported administering it in 4 classes, 21% reported administering it in 5 or more classes, and 4% were unsure. Across all classes, teachers reported teaching an average of 74.33 (SD = 52.83) students during the semester.

Chi-square and independent sample t-tests were used to explore whether there were differences in teacher characteristics related to group assignment. When testing for regional differences, the teacher from Canada was removed. There were no significant group differences by teacher location (χ2 = 3.02, p = 0.39), number of classes taught (χ2 = 7.60, p = 0.11), or number of students served (t = -0.87, p = 0.39).

Student Pre/Post Sample

The current study relies on two overlapping samples of students—a “pre/post sample” and a “longitudinal sample.” The pre/post sample consisted of 2,325 (1251 control group; 1,074 experimental group) students who completed both a pre- and post-survey and was used to address RQ1 and RQ2. The longitudinal sample was used to address RQ3 and RQ4 and consisted of experimental group students who: 1) completed a pre or post survey and 2) completed a time 3 survey. A total of N = 1,257 students from the experimental group met the inclusion criteria for the longitudinal sample.

We compared students who completed both a pre-and post-survey to those who did not in terms of gender, age, grades in school, race, and prior prescription drug abuse or misuse. Results suggested that, students who completed both a pre- and post-survey were younger (χ2 = 59.21, p = 0.001), had higher grades in school (χ2 = 115.08, p = 0.001) and less likely to be Hispanic (χ2 = 80.26, p < 0.001) than students who did not complete both a pre- and post-survey. These findings were replicated when we compared students who completed both a pre-and post-survey to those who did not separately in the experimental and control groups. Although we observed differences between students who completed both a pre- and post-survey and those who did not, we accounted for potential biases introduced by controlling for these differences statistically in our modeling technique (described under “Analysis Plan” below).

Demographic information for the pre/post sample was collected from students during the pre-survey. Forty-two percent of students identified as male, 49% as female, 3% identified as non-binary, 3% preferred not to answer and 3% did not report their gender. Eighteen percent of students were in 8th grade, 36% in 9th grade, 19% in 10th grade, 10% in 11th grade, 14% in 12th grade, and 4% did not report their year in school. Forty-eight percent of students had grades of Mostly A’s in school, 43% had grades of Mostly B’s or lower and 9% did not report their grades. Students identified their race/ethnicity 49% were White, 17% Latinx, 13% Black, 5% Asian, 1% American Indian or Alaska Native, 5% Multiracial, 5% Other, and 5% did not report their race/ethnicity. Six percent of students reported prior prescription drug abuse or misuse. Compared to the national population of youth ages 10–19, our sample was similar in terms of gender and race/ethnicity, but with fewer youth identifying as Latinx (25% in the United States compared to 17% in our sample; Act for Youth, 2022).

We assessed baseline equivalence of the experimental and control groups by examining chi-square models in which student characteristics (gender, year in school, academic grades, race/ethnicity, prior exposure to prescription drug education, personal knowledge of an individual impacted by prescription drug abuse or misuse, ease of prescription drug access, and prior prescription drug abuse or misuse) were used to predict experimental group status. Results suggested that students in the experimental group were older (χ2 = 179.88, p < 0.001), had lower grades (χ2 = 10.15, p = 0.001), were more likely to be Black (χ2 = 16.07, p < 0.001), Latinx (χ2 = 44.68, p < 0.001), American Indian or Alaska Native (χ2 = 17.73, p < 0.001) or Asian (χ2 = 12.94, p < 0.001) versus White, and were more likely to have abused or misused prescription drugs in the past (χ2 = 13.44, p < 0.001). Both samples were equivalent by student gender, prior exposure to prescription drug education, personal knowledge of an individual impacted by prescription drug abuse or misuse, and ease of prescription drug access. Results of independent samples t-tests suggested that on the pre-survey, the experimental group scored lower than the control group on Personal Responsibility (t = 2.12, p < 0.05), Social Norms (t = 4.23, p < 0.01), Refusal Skills (t = 4.74, p < 0.01), and Future Actions (t = 4.04, p < 0.01). Tables 1 and 2 provide descriptive statistics broken down by experimental group status. Although we observed baseline differences between the experimental and control groups, we eliminated their impact on study findings by controlling for these differences statistically in our modeling technique (described under “Analysis Plan” below).

Table 1 Student demographic characteristics in the control group, experimental group and longitudinal sample
Full size table
Table 2 Student prescription drug history in the control group, experimental group and longitudinal sample
Full size table

Student Longitudinal Sample

The longitudinal sample consisted of N = 1257 experimental group students who completed a time 3 survey and a pre or post survey. Forty-three percent of students identified as male, 51% as female, 7% identified as non-binary. Four percent of students were in 8th grade, 41% in 9th grade, 19% in 10th grade, 16% in 11th grade, and 20% in 12th grade. Forty-five percent of students had grades of Mostly A’s in school and 55% had grades of Mostly B’s or lower. Students identified their race/ethnicity as 34% White, 28% Latinx, 14% Black, 5% Asian, 1% American Indian or Alaska Native, 13% Multiracial, and 5% Other. Six percent of students reported a prior history of prescription drug misuse. Table 1 provides descriptive statistics of the longitudinal sample.

Measures

Below we describe our measures of student characteristics, five key outcomes, and implementation characteristics.

Student Demographic Characteristics

Students reported their gender as 0 = Male, 1 = Female or 2 = Non-Binary. Students also indicated their year in school as 1 = “8th Grade” through 5 = “12th Grade.” Given that 48% of students reported receiving “Mostly As”, academic grades were measured dichotomously as 1 = “Mostly A’s” and 0 = “All other grades.” Students identified their race/ethnicity as White, Latinx, Black, Asian, American Indian or Alaska Native, Multiracial, or Other. Students were given a code of “1” for prior history of prescription drug misuse or abuse if they indicated they had ever misused or abused prescription opiates, stimulants, or depressants and a code of “0” if they reported they had never misused or abused any of these prescription drugs. For each category of prescription drugs, a list of sample medications was provided to aid students’ responses.

Student Prescription Drug History

Students were asked whether they had heard about the problem of prescription drug abuse through any of eight sources (family members, friends, at school, newspapers or online news, television or radio, public service announcements on television or radio, public service announcements online, social media). A variable representing prior exposure to prescription drug education was created by counting the number of positive endorsements of these items—if a student had 3 + prior exposures, they were given a score of “1.” If a student had fewer than 3 exposures, they were given a score of “0.” Students were identified as being personally affected by prescription drug misuse if they endorsed “Yes'' to the prompt, “I personally know someone who has misused or abused prescription drugs.”

Students were asked how easy it would be for them to obtain prescription opiates, stimulants, or depressants. Their ease of access was coded as “1” if they indicated it was “Very Easy” or “Easy” to obtain these prescription drugs or “0” if they indicated obtaining these drugs was “Difficult” or “Impossible.” Students received a score of “1” if they indicated that they believed prescription drug misuse was a problem in their community and a score of “0” if they reported prescription drug misuse was NOT a problem in their community. Descriptive statistics of student prescription drug history variables for the experimental, control, and longitudinal sample can be found in Table 2.

Student Outcomes

Personal Responsibility (Cronbach’s α = 0.79) was defined as a sense of responsibility for helping others in their community who may be at risk for abusing or misusing prescription drugs. This outcome was measured using four items on a 7-point scale (1 = “Strongly Disagree” to 7 = “Strongly Agree”). A sample item was, “It is my responsibility to help prevent prescription drug misuse at my school.”

Social Norms (α = 0.84) was defined as the perception of the prevalence of prescription drug abuse and peer engagement in preventing prescription drug abuse and misuse. This outcome was assessed using four items reported on a 7-point scale (1 = “Strongly Disagree” to 7 = “Strongly Agree”). A sample item was “Most students at my school would offer support to a friend who was abusing prescription drugs.”

Knowledge (α = 0.70) was defined as the understanding of addiction, safe use of prescriptions, and refusal skills. This outcome included four items and was reported on a 7-point scale (1 = “Strongly Disagree” to 7 = “Strongly Agree”). A sample item was “I can identify the signs of prescription drug abuse and misuse.”

Future Actions (α = 0.78) was the student-reported likelihood of future prescription drug abuse or misuse. This outcome included two items: “Do you think you will be misusing or abusing prescription drugs one year from now?” and “Do you think you will be storing and disposing prescription drugs appropriately one year from now?” that were measured on a 4-point scale (1 = “I definitely will” to 4 = “I definitely will not”). Because these items asked students to report about their anticipated behaviors within the next year, we did not re-administer these items at time 3, given the timeline of the question (i.e., “in the next year”) would overlap significantly between the post-survey and time 3, therefore limiting the likelihood that students' responses would change.

Refusal Skills were measured using a single item, “I would be able to avoid misusing prescription drugs in a situation where they were offered to me,” with a response scale that ranged from 1 = “Strongly Disagree” to 7 = “Strongly Agree”. Sample group scores on the five key outcomes for the Control Group and the Experimental Group (both the pre/post sample and longitudinal sample) can be found in Table 3.

Table 3 Average scores in the five key outcomes among control group, experimental group and longitudinal sample
Full size table

Implementation Characteristics

Variation in implementation of the PDS curriculum was expected and teachers were asked a series of questions designed to assess dosage, quality of administration, student engagement, and teacher experience with the PDS curriculum. Dosage was assessed by counting the number of items teachers endorsed from a list of four activities teachers may have engaged in while teaching the PDS curriculum (e.g., “I gave students time to discuss the course materials in class”). This variable ranged from 0—4.

Quality of Administration (α = 0.91) was measured with six items assessing the teacher’s comfort and engagement in teaching the material (e.g., “I was enthusiastic about the material”). Responses ranged from 1 = “Strongly Disagree” to 5 = ”Strongly Agree.”

Teachers’ perceptions of student engagement (α = 0.84) was measured using five items that assessed how students engaged with the PDS curriculum (e.g., “My students appeared to be genuinely interested in the course”). The response scale ranged from 1 = “Strongly Disagree” to 5 = “Strongly Agree.”

Finally, teachers were given a code of “1” on our measure of teacher experience if they had ever taught the PDS curriculum before and a code of “0” if this was their first time teaching the curriculum.

Analysis and Results

The corresponding author, Dr. Alicia Lynch, takes responsibility for the integrity of the data and the accuracy of the data analysis.

Analysis Plan

Prior to running analyses, we reviewed measures of central tendency, variance, and normality and removed 19 invalid responses (straight-line responses, illogical or inconsistent responses). Students who failed to complete at least 50% of their pre- or post-surveys were removed from the dataset due to concerns that these individuals may have been distracted or disengaged during survey completion. Under these criteria, N = 550 students were removed from the pre/post sample and N = 78 were removed from the longitudinal sample. Note that the samples described in the above sections reflect the final samples after these observations had been removed. Patterns of missing data suggested data were Missing at Random (MAR); accordingly, we employed Full Information Maximum Likelihood (FIML) estimation to account for missing data in the analyses. All analyses were conducted in the statistical software package, Stata 16.0 (StataCorp, 2019).

We also conducted psychometric testing of all scales including an examination of inter-item correlations and Cronbach’s alphas. The alphas for all scales (reported above) were greater than 0.70. Global Reliability of Change (RCI) scores for Knowledge, Social Norms and Personal Responsibility, and Refusal Skills were all greater than 1.96, ranging from 4.8 to 8.5. On average, 64% of students in the experimental group had individual RCI scores higher than the global RCI for each outcome, suggesting that observed changes between the pre- and post-surveys are practically significant and reliable. The RCI for Future Actions, however, was −1, suggesting that changes in this measure may not be reliably different from zero. We have highlighted this limitation in our discussion of findings related to Future Actions.

The smallest unit of analysis to assess intervention effects was the individual. Due to the nested nature of the data, a series of three-level Multilevel Models (MLM) with time (level 1) nested within students (level 2) nested within teachers (level 3) were used to examine links between the PDS curriculum and growth in the five key student outcomes (Personal Responsibility, Knowledge, Social Norms, Future Actions, and Refusal Skills). In each model, we used covariate adjustment (Elze et al., 2017) to account for student characteristics that were not equivalent across the experimental and control groups at baseline. Although traditionally analysts use an alpha of 0.05 as a cut off to identify statistically significant results, because we were testing multiple models (and therefore had an increased risk of type 2 error), we employed a Bonferroni correction and reduced our required alpha for each set of analysis by dividing 0.05 by the number of models being tested in each research question. For example, RQ1 was examined using five separate models. Therefore, the alpha cut off we used to determine statistical significance for RQ1 became 0.05/5 = 0.01.

Effect sizes in standard deviation units were calculated based on adjusted regression coefficients from the multilevel models (Snijders & Bosker, 2012) to determine the magnitude of the effect and were interpreted such that effect sizes below 0.20 were considered “small effects,” effect sizes 0.20-0.50 were considered “medium effects” and effect sizes > 0.50 were considered “large effects” (Tymms, 2004). There are very few studies that examine the effectiveness of technology-based prescription drug misuse prevention courses. One study of a prescription opioid misuse prevention program (POP4Teens) found moderate within-condition effects (Cohen’s d = 0.30; calculated by current manuscript authors based on data provided in original article) on growth in opioid-related knowledge one month after baseline (Marsch et al., 2021). When considering technology-based programs that target other types of substance use (e.g., alcohol, cannabis, cigarettes), research suggests that programs have mixed results (see Schinke & Schwinn, 2017 for a comprehensive review). For example, a 2014 study of the effectiveness of the “Healthy School and Drugs” prevention curriculum (Malmberg et al., 2014) suggested the program had no effects on student substance use. A 2013 study of “What Do You Drink” among heavy drinking adolescents suggested a small reduction in alcohol consumption at one month (Cohen’s d = 0.06). Alternately, “Climate Schools: Ecstasy and Emerging Drugs” produced large increases in knowledge of new psychoactive substances of (Cohen’s d = 0.70; Champion et al., 2016).

To address RQ1, we used the pre/post sample to consider whether students in the experimental group demonstrated different rates of growth in the five key outcomes than students in the control group by observing the strength and direction of the interaction between time and experimental group status. To test whether student demographic or prescription drug history characteristics moderated the relationship between participation in the PDS curriculum and growth in the five key outcomes (RQ2), we examined the strength and direction of three-way interactions among experimental group status, time, and the student characteristic of interest. We explored whether growth in the five key outcomes was maintained up to one month following program completion (RQ3) using data from the longitudinal sample. To test whether implementation characteristics moderated the relationship between participation in the PDS curriculum and growth in the five key outcomes (RQ4), we examined the strength and direction of two-way interaction terms between time and the teacher characteristic of interest.

Results

RQ1: Do Students Demonstrate Growth in the Five Key Outcomes Between the Pre-Survey and Post-Survey?

As described above, to address whether participation in the PDS curriculum was associated with growth (i.e., growth between the pre- and post-survey) in the five key outcomes, we examined the results of five separate multilevel regression models (one per outcome) within our pre/post sample. Results suggested that between the pre- and post-survey, students in the experimental group demonstrated significantly more growth in Personal Responsibility (B = 0.24, p < 0.001, Effect Size = 0.18; See Fig. 3a), Social Norms (B = 0.15, p < 0.01, Effect Size = 0.11; See Fig. 3b), Knowledge (B = 0.20, p < 0.001, Effect Size = 0.16; See Fig. 3c), and Future Actions (B = 0.07, p < 0.01, Effect Size = 0.14; See Fig. 3d). Students’ growth in Refusal Skills (B = −0.04, p = 0.56, Effect Size = -0.03) did not reach statistical significance. Table 4 provides complete model results for all five outcomes.

Fig. 3
figure 3

a Covariate adjusted predicted growth in Personal Responsibility in the experimental and control groups between the pre- and post-survey b Covariate adjusted predicted growth in Social Norms in the experimental and control groups between the pre- and post-survey c Covariate adjusted predicted growth in Knowledge in the experimental and control groups between the pre- and post-survey d Covariate adjusted predicted growth in Future Actions in the experimental and control groups between the pre- and post-survey

Full size image
Table 4 Results for models addressing RQ1: Do students demonstrate growth in the five key outcomes between the pre-survey and post-survey?
Full size table

RQ2: Is Growth in the Five Key Outcomes Moderated by Student Demographic Characteristics or Student Prescription Drug History?

Next, we considered whether student characteristics moderated the relationship between experimental group status and growth in the five key outcomes. None of the student demographic or prescription drug history characteristics we examined moderated the effect of the PDS curriculum on growth in the five key outcomes. That is, the PDS curriculum worked similarly for all students, regardless of demographic and background characteristics.

RQ3: Is Observed Growth in the Five Key Outcomes Maintained One Month After Completion of the PDS Curriculum?

To examine whether the effects of the PDS curriculum persist up to a month after course completion, we explored two sets of models among our longitudinal sample. For each set of models, we again used an alpha of 0.01 as a cutoff for identifying statistically meaningful results. In the first set of models, we used all three timepoints (pre, post, and time 3) and found that for Personal Responsibility (B = 0.08, p < 0.001, Effect Size = 0.11), Social Norms (B = 0.12, p < 0.001, Effect Size = 0.17), and Knowledge (B = 0.17, p < 0.001, Effect Size = 0.27), the slope of time remained significant when time 3 was added to the model. Growth in Refusal Skills was marginally significant (B = 0.06, p = 0.05, Effect Size = 0.07), but did not meet our criteria for identifying statistically meaningful results (cutoff alpha = 0.01). Complete model results are available in Table 5.

Table 5 Results for models addressing RQ3: Is observed growth in the five key outcomes maintained one month after completion of the PDS curriculum?
Full size table

In the next set of models, we looked specifically at growth in Personal Responsibility, Social Norms, Knowledge, and Refusal skills between the post-survey and time 3 only (i.e., we removed pre-survey scores from the model). Results suggested that growth in Personal Responsibility (B = −0.02, p = 0.68, Effect Size = 0.02), Social Norms (B = −0.05, p = 0.42, Effect Size = 0.06), and Knowledge (B = 0.10, p = 0.10, Effect Size = 0.12) remained stable between the post-survey and time 3. That is, there was no observed increase or decrease in students' scores between the post-survey and time 3. Interestingly, although Refusal Skills had only reached marginal significance in the prior models, in the current model, we observed statistically meaningful growth in Refusal Skills (B = 0.27, p < 0.001, Effect Size = 0.32), suggesting a longer effect latency—students’ Refusal Skills did not increase immediately after the program, but did increase one month later.

RQ4: Are Implementation Characteristics Related to Variation in Student Growth in the Five Key Outcomes?

To address RQ4, we again employed data from the longitudinal sample. Using the same multilevel regression models described under RQ3, we added interaction terms between the slope of the time and our measures of implementation characteristics (Dosage, Quality of Administration, Engagement, and Teacher Experience) both in separate models as well as simultaneously. Model results suggested that higher Engagement predicted lower growth in Social Norms over time (B = -0.17, p < 0.01; see Fig. 4). There was an additional, marginally significant finding suggesting that students of teachers who had taught the PDS curriculum more than once demonstrated more growth in Knowledge than students whose teachers were administering the PDS curriculum for the first time (B = 0.26, p = 0.027). In all, these results suggested that, for the most part, students demonstrated similar growth in the key outcomes regardless of implementation characteristics.

Fig. 4
figure 4

Student engagement moderates the effectiveness of the PDS curriculum on Social Norms. Note Numbers in figure reflect covariate adjusted predicted means

Full size image

Discussion

This quasi-experimental evaluation demonstrated that youth who complete the EVERFI Prescription Drug Safety (PDS) curriculum show growth in Personal Responsibility, Social Norms, Knowledge, and Future Actions surrounding prescription drug safety. We considered gender, race, year in school, and grades as potential moderators of program effectiveness, as well as students’ prior experience with prescription drug misuse (PDM). Although extant literature points to mixed results for student-level moderators of prevention program effectiveness (Fagan & Lindsey, 2014; Rowe & Trickett, 2018), our findings suggest that growth in student outcomes was equitable across these demographic characteristics.

In addition to examining growth between the pre- and post-survey, the present study examined whether growth in student outcomes would be maintained one-month after completing the PDS curriculum. Results provided evidence for retention in Personal Responsibility, Social Norms, and Knowledge growth. Students’ Refusal Skills did not increase immediately after the program, but did increase one month later. Past research has shown long-term reduction in PDM among family- and community-based interventions that encompass a whole-child approach to wellbeing (e.g., enhancing life skills and family functioning; reducing substance misuse and conduct problems; Spoth et al., 2013). Nevertheless, these programs can be difficult to implement and replicate in diverse populations (Gottfredson et al., 2006). The EVERFI PDS curriculum evaluated in the present study targets prescription drug safety specifically and demonstrates stronger retention than some of the whole-child approaches. Effect sizes for outcomes demonstrating statistically significant growth in the current study ranged from 0.11–0.27, aligning with effect sizes observed in studies of other technology-based substance use curriculum (Marsch et al., 2021; see Schinke & Schwinn, 2017 for a comprehensive review). The small and moderate effect sizes found here should be considered in light of the ease of implementation of the EVERFI program, which leverages web-based technology, is relatively brief, requires little or no training, and is free to administer.

Similar to other prevention programs, the classroom dynamic in which the EVERFI PDS curriculum was administered was associated with variation in student outcomes. First, findings suggested that, although all students demonstrated growth in Social Norms from pre- to post-survey, higher teacher perceptions of student engagement (such as, “My students appeared to be genuinely interested in the course”) was associated with slower rate of growth in Social Norms compared to lower teacher perceptions of student engagement. It could be that students who demonstrated engagement in the material already had an accurate understanding of PDM norms, and were therefore less likely to demonstrate growth in norms. It is possible that the curriculum does not provide novel information regarding social norms for highly engaged students. Other teacher-centric aspects of the classroom dynamic, including dosage, quality of administration, and prior experience with the curriculum, were not associated with student growth in the five key outcomes. Future research might include observational assessments of implementation or incorporate qualitative data to elucidate nuances in variation of PDS curriculum implementation.

We also examined whether student characteristics, including year in school, academic grades, race/ethnicity, prior to exposure to prescription drug messaging or use, and history of misuse were related to variation in growth in the four key outcomes. Results of these analyses suggested that students demonstrate similar growth regardless of these individual factors and that program efficacy may be maintained across diverse populations. We interpret the consistent findings across subgroups of students as promising in terms of the program’s generalizability across various populations within the United States.

The primary limitation of the present study is that it relied on self-report student assessments of PDM, rather than observational or other-reported measures. Although intentions are closely related to actual behaviors (Ajzen, 1991), the Future Action items asked students about their future intentions to engage in safe prescription drug behavior and did not directly observe such behaviors. Moreover, it is difficult to attribute growth measured for the Future Actions outcome to the PDS curriculum. For example, if students already behaviorally refrain from taking prescription drugs when offered, it is not possible for them to demonstrate growth in that area. Thus, our measure did not account for the fact that most students are currently already far enough along that continuum of potential growth. These challenges are captured in the RCI score calculated for Future Actions, which suggest that observed changes in the measure may not be reliable. Measures of future actions and intentions should be less hypothetically future-oriented and incorporate objective indicators of prescription drug safety behaviors. A second limitation: given the practical limitations of the program, we were only able to assess student outcome retention after one month. Within our sample, teacher and student schedules were constrained by academic quarters, trimesters, or semesters, limiting the time available to administer the time 3 survey. Ideally, we would have a longer period between the intervention and follow up to assess persistence of the effect.

In summary, our evidence indicates that the EVERFI PDS curriculum is an effective, no cost intervention to promote Personal Responsibility, Social Norms, Knowledge, and Refusal Skills in relation to prescription drug misuse. As a relatively brief, school-based intervention with potential to target a large number of students, the PDS curriculum provides an efficient and economical method for promoting growth in prescription drug safety attitudes and intentions equitably across diverse student populations.