ÌÇÐÄvlog

Key Points

QuestionÌý Does a combination of parent reminder/recall and health care professional audit/feedback improve human papillomavirus (HPV) vaccine uptake among 11- to-12-year-old children?

FindingsÌý This stepped-wedge factorial, cluster randomized trial included 6 primary care practices over 4-year-long steps with 9242 adolescents, all due for at least 1 dose of HPV vaccine. Interventions included parent reminder/recall and health care professional audit/feedback; both interventions together resulted in a statistically significant increase in HPV vaccine uptake compared with usual care.

MeaningÌý Practices should consider adopting both parent reminder/recall and health care professional audit/feedback to improve HPV vaccine uptake.

ImportanceÌý Despite availability of a safe and effective vaccine, an estimated 36 500 new cancers in the US result from human papillomavirus (HPV) annually. HPV vaccine uptake falls short of national public health goals and lags other adolescent vaccines.

ObjectiveÌý To evaluate the individual and combined impact of 2 evidence-based interventions on HPV vaccination rates among 11- and 12-year-old children.

Design, Setting, and ParticipantsÌý The study team conducted a cluster randomized clinical trial with a stepped-wedge factorial design at 6 primary care practices affiliated with Mayo Clinic in southeastern Minnesota. Using block randomization to ensure balance of patient volumes across interventions, each practice was allocated to a sequence of four 12-month steps with the initial baseline step followed by 2 intermediate steps (none, 1, or both interventions) and a final step wherein all practices implemented both interventions. Each month, all eligible children who turned 11 or 12 years in the 2 months prior were identified and followed until the end of the step. Data were analyzed from April 2018 through March 2019. Participants included children who turned 11 or 12 years old and were due for a dose of the HPV vaccine.

InterventionsÌý Parents of eligible patients were mailed reminder/recalls following their child’s birthdays. Health care professionals received confidential audit/feedback on their personal in-office success with HPV vaccine uptake via intra-campus mail. These 2 interventions were assessed separately and in combination.

Main Outcomes and MeasuresÌý Eligible patients’ receipt of any valid dose of HPV vaccine during the study step.

ResultsÌý The cohort was comprised of 9242 11-year-olds (5165 [55.9%]) and 12-year-olds (4077 [44.1%]), and slightly more males (4848 [52.5%]). Parent reminder/recall resulted in 34.6% receiving a dose of HPV vaccine, health care professional audit/feedback, 30.4%, both interventions together resulted in 39.7%—all contrasted to usual care, 21.9%. Compared with usual care, the odds of HPV vaccination were higher for parent reminder/recall (odds ratio [OR], 1.56; 95% CI, 1.23-1.97) and for the combination of parent reminder/recall and health care professional audit/feedback (OR, 2.03; 95% CI, 1.44-2.85). Health care professional audit/feedback alone did not differ significantly from usual care (OR, 1.19; 95% CI, 0.94-1.51).

Conclusions and RelevanceÌý In this cluster randomized trial, the combination of parent reminder/recall and health care professional audit/feedback increased the odds of HPV vaccination compared with usual care. These findings underscore the value of simultaneous implementation of evidence-based strategies to improve HPV vaccination.

Trial RegistrationÌý ClinicalTrials.gov Identifier:

The US Centers for Disease Control and Prevention estimate 36 500 human papillomavirus (HPV)-attributable cancers occur each year in the US,¹ the Advisory Committee on Immunization Practices (ACIP) recommends HPV vaccination for children aged 11 to 12 years with permission to initiate at 9 years with catch-up through age 26 years.² Only 61.7% of US adolescents aged 13 to 17 years have completed the series—lagging public health goals and other adolescent vaccines.^3,4

The ACIP prioritized the need for health care professionals to strengthen their recommendations for HPV vaccinations.^5,6 Patient reminder/recall and health care professional audit/feedback support recommendation of the HPV vaccine and have consistently been shown to increase HPV vaccination.^7,8 Reminder/recall alerts patients or parents of children due or past due for vaccination.^7-17 Audit/feedback informs health care professionals about their vaccination performance.^12,16,18-21 Given that all of the practices in our study had already adopted other effective strategies to improve HPV vaccination (ie, starting vaccination at age 9 years, pain preventive measures, standing orders to support nurse-only visits, and point-of-care prompts), we sought to ascertain the independent and synergistic impact of implementation of these evidence-based interventions on HPV vaccination rates.²² We evaluated the following hypotheses:(1) compared with usual care, parent reminder/recall will improve the odds of an adolescent receiving an HPV vaccine dose by at least 20%; (2) compared with usual care, health care professional audit/feedback will improve the odds of an adolescent receiving an HPV vaccine dose by at least 20%; and (3) simultaneous implementation of reminder/recall and audit/feedback will have a synergistic effect on HPV vaccine uptake, more than doubling odds of HPV vaccination.

The protocol for this trial was registered and published (Supplement 2). The Mayo Clinic institutional review board approved the trial as minimal risk and informed consent was waived. We evaluated individual and combined effects of the 2 evidence-based interventions on HPV vaccination rates. We used a stepped-wedge design to test the presence of each intervention in each practice. This design allowed each practice to serve as its own control, thereby reducing potential bias due to imbalanced risk factors across practices.²³ The stepped-wedge allocation naturally incorporates a factorial design, which enabled evaluation of the impact of the interventions individually and in combination within a single trial.²⁴ This design is generally more efficient than a parallel randomized design, conserving sample size while maintaining statistical power.²⁵

The trial used 4 steps, each 12 months long (Figure 1). The first step began on April 1, 2018. We postponed initiation of step 3 (originally planned for April 1, 2020) until September 1, 2020, because of the disruption in health care early in the COVID-19 pandemic. Step 4 concluded August 31, 2022. All analyses were intention-to-treat.

We conducted the trial in 6 Mayo Clinic primary care practices staffed with salaried pediatricians, family physicians, advanced-practice health care professionals, and resident-trainees in southeastern Minnesota. All practices provide care to empaneled patient populations who are assigned to a specific clinical practice and a specific health care professionals.

During each trial month and at each practice, we identified age- and dose-eligible patients using ACIP guidelines.² To be age eligible for a particular trial month, the patient must have turned 11 or 12 years of age during the second calendar month prior to the given trial month. We targeted this age group per ACIP guidelines and American Academy of Pediatrics recommendations to vaccinate at ages 11 to 12 years, which aligns with other adolescent platform vaccines. Thus, our focus on 11 and 12 years of age was in recognition of the age recommended officially and what typical practices around the US are doing.

The HPV vaccination status of each patient was assessed starting the month following their 11th or 12th birthday (ie, the month prior to the relevant trial month). For each trial month, we assessed whether the age-eligible group was dose eligible.²⁶ We determined eligibility for any dose (initiation and/or completion of the HPV vaccine series) by using the medical record to ascertain receipt of valid doses.^26,27 Additional details on eligibility are included in eMethods 1 in Supplement 1.

A patient empaneled to a practice who was both age and dose eligible for one step may have been age and dose eligible the following step. We included adolescents in a step who were age and dose eligible for a preceding step who remained empaneled to a practice and found to remain age and dose eligible in the next step.

Our numerators included counts of empaneled, age, and dose eligible adolescents who received a dose of HPV vaccine during the timespan from age eligibility until the end of the step. Thus, some children had 12 months to receive a dose, some 11 months, and so on, with those children who became eligible in the last month of the step only having that month to receive a dose.

Our factorial design consisted of 6 sequences of four 12-month steps. We allocated each practice to one 12-month sequence, as illustrated in the CONSORT diagram (Figure 2). We grouped the 6 practices into 3 pairs according to patient volume and used block randomization of pairs to ensure balance of patient volumes across the interventions. Step 1 was usual care for all sequences. Initiation of step 3 was delayed due to the COVID-19 pandemic (Figure 1). The allocation sequence was developed by and known only to the statistical team until point of allocation. Interventions were randomized at the site level and implemented as part of clinical care and quality improvement processes; therefore, patients and clinicians were blinded to the interventions and assent/consent was not required.

All 6 practices participated in Vaccines for Children, used the state immunization registry for immunization reconciliation, held team huddles, offered nurse visits for vaccination (standing orders), initiated HPV vaccination beginning at age 9 years, routinely offered pain preventive measures, such as vapor-coolant sprays and lidocaine creams, and had point-of-care prompts for vaccines due. Prevalence and incidence rates of HPV vaccination series initiation and completion among patients empaneled to the practices included in our study in the years prior to initiation of our interventions were below public health goals and previously published.²⁸ In 2016, among those ages 11 to 12 years, the average vaccine series initiation prevalence rate was 51.4% and the completion prevalence rate was 21.5%; among those ages 13 to 17 years, the average initiation prevalence rate was 65.0% and completion rate 40.2%. In 2016, among those ages 11 to 12 years, average incidence of vaccination series initiation was 24.1% and completion was 9.6%; for those ages 13 to 17 years, initiation incidence was 15.3% and completion was 9.1%.

Before allocation of any practices to intervention, we conducted individual and focus group interviews with parents of age- and dose-eligible empaneled patients to obtain feedback on the clarity, understandability, and acceptability of the parent reminder/recall intervention. Parents preferred the reminders to be sent by postal mail rather than by text or patient portal messaging and to be alerted to all vaccinations needed (not just the HPV vaccine). Parents also expressed interest in having the reminder/recall letters provide sources for additional information. The reminder/recall intervention was modified per parent input. The reminder/recall also used presumptive language to convey a strong recommendation for the vaccines due,^29,30 described availability of topical anesthetics to reduce injection pain, and emphasized the convenience of nurse visits. The final version is included as eMethods 2 in Supplement 1.

We conducted focus group interviews with health care professionals to obtain feedback on audit/feedback reports. Findings from the health care professional focus group interviews, informed our approach. We used confidential intra-campus mail to distribute the reports. Reports included rates of successful receipt of HPV vaccine among patients seen by that health care professional rather than the inverse of missed opportunities and we included information about access to online training materials for making strong recommendations and addressing vaccine hesitancy. Reports summarized HPV vaccination rates averaged for the prior 3 months for the individual health care professional and for their specific clinical practice calculated using the counts of HPV vaccine doses given as the numerator and the number of visits during which they saw 9- to 26-year-old patients eligible for a dose of HPV vaccine as the denominator. We included all ages for which an HPV vaccine may be due as we wanted to nudge health care professionals to make a strong recommendation at every encounter in which an HPV vaccine was due.

Each audit/feedback report provided information for accessing a toolkit to support making strong recommendations using presumptive language^29,30 and to address vaccine hesitancy using the Corroborate, About Me, Science, Explain/Advice approach.^31,32 The toolkit was made available online only to staff in practices allocated to the intervention. Supervisors and research staff communicated the availability of the toolkit through email. The audit/feedback report with links to the toolkit content is included as eMethods 3 in Supplement 1.

For each step, we extracted demographic data (race and ethnicity, sex, and age) for eligible patients from the electronic medical record. We also extracted occurrences and dates of all HPV vaccinations for these adolescents using Current Procedural Terminology codes (90649, 90650, and 90651). The primary outcome was the receipt of at least 1 dose of HPV vaccine for the age- and dose-eligible children between the first day of their age- and dose-eligible month until the last day of the corresponding 12-month step.

Statistical power was estimated using simulation, which accounted for potential differences in baseline rates across sites.³³ Based on pilot data from a 12-month period, we assumed an average cluster size of 800 children (400 males and 400 females), a baseline probability of requiring any vaccination of 25% (log-odds of −1.1), and a conservative estimate of the SD of the random effect of 0.5, assuming a secular treatment effect. Under these assumptions, we estimated having 90% power to detect a true odds ratio (OR) of 1.25 across all cohorts and 87% power to detect a true odds ratio of 1.35 within either the male or female subcohorts. Methodological details and results of various power scenarios have been previously published.²⁸

We summarized demographics and eligibility group (vaccine series initiation or completion) by intervention status. To assess whether randomization was successful in balancing age, sex, and race and ethnicity, we reported the maximum standardized mean difference (SMD) across the 6 contrasts of the 4 intervention states. Because follow-up time varied according to the month of eligibility, we also summarized days from the first day of the eligibility month to the end of the step. All patients were analyzed on an intention-to-treat basis.³⁴ We used generalized linear mixed models to assess the effects of the interventions overall and stratified by sex.³⁴ Our main model was a mixed-effects logistic regression model as detailed in the protocol.²⁸ Specifically, for each cohort, we estimated a single mixed-effects logistic model where vaccination status was the outcome; each model included an indicator for intervention status (usual care omitted as reference), indicators for study steps (step 1 omitted as reference), and a random effect for practice. For each model, we reported the ORs corresponding to the 3 intervention indicators, with 95% CIs, and the Wald P value for the joint hypothesis that the 3 indicators are 0. For mixed-effects models, we reported between-practice variance. For secondary analyses, we estimated models similar to the main model, using only HPV vaccine series initiation and completion-eligible patients. We conducted 2 sensitivity analyses. First, we reestimated the primary and secondary models adjusting for factors that were imbalanced across intervention groups. Second, because of the small number of practices, estimation of between-practice variance may give biased results.³⁵ Therefore, we replicated the primary and secondary models using fixed-effects models. Fixed-effects models account conservatively for clustering of outcomes and have been shown to be least biased and more efficient than other models when there are small clusters and no cluster-level predictors.³⁶ We performed all analyses using Stata version 17 (StataCorp).

The final cohort (n = 9142) was comprised of 55.9% 11-year-olds, 41.1% 12-year-olds, and was 52.5% male (Table 1). Parent reminder/recall resulted in 34.6% receiving a dose of HPV vaccine, health care professional audit/feedback resulted in 30.4% receiving a dose of HPV vaccine, both interventions together resulted in 39.7% receiving a dose of HPV vaccine—all contrasted with usual care, 21.9%. Distributions of demographic and vaccine-status characteristics (initiation vs completion, dose eligibility, and average number of days of follow-up to assess vaccination status) were similar across the 4 intervention groups (usual care, reminder/recall, audit/feedback, and both), with the maximum SMDs ranging from 0.024 to 0.278, suggesting, at most, modest imbalance.³⁷

Proportions of age- and dose-eligible children receiving at least 1 HPV vaccine during the follow-up period for each intervention are summarized overall, by sex, and by vaccine due (initiation or completion) in Table 2. Among all age- and dose-eligible patients, 782 of 3572 received an HPV vaccine during usual care steps (21.9%), 578 of 1670 received an HPV vaccine during parent reminder/recall steps (34.6%), 408 of 1340 during health care professional audit/feedback steps (30.4%), and 1056 of 2660 during steps with simultaneous reminder/recall and audit/feedback (39.7%) (Table 2).

The odds of HPV vaccination among children exposed to reminder/recall were significantly higher (OR, 1.56; 95% CI, 1.23-1.97) than for usual care. Audit/feedback alone did not significantly increase odds of vaccination relative to usual care (OR, 1.19; 95% CI, 0.94-1.51); however, the combination of reminder/recall and audit/feedback more than doubled the odds of vaccination over usual care (OR, 2.03; 95% CI, 1.44-2.85). The overall P value for an effect was P < .001 (Table 3). When stratified by sex, odds of receiving a vaccination after audit/feedback compared with usual care were higher for males (OR, 1.39; 95% CI, 1.01-1.90) but not for females (OR, 1.04; 95% CI, 0.74-1.47). Other sex-stratified results resembled the unstratified effects (Tables 2 and 3). When stratifying by whether patients were initiating or completing the HPV vaccine series, the effects of the interventions were similar to the full cohort (Table 3). Adjusting the main models for race and eligibility status (initiation vs completion) gave very similar results (eMethods 4 in Supplement 1). Replicating the main analyses using fixed-effects models gave very similar results with uniformly smaller P values (eMethods 4 in Supplement 1). Between cluster variances ranged from 0.01 to 0.02 for random-effects models.

We evaluated the independent and combined effects of parent reminder/recall and health care professional audit/feedback on HPV vaccine uptake among patients empaneled to 6 primary care practices. Reminder/recall increased odds of HPV vaccination by more than 50% over usual care, exceeding the threshold specified in our hypothesis. The 19% increase in the odds of HPV vaccination for the health care professional audit/feedback intervention did not reach the hypothesized 20% increase. Consistent with our hypothesis, the concurrent implementation of both interventions had a synergistic effect on HPV vaccination rates, more than doubling the odds of a child receiving an HPV vaccine dose.

Males have consistently lagged females in HPV vaccine uptake.^3,38,39 Indeed, we observed a higher percentage of male patients as eligible and due for the HPV vaccine in each of our cohorts. Furthermore, we found the health care professional audit/feedback achieved a statistically significant increase for males but not females. Thus, audit/feedback in addition to parent reminder/recall may be a useful means to close the gap between male and female uptake of the HPV vaccine. The synergistic impact of concurrent use of audit/feedback and reminder/recall was consistently statistically and clinically significant for males and females and for vaccine series initiation and completion.

Our success is consistent with prior pragmatic trial evaluations of combined interventions to increase HPV vaccination. Fiks et al¹² found the combination of decision support for HPV vaccination for both families and health care professionals was more effective in improving uptake than either on its own. Dempsey et al⁴⁰ reported a successful strategy to improve vaccine uptake combining parental education regarding HPV disease and vaccines with health care professional education on making a strong recommendation and addressing HPV vaccine concerns. Perkins et al⁴¹ used a combination of health care professional-level and practice-level interventions to improve HPV vaccine uptake.

All practices included in this trial routinely initiate the HPV vaccine at age 9 years, so those aged 11 to 12 years were considered by the practices as due for HPV vaccination for 2 years or more by the time they became eligible for our interventions. Thus, children of parents who were vaccine hesitant or less likely to pursue vaccination may have been over-represented in our trial, suggesting further value of these interventions. Furthermore, we only measured the impact for the duration of the step, which was for the children a median of 6.5 months. The interventions may have continued to improve vaccine uptake beyond those first months.

Given the mathematical and analytical complexity of following either a closed or open cohort from step to step,^42,43 we included patients only as they turned 11 or 12 years old and followed them only until the end of that current step. This allowed us to attribute any vaccine during follow up to the relevant intervention. However, doses were not counted if they occurred after the step ended. While this approach may underestimate the crude rate of uptake, there is no reason to think it biases any effects toward or away from the null.

The observed effectiveness of the interventions may not be generalizable to other populations and clinical settings. Application and evaluation of evidence-based strategies in populations with greater racial and ethnic, geographic, and socioeconomic diversity is encouraged and efforts to scale up these interventions across more diverse clinical settings, are needed.

Our trial was under way during the COVID-19 pandemic. Early in the pandemic, the US Centers for Disease Control and Prevention recommended that health care professionals delay nonemergency and elective health care to minimize risk and preserve health care capacity. Several states, including Minnesota, issued stay-at-home orders. Thus, we delayed step 3 of the trial for a 6-month period. While significant declines in pediatric rates of preventive services were observed during the COVID-19 pandemic, especially during that 6-month period,⁴⁴ our model accounted for secular trends, and no data from the 6-month hiatus were included in our cohort.

In this cluster randomized trial, simultaneous implementation of patient reminder/recall and health care professional audit/feedback increased odds of HPV vaccination compared with usual care, offering an effective and scalable approach to improving HPV vaccination and reducing HPV-associated cancer burden. We recommend practices adopt reminder/recall to draw patients into the practice for encounters and audit/feedback to encourage use of every encounter to vaccinate. Future pediatric research may consider the utility of such interventions for other preventive measures.

Accepted for Publication: August 8, 2023.

Published Online: November 20, 2023. doi:10.1001/jamapediatrics.2023.4932

Corresponding Author: Lila J. Finney Rutten, PhD, Mayo Clinic, Harwick Building, 6th Floor, 200 First St SW, Rochester, MN 55905 (rutten.lila@mayo.edu).

Author Contributions: Drs Jenkins and Herrin had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Finney Rutten, Griffin, St. Sauver, Jacobson.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: Finney Rutten, Griffin, Austin, Jenkins, Jacobson.

Critical review of the manuscript for important intellectual content: Finney Rutten, Griffin, St. Sauver, MacLaughlin, Jenkins, Herrin, Jacobson.

Statistical analysis: Finney Rutten, Jenkins, Herrin, Jacobson.

Obtained funding: Finney Rutten, St. Sauver, Jacobson.

Administrative, technical, or material support: Finney Rutten, Griffin, MacLaughlin, Jenkins, Jacobson.

Supervision: Finney Rutten, Griffin, Jacobson.

Conflict of Interest Disclosures: Dr Finney Rutten reported grants from the National Cancer Institute during the conduct of the study. Dr St. Sauver reported grants from the National Institutes of Health during the conduct of the study and grant funding from the National Institutes of Aging for Interdisciplinary Infrastructure for Aging Research. Dr MacLaughlin reported grants from the National Cancer Institute during the conduct of the study. Dr Jenkins reported grants from the National Cancer Institute during the conduct of the study. Dr Herrin reported grants from the National Cancer Institute during the conduct of the study. Dr Jacobson reported grants from the National Institutes of Health during the conduct of the study; personal fees from Merck outside the submitted work. No other disclosures were reported.

Funding/Support: This research is supported by the National Cancer Institute of the National Institutes of Health (R01CA217889).

Role of the Funder/Sponsor: The National Cancer Institute of the National Institute of Health had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Disclaimer: The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Data Sharing Statement: See Supplement 3.

Additional Contributions: The authors wish to acknowledge the support and guidance we received from our dear friend, mentor, colleague, co-investigator, and collaborator Carmen Radecki Breitkopf, PhD (deceased April 25, 2019), in the development and execution of this work.