Key PointsQuestion听
Are chatbot and standard-of-care approaches equivalent in completion of pretest cancer genetic services and genetic testing?
Findings听
In this equivalence trial involving 3073 patients, equivalence between service delivery models was observed for completion of pretest cancer genetic services (estimated percentage point difference, 2.0; 95% CI, 鈭�1.1 to 5.0) and completion of genetic testing (estimated percentage point difference, 鈭�1.3; 95% CI, 鈭�3.7 to 1.1).
Meaning听
These findings have important implications for clinical practice because they support chatbot approaches for meeting the rapidly increasing demand for genetic services in light of the limited genetic counseling workforce.
Importance听
Increasing numbers of unaffected individuals could benefit from genetic evaluation for inherited cancer susceptibility. Automated conversational agents (ie, chatbots) are being developed for cancer genetics contexts; however, randomized comparisons with standard of care (SOC) are needed.
Objective听
To examine whether chatbot and SOC approaches are equivalent in completion of pretest cancer genetic services and genetic testing.
Design, Setting, and Participants听
This equivalence trial (Broadening the Reach, Impact, and Delivery of Genetic Services [BRIDGE] randomized clinical trial) was conducted between August 15, 2020, and August 31, 2023, at 2 US health care systems (University of Utah Health and NYU Langone Health). Participants were aged 25 to 60 years, had had a primary care visit in the previous 3 years, were eligible for cancer genetic evaluation, were English or Spanish speaking, had no prior cancer diagnosis other than nonmelanoma skin cancer, had no prior cancer genetic counseling or testing, and had an electronic patient portal account.
Intervention听
Participants were randomized 1:1 at the patient level to the study groups at each site. In the chatbot intervention group, patients were invited in a patient portal outreach message to complete a pretest genetics education chat. In the enhanced SOC control group, patients were invited to complete an SOC pretest appointment with a certified genetic counselor.
Main Outcomes and Measures听
Primary outcomes were completion of pretest cancer genetic services (ie, pretest genetics education chat or pretest genetic counseling appointment) and completion of genetic testing. Equivalence hypothesis testing was used to compare the study groups.
Results听
This study included 3073 patients (1554 in the chatbot group and 1519 in the enhanced SOC control group). Their mean (SD) age at outreach was 43.8 (9.9) years, and most (2233 of 3063 [72.9%]) were women. A total of 204 patients (7.3%) were Black, 317 (11.4%) were Latinx, and 2094 (75.0%) were White. The estimated percentage point difference for completion of pretest cancer genetic services between groups was 2.0 (95% CI, 鈭�1.1 to 5.0). The estimated percentage point difference for completion of genetic testing was 鈭�1.3 (95% CI, 鈭�3.7 to 1.1). Analyses suggested equivalence in the primary outcomes.
Conclusions and Relevance听
The findings of the BRIDGE equivalence trial support the use of chatbot approaches to offer cancer genetic services. Chatbot tools can be a key component of sustainable and scalable population health management strategies to enhance access to cancer genetic services.
Trial Registration听
ClinicalTrials.gov Identifier:
Identifying unaffected individuals with inherited cancer susceptibility allows for genetic evaluation and implementation of individualized prevention and screening recommendations.1-6 However, most people with inherited cancer susceptibility are unaware of their condition.7-10 Additionally, more genes have been associated with inherited cancers,11-19 and broader ranges of family histories are indications for genetic testing,20 increasing the opportunity to identify individuals at increased risk. Once identified, at-risk individuals may face barriers to accessing genetic services due to the limited number of trained genetic specialists for direct patient care.21-23 Scalable and sustainable strategies are needed to effectively identify individuals with inherited cancer susceptibility and deliver genetic services.
Prior research has compared different delivery models for genetic services,24 and technology-based tools have received attention as potentially scalable approaches to service delivery.25,26 Interest in chatbots, automated conversational agents that use artificial intelligence and natural language processing to simulate human conversation, has grown rapidly.26,27 Chatbots have advantages for delivering information, including interactivity, chunking information into small segments, and allowing for choice in information received.28-31 In genetics contexts, chatbots have been deployed for various purposes.32-37 Prior research has provided support for the acceptability and feasibility of chatbots in genetics contexts27,32,37,38 and has shown that chatbots may facilitate family communication.35,39 However, few studies have examined how chatbots affect delivery of cancer genetic services,40 and randomized designs are needed to compare chatbots vs standard of care (SOC). This study aimed to address these research gaps.41
This equivalence trial (Broadening the Reach, Impact, and Delivery of Genetic Services [BRIDGE] randomized clinical trial [RCT]) was approved as a single鈥搃nstitutional review board protocol by the University of Utah Institutional Review Board, consistent with current US National Institutes of Health policy for multisite RCTs. The trial protocol and statistical analysis plan are presented in Supplement 1. Because the trial compared 2 clinical service delivery models, the institutional review board approved a waiver of consent for the procedures described here. The study followed the Consolidated Standards of Reporting Trials () reporting guideline.
The BRIDGE trial was conducted between August 15, 2020, and August 31, 2023. First, the trial investigators identified unaffected primary care patients who qualified for genetic risk assessment in 2 health care systems (University of Utah Health and NYU Langone Health) using a standards-based platform to automatically evaluate cancer family history information in a patient鈥檚 electronic health record (EHR). The trial investigators then compared the primary outcomes of uptake of pretest cancer genetic services and genetic testing for chatbot vs enhanced SOC genetic services delivery models among eligible patients randomly selected from those identified by the algorithm (Figure). Participants were randomized using a random number generator 1:1 at the patient level. They were stratified by site from the primary care departments at each site, with at least 100 patients meeting the algorithm criteria. In both groups, a genetic counseling assistant (GCA) confirmed trial eligibility, sent outreach messages to primary care clinicians (PCCs) and patients, and placed genetic testing orders (as described in the Procedures subsection). The difference between groups was that in the enhanced SOC control group, patients were invited in the outreach message to complete an SOC pretest appointment with a certified genetic counselor. In the chatbot intervention group, patients were invited to complete a pretest genetics education chat. The null hypothesis of nonequivalence in the primary outcomes between groups was tested.
We included unaffected patients aged 25 to 60 years who had a primary care visit in the University of Utah Health or NYU Langone Health systems in the previous 3 years and were eligible for cancer genetic testing according to modified National Comprehensive Cancer Network (NCCN) guidelines based on their previously obtained family history in the EHR (Epic; Epic Systems).42-44 Patients were identified using GARDE, an open-source, standards-based platform that extracts and evaluates cancer family history information available in structured EHR data elements and adds eligible patients to a registry in the EHR鈥檚 population health management tool.42,43 Primary care visits in internal medicine or family medicine were included at both sites; obstetrics and gynecology visits were also included at University of Utah Health. Patients were not required to have a PCC on record. Additional eligibility criteria for the trial were being English or Spanish speaking, having no prior cancer diagnosis other than nonmelanoma skin cancer, having no prior genetic counseling or testing related to hereditary cancer, and having an electronic patient portal (MyChart in Epic) account or creating one. Patients with a known variant in the family were excluded from the trial and referred for clinical genetic services.
Random samples of 2241 patients at University of Utah Health and 3061 at NYU Langone Health were selected from the pool of eligible patients identified by GARDE and allocated to study groups. Additional trial eligibility criteria were reviewed by a GCA before and after patient outreach. In both groups, 1 week before patient outreach, PCCs were notified that their patients would be contacted. Procedures were available in English or Spanish. Patients were not billed for pretest genetic services in either group. Genetic testing was billed as a clinical test in both groups.
In the enhanced SOC control group, selected patients were first sent a patient portal message, recommending genetic services and encouraging them to contact the genetics clinic at their study site to schedule a pretest genetic counseling appointment. Nonresponders received a reminder message 1 week later and up to 2 additional follow-up telephone calls from a GCA. Interested eligible patients were scheduled for a pretest genetic counseling appointment with a certified genetic counselor. As described in the BRIDGE protocol previously,41 genetic counseling appointments in this group followed clinical SOC. Most were performed by telephone. Genetic testing orders were entered into the laboratory portal by a GCA, and a saliva kit, instructions, and return mailing materials were sent to patients鈥� home.
In the patient-directed chatbot intervention group, selected patients were first sent a patient portal message recommending genetic services and including a hyperlink to complete pretest genetics education via chatbot. The pretest genetics education chat used the Invitae technical platform (Invitae Corporation) and chatbot interface for delivery,45 with content specifically scripted for the BRIDGE study by an interdisciplinary team.31,41 The chatbot content was developed to represent key information delivered during SOC pretest genetic counseling appointments and included text, images, and video. The first part of the chat was a video message from the lead genetic counselor at the patient鈥檚 site introducing the chat. Patients then moved through a core set of scripted information (eFigure in Supplement 2) and could request additional information on preselected topics or ask open-ended questions in a free-text format. The team scripted responses to possible open-ended questions; questions without a scripted response were emailed to the genetic counseling team. Patients with incomplete chats received a reminder message 1 week later and up to 2 follow-up telephone calls from a GCA. Patients could request to speak with a genetic counselor instead of or in addition to completing the pretest genetics education chat. At the end of the chatbot script, patients were offered the option to continue with genetic testing. A GCA contacted patients to confirm their decision about genetic testing and collect additional family history information if needed. The GCA entered the genetic testing order into the laboratory portal, and a saliva collection kit, instructions, and return mailing supplies were sent to patients鈥� home. Patient interactions with the chatbot, including their testing decisions, were saved in the patient EHR as a note.
Genetic testing was performed at Clinical Laboratory Improvement Act鈥揷ertified commercial laboratories. Patients received pancancer, multigene panel tests for cancer susceptibility genes that included approximately 34 to 36 genes. The laboratory performing the testing billed the patients鈥� insurance. Uninsured patients were informed about free or reduced cost options.
Data were abstracted from the population health management registry and study clinical records at each site. Outcomes and patient characteristics were assessed as follows.
The primary outcomes were defined as completion of pretest cancer genetic services (ie, completed pretest genetics education chat in the intervention group or pretest genetic counseling appointment in the control group) and completion of genetic testing. These 2 primary outcomes examined whether patients received the information needed to consider genetic testing through pretest services and whether they chose to complete testing. The secondary outcomes were starting pretest cancer genetic services (opening pretest genetics education chat in intervention group or scheduling pretest genetic counseling appointment in control group) and ordering genetic testing.
We abstracted EHR data on patient age at outreach, sex, race and ethnicity, language preference, zip code, and having a recorded PCC. Race and ethnicity were included to characterize the patient populations of the 2 health care systems and are reported as Black, Latinx, White, or other race or ethnicity (American Indian, Alaska Native, Asian, Native Hawaiian, or Pacific Islander). Zip code data were used to determine urbanicity through Rural-Urban Commuting Area codes.46 We examined whether the patient met more than 1 NCCN criterion for cancer genetic testing.42-44
Descriptive statistics were calculated using the gtsummary package in R, version 4.3.0 (R Project for Statistical Computing).47 Pearson 蠂2, Wilcoxon rank sum, and Fisher exact tests assessed bivariate associations by experimental condition. We also computed descriptive statistics and bivariate associations stratified by study site. We used the Farrington-Manning test48 to test an equivalence hypothesis between study groups for the overall patient population and for each study site. The equivalence margin was set at 11 percentage points for completion of pretest cancer genetic services and 5 percentage points for completion of genetic testing based on a priori power calculations.41 All statistical analyses were performed using R, version 4.3.0,49 with P鈥�<鈥�.05 (2-sided) considered statistically significant.
Participant Characteristics
Of the 5302 patients randomly selected, 3073 were eligible and included in this study (1554 in the chatbot group and 1519 in the enhanced SOC control group; Table 1). There were 1444 patients in the University of Utah Health sample (Table 2) and 1629 in the NYU Langone Health sample (Table 3). The mean (SD) age of patients at outreach was 43.8 (9.9) years. Of the 3063 patients with data on sex available, 2233 (72.9%) were female and 830 (27.1%) were male. Race and ethnicity were available for 2792 patients: 204 (7.3%) were Black, 317 (11.4%) were Latinx, 2094 (75.0%) were White, and 177 (6.3%) were of other race or ethnicity. Most patients had a recorded PCC (2360 of 3072 [76.8%]), lived in an urban area (2960 of 3071 [96.4%]), and indicated a preference for English (3028 of 3067 [98.7%]).
Completion of Pretest Cancer Genetic Services and Genetic Testing
Analyses suggested equivalence between groups overall and for each site (Table 4). For the primary outcome of completion of pretest cancer genetic services, the estimated percentage point difference between groups was 2.0 (95% CI, 鈭�1.1 to 5.0). Outcomes were similar by site; estimated percentage point differences were 1.5 (95% CI, 鈭�3.1 to 6.1) for University of Utah Health and 2.4 (95% CI, 鈭�1.7 to 6.4) for NYU Langone Health. Few patients in the chatbot group who started the pretest genetics education chat requested to meet with a genetic counselor (4 [2.8%] at University of Utah Health and 4 [2.4%] at NYU Langone Health).
For the primary outcome of completion of cancer genetic testing, the estimated percentage point difference was 鈭�1.3 (95% CI, 鈭�3.7 to 1.1) (Table 4). Analyses suggested equivalence between groups overall and for University of Utah Health, but not for NYU Langone Health. The estimated percentage point differences for completion of genetic testing were 0.1 (95% CI, 鈭�3.6 to 3.7) for University of Utah Health and 鈭�2.5 (95% CI, 鈭�5.6 to 0.7) for NYU Langone Health (Table 4).
Analyses suggested equivalence between groups in the secondary outcome of beginning pretest services overall and for each study site (Table 4). There was no evidence to suggest equivalence between groups in the secondary outcome of ordering genetic testing overall and for each study site (Table 4): estimated percentage point differences were 鈭�3.1 (95% CI, 鈭�5.7 to 鈭�0.5) overall, 鈭�3.7 (95% CI, 鈭�7.7 to 0.4) for University of Utah Health, and 鈭�2.6 (95% CI, 鈭�6.1 to 0.8) for NYU Langone Health.
The BRIDGE trial compared uptake of cancer genetic services for chatbot vs SOC approaches among 3073 unaffected primary care patients in 2 large health care systems eligible for cancer genetic evaluation based on their family history. The trial findings suggested equivalence between these genetic services delivery models for the primary outcomes of uptake of pretest cancer genetic services and genetic testing and for the secondary outcome of beginning pretest genetic services, although statistically significantly more patients in the SOC group ordered genetic testing. The equivalence findings have important implications for clinical practice because chatbot approaches are supported to offer pretest cancer genetic services and genetic testing after outreach to unaffected patients eligible for genetic evaluation, providing a way to meet the rapidly increasing demand for these services. Few patients in the chatbot group requested a clinical genetic counseling appointment, suggesting acceptability of the chatbot approach.
A chatbot can provide multiple advantages as a technology-based approach to expand access to genetic services because it is highly scalable, private, and cost-effective and can be used at places and times of individuals鈥� choosing.26,41,50 Chatbots could be used to conduct routine tasks, allowing genetic counselors to focus on more specialized care,51,52 and to facilitate a more personalized counseling approach meeting patients鈥� individual needs.25 In the BRIDGE trial, the pretest genetics education chatbot achieved these goals by providing a core set of information to all patients while allowing some degree of personalization through selection of additional information or asking open-ended questions. A prior analysis of use of the chatbot in a pilot phase showed that most patients selected a limited amount of additional information and few asked open-ended questions, suggesting that the content generally met patients鈥� educational needs.31 This observation is supported by a small RCT of women with stage 0 to III breast cancer not meeting NCCN criteria for genetic testing, in which investigators found similar knowledge and satisfaction among those randomized to pretest counseling via chatbot compared with a certified genetic counselor.40 The BRIDGE study team has created an open-source version of the chat, enhancing the scalability of this approach in other health care systems.
The trial findings indicate that a chatbot could be part of what has been described as a mainstreaming model, with standardized pretest education delivered outside of traditional genetic counseling.32,53 Chatbots may support pretest genetics education without individualized genetic counseling for many patients. However, increasing reach and engagement is important, and others have highlighted the need to increase uptake of digital approaches.54 Although the completion of pretest genetic services observed in the BRIDGE trial is consistent with prior population screening efforts,54-57 most patients did not complete pretest genetic counseling or testing in either service delivery model. Optimizing patient engagement is critical. It will be important for future research to determine what aspects of cancer genetic counseling can be effectively delivered via chatbot, particularly as this tool becomes more personalized and empathic in its responses.58 Approaches that limit clinician and staff contacts to those essential to improving patient outcomes will be more scalable at a population level. The MAGENTA (Making Genetic Testing Accessible) trial compared 4 different approaches with or without individualized pretest and posttest genetic counseling.59 The MAGENTA investigators found the highest rates of testing completion in the 2 groups without pretest genetic counseling,59 which suggests that studies that compare outcomes of chatbot use to approaches without pretest education are also important. Patient cognitive and emotional responses, such as decision regret, to different service delivery models should also be investigated.
The findings of this study can inform the use of chatbots in cancer care more generally. A systematic review of 21 studies of chatbot applications across the cancer continuum provided some support for patient satisfaction and efficacy.60 Few prior studies have conducted randomized comparisons with SOC or examined service delivery outcomes.60,61 The present findings provide support for expanded use of chatbots. Chatbots may be a viable approach to address underidentification of individuals with inherited cancer syndromes through genetic risk assessment of unaffected individuals. Traditional strategies for identifying patients with inherited cancer susceptibility start with the person with cancer, prompting cascade testing of at-risk relatives if a pathogenic variant is identified.62 However, testing of patients with cancer is low and the majority of people who harbor pathogenic variants associated with cancer risk are not aware,7-10 underscoring the need for efficient and scalable approaches for identifying those at increased risk.10,25,63 A recent systematic review identified 6 studies with chatbots used to perform genetic cancer risk assessment and found a pooled estimated completion rate for risk assessment of 36.7% (95% CI, 14.8% to 65.9%).64 Completion rates may vary by factors such as population or affected status. Also, in several studies, chatbots were offered to patients in conjunction with medical appointments, indicating that timing and integration with other health care services may be important. Notably, none of these studies involved a comparison, highlighting the need for randomized designs. In addition, although the present trial focused on unaffected patients, automated approaches may be worth exploring to address the lack of genetic testing in patients with cancer.65 Future studies can also examine factors affecting adoption and implementation of chatbots in cancer genetics care.66,67
The trial findings should be considered in light of the following limitations. Genetic testing procedures were not fully automated for patients randomized to the chatbot group; to enhance patient safety in the trial, GCAs contacted all patients in the chatbot group to confirm testing decisions. The genetic counseling teams were not blinded to study group since the EHR indicated whether pretest services had been delivered via chatbot or an appointment. Most participants were White and female, and our prior analyses showed that use of EHR data introduced bias into the population identified as eligible for testing.68 Despite the availability of trial procedures in Spanish, few Spanish-speaking patients met the study criteria68 or completed genetic services and testing. Intentional approaches may be needed to deliver genetic services to these populations. All trial participants were affiliated with University of Utah Health or NYU Langone Health, had a primary care visit within the last 3 years, and had access to the patient portal. It will be important to expand the reach of genetic testing, with or without chatbots, among patients who are uninsured or face other access barriers. Future studies can also examine how usage of patient portal accounts affects responses to outreach for genetic services. Because we received a waiver of consent for clinical procedures in the trial, we were not able to collect baseline information about patients beyond what was available in the EHR. However, this pragmatic design allowed us to contact a random sample of eligible primary care patients rather than a highly selected group who agreed to participate in a research study.
Despite this trial鈥檚 limitations, its findings advance the state of the science in delivery of cancer genetic services to unaffected patients meeting criteria for genetic evaluation. By comparing chatbot service delivery vs SOC using a multisite RCT design, the findings of the BRIDGE equivalence trial support the use of chatbot approaches to offer cancer genetic services. As referrals for cancer genetic services increase,24,69-71 models that direct genetic counseling time and resources to those patients with the greatest needs are warranted.63 The findings show that use of chatbots to deliver pretest genetic services has strong potential to increase access to these services for unaffected patients. These trial findings therefore support the implementation of systemwide population health management strategies to deliver cancer genetic services.
Accepted for Publication: July 12, 2024.
Published: September 9, 2024. doi:10.1001/jamanetworkopen.2024.32143
Open Access: This is an open access article distributed under the terms of the CC-BY License. 漏 2024 Kaphingst KA et al. 糖心vlog Open.
Corresponding Author: Kimberly A. Kaphingst, ScD, Huntsman Cancer Institute, 2000 Circle of Hope Dr, Salt Lake City, UT 84112 (kim.kaphingst@hci.utah.edu).
Author Contributions: Drs Kaphingst and Goodman had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.
Concept and design: Kaphingst, Kohlmann, Goodman, Chavez-Yenter, Colonna, Espinel, Wetter, Mann, Ginsburg, Sigireddi, Del Fiol.
Acquisition, analysis, or interpretation of data: Kaphingst, Kohlmann, Lorenz Chambers, Bather, Goodman, Bradshaw, Chavez-Yenter, Colonna, Everett, Flynn, Gammon, Harris, Hess, Kaiser-Jackson, Lee, Monahan, Schiffman, Volkmar, Wetter, Zhong, Ginsburg, Sigireddi, Kawamoto, Del Fiol, Buys.
Drafting of the manuscript: Kaphingst, Bather, Chavez-Yenter, Mann, Sigireddi.
Critical review of the manuscript for important intellectual content: Kohlmann, Lorenz Chambers, Bather, Goodman, Bradshaw, Chavez-Yenter, Colonna, Espinel, Everett, Flynn, Gammon, Harris, Hess, Kaiser-Jackson, Lee, Monahan, Schiffman, Volkmar, Wetter, Zhong, Ginsburg, Sigireddi, Kawamoto, Del Fiol, Buys.
Statistical analysis: Bather, Goodman, Chavez-Yenter, Harris.
Obtained funding: Kaphingst, Goodman, Wetter, Del Fiol, Buys.
Administrative, technical, or material support: Lorenz Chambers, Bradshaw, Chavez-Yenter, Colonna, Espinel, Everett, Flynn, Gammon, Kaiser-Jackson, Lee, Monahan, Volkmar, Mann, Kawamoto, Del Fiol.
Supervision: Kaphingst, Kohlmann, Goodman, Espinel, Mann, Sigireddi.
Conflict of Interest Disclosures: Dr Kaphingst reported receiving grants from the National Cancer Institute (NCI) and the National Institutes of Health (NIH) during the conduct of the study. Dr Kohlmann reported receiving grants from the NCI during the conduct of the study. Dr Goodman reported receiving grants from the NIH during the conduct of the study. Dr Hess reported serving as a data and safety monitoring board member for Astellas Pharmaceuticals outside the submitted work. Ms Monahan reported receiving grants from the NIH during the conduct of the study. Dr Schiffman reported receiving personal fees from Peel Therapeutics Inc outside the submitted work. Dr Kawamoto reported receiving grants from the NCI during the conduct of the study. In addition, Dr Kawamoto reported receiving grants from Hitachi and personal fees from Pfizer, RTI International, University of California at San Francisco, Indiana University, NORC at the University of Chicago, University of Pennsylvania, Yale University, Elsevier, MD Aware, Custom Clinical Decision Support, Regenstrief Foundation, Korean Society of Medical Informatics, University of Nebraska, and the US Office of the National Coordinator for Health IT (via Security Risk Solutions) outside the submitted work. Dr Del Fiol reported receiving grants from the NCI during the conduct of the study. No other disclosures were reported.
Funding/Support: Research reported in this publication was supported by award U01CA232826 from the NCI (Drs Kaphingst, Sigireddi, and Buys).
Role of the Funder/Sponsor: The funder had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication. As part of the grant review process, the study protocol underwent peer review by the funding body.
Disclaimer: The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
Data Sharing Statement: See Supplement 3.
Additional Contributions: We thank the members of the BRIDGE Research Team and the genetic counseling teams at both sites who worked on this study. We also thank the patients who were part of the study.
1.Hall
听MJ锘�, Forman
听AD锘�, Pilarski
听R锘�, Wiesner
听G锘�, Giri
听VN锘�. 听Gene panel testing for inherited cancer risk.听锘� 听J Natl Compr Canc Netw. 2014;12(9):1339-1346. doi:锘�
2.Hiraki
听S锘�, Rinella
听ES锘�, Schnabel
听F锘�, Oratz
听R锘�, Ostrer
听H锘�. 听颁补苍肠别谤 risk assessment using genetic panel testing: considerations for clinical application.听锘� 听J Genet Couns. 2014;23(4):604-617. doi:锘�
3.Kurian
听AW锘�, Hare
听EE锘�, Mills
听MA锘�,
听et al. 听Clinical evaluation of a multiple-gene sequencing panel for hereditary cancer risk assessment.听锘� 听J Clin Oncol. 2014;32(19):2001-2009. doi:锘�
4.Maxwell
听KN锘�, Wubbenhorst
听B锘�, D鈥橝ndrea
听K锘�,
听et al. 听Prevalence of mutations in a panel of breast cancer susceptibility genes in BRCA1/2-negative patients with early-onset breast cancer.听锘� 听Genet Med. 2015;17(8):630-638. doi:锘�
5.Desmond
听A锘�, Kurian
听AW锘�, Gabree
听M锘�,
听et al. 听Clinical actionability of multigene panel testing for hereditary breast and ovarian cancer risk assessment.听锘� 听闯础惭础 Oncol. 2015;1(7):943-951. doi:锘�
6.Slavin
听TP锘�, Niell-Swiller
听M锘�, Solomon
听I锘�,
听et al. 听Clinical application of multigene panels: challenges of next-generation counseling and cancer risk management.听锘� 听Front Oncol. 2015;5:208.
7.Hull
听LE锘�, Haas
听JS锘�, Simon
听SR锘�. 听Provider discussion of genetic testing with US women at risk for BRCA mutation.听锘� 听Am J Prev Med. 2018;54(2):221-228. doi:锘�
8.Frey
听MK锘�, Finch
听A锘�, Kulkarni
听A锘�, Akbari
听MR锘�, Chapman-Davis
听E锘�. 听Genetic testing for all: overcoming disparities in ovarian cancer genetic testing.听锘� 听Am Soc Clin Oncol Educ Book. 2022;42:1-12. doi:锘�
9.Cancer Moonshot Blue Ribbon Panel Report 2016. National Cancer Institute; October 17, 2016. Accessed March 1, 2024.
10.Childers
听CP锘�, Childers
听KK锘�, Maggard-Gibbons
听M锘�, Macinko
听J锘�. 听National estimates of genetic testing in women with a history of breast or ovarian cancer.听锘� 听J Clin Oncol. 2017;35(34):3800-3806. doi:锘�
11.Pennington
听KP锘�, Swisher
听EM锘�. 听Hereditary ovarian cancer: beyond the usual suspects.听锘� 听Gynecol Oncol. 2012;124(2):347-353. doi:锘�
12.Genetics of breast and gynecologic cancers (PDQ)鈥攈ealth professional version. National Cancer Institute. Accessed March 1, 2024.
13.Domchek
听SM锘�, Bradbury
听A锘�, Garber
听JE锘�, Offit
听K锘�, Robson
听ME锘�. 听Multiplex genetic testing for cancer susceptibility: out on the high wire without a net?听锘� 听J Clin Oncol. 2013;31(10):1267-1270. doi:锘�
14.Pritchard
听CC锘�, Mateo
听J锘�, Walsh
听MF锘�,
听et al. 听Inherited DNA-repair gene mutations in men with metastatic prostate cancer.听锘� 听N Engl J Med. 2016;375(5):443-453. doi:锘�
15.Shindo
听K锘�, Yu
听J锘�, Suenaga
听M锘�,
听et al. 听Deleterious germline mutations in patients with apparently sporadic pancreatic adenocarcinoma.听锘� 听J Clin Oncol. 2017;35(30):3382-3390. doi:锘�
16.Mandelker
听D锘�, Zhang
听L锘�, Kemel
听Y锘�,
听et al. 听Mutation detection in patients with advanced cancer by universal sequencing of cancer-related genes in tumor and normal DNA vs guideline-based germline testing.听锘� 听闯础惭础. 2017;318(9):825-835. doi:锘�
17.Yurgelun
听MB锘�, Kulke
听MH锘�, Fuchs
听CS锘�,
听et al. 听颁补苍肠别谤 susceptibility gene mutations in individuals with colorectal cancer.听锘� 听J Clin Oncol. 2017;35(10):1086-1095. doi:锘�
18.Pearlman
听R锘�, Frankel
听WL锘�, Swanson
听B锘�,
听et al; Ohio Colorectal Cancer Prevention Initiative Study Group. 听Prevalence and spectrum of germline cancer susceptibility gene mutations among patients with early-onset colorectal cancer.听锘� 听闯础惭础 Oncol. 2017;3(4):464-471. doi:锘�
19.Stoffel
听EM锘�, Koeppe
听E锘�, Everett
听J锘�,
听et al. 听Germline genetic features of young individuals with colorectal cancer.听锘� 听骋补蝉迟谤辞别苍迟别谤辞濒辞驳测. 2018;154(4):897-905. doi:锘�
20.Giri
听VN锘�, Knudsen
听KE锘�, Kelly
听WK锘�,
听et al. 听Role of genetic testing for inherited prostate cancer risk: Philadelphia Prostate Cancer Consensus Conference 2017.听锘� 听J Clin Oncol. 2018;36(4):414-424. doi:锘�
21.Hoskovec
听JM锘�, Bennett
听RL锘�, Carey
听ME锘�,
听et al. 听Projecting the supply and demand for certified genetic counselors: a workforce study.听锘� 听J Genet Couns. 2018;27(1):16-20. doi:锘�
22.Attard
听CA锘�, Carmany
听EP锘�, Trepanier
听AM锘�. 听Genetic counselor workflow study: the times are they a-changin鈥�?听锘� 听J Genet Couns. 2019;28(1):130-140. doi:锘�
23.Maiese
听DR锘�, Keehn
听A锘�, Lyon
听M锘�, Flannery
听D锘�, Watson
听M锘�; Working Groups of the National Coordinating Center for Seven Regional Genetics Service Collaboratives. 听Current conditions in medical genetics practice.听锘� 听Genet Med. 2019;21(8):1874-1877. doi:锘�
24.Buchanan
听AH锘�, Rahm
听AK锘�, Williams
听JL锘�. 听Alternate service delivery models in cancer genetic counseling: a mini-review.听锘� 听Front Oncol. 2016;6:120. doi:锘�
25.Amendola
听LM锘�, Golden-Grant
听K锘�, Scollon
听S锘�. 听Scaling genetic counseling in the genomics era.听锘� 听Annu Rev Genomics Hum Genet. 2021;22:339-355. doi:锘�
26.Cazzaniga
听A锘�, Plebani
听M锘�, Crimi
听M锘�. 听Genome access and other web-based IT solutions: genetic counseling in the digital era.听锘� 听Front Public Health. 2022;10:1035316. doi:锘�
27.Luca
听S锘�, Clausen
听M锘�, Shaw
听A锘�,
听et al; Genetics Navigator Study Team. 听Finding the sweet spot: a qualitative study exploring patients鈥� acceptability of chatbots in genetic service delivery.听锘� 听Hum Genet. 2023;142(3):321-330. doi:锘�
28.Doak
听CC锘�, Doak
听LG锘�, Root
听JH锘�. 听Teaching Patients with Low Literacy Skills. 2nd ed. Lippincott Williams & Wilkins; 1996. doi:
29.Mayer
听RE锘�, Dow
听GT锘�, Mayer
听S锘�. 听Multimedia learning in an interactive self-explaining environment: what works in the design of agent-based microworlds.听锘� 听J Educ Psychol. 2003;95(4):806-813. doi:锘�
30.Health Literacy Universal Precautions Toolkit. 3rd ed. Agency for Healthcare Research & Quality; April 2010. Reviewed June 2024. Accessed November 20, 2023.
31.Chavez-Yenter
听D锘�, Kimball
听KE锘�, Kohlmann
听W锘�,
听et al. 听Patient interactions with an automated conversational agent delivering pretest genetics education: descriptive study.听锘� 听J Med Internet Res. 2021;23(11):e29447. doi:锘�
32.Heald
听B锘�, Keel
听E锘�, Marquard
听J锘�,
听et al. 听Using chatbots to screen for heritable cancer syndromes in patients undergoing routine colonoscopy.听锘� 听J Med Genet. 2021;58(12):807-814. doi:锘�
33.Ponathil
听A锘�, Ozkan
听F锘�, Welch
听B锘�, Bertrand
听J锘�, Chalil Madathil
听K锘�. 听Family health history collected by virtual conversational agents: an empirical study to investigate the efficacy of this approach.听锘� 听J Genet Couns. 2020;29(6):1081-1092. doi:锘�
34.Ireland
听D锘�, Bradford
听D锘�, Szepe
听E锘�,
听et al. 听Introducing Edna: a trainee chatbot designed to support communication about additional (secondary) genomic findings.听锘� 听Patient Educ Couns. 2021;104(4):739-749. doi:锘�
35.Schmidlen
听T锘�, Jones
听CL锘�, Campbell-Salome
听G锘�, McCormick
听CZ锘�, Vanenkevort
听E锘�, Sturm
听AC锘�. 听Use of a chatbot to increase uptake of cascade genetic testing.听锘� 听J Genet Couns. 2022;31(5):1219-1230. doi:锘�
36.Schmidlen
听T锘�, Schwartz
听M锘�, DiLoreto
听K锘�, Kirchner
听HL锘�, Sturm
听AC锘�. 听Patient assessment of chatbots for the scalable delivery of genetic counseling.听锘� 听J Genet Couns. 2019;28(6):1166-1177. doi:锘�
37.Siglen
听E锘�, Vetti
听HH锘�, Augestad
听M锘�, Steen
听VM锘�, Lunde
听脜锘�, Bjorvatn
听C锘�. 听Evaluation of the Rosa chatbot providing genetic information to patients at risk of hereditary breast and ovarian cancer: qualitative interview study.听锘� 听J Med Internet Res. 2023;25:e46571. doi:锘�
38.Soley
听N锘�, Klein
听A锘�, Taylor
听CO锘�,
听et al. 听Feasibility of the genetic information assistant chatbot to provide genetic education and study genetic test adoption among pancreatic cancer patients at Johns Hopkins Hospital.听锘� 听AMIA Jt Summits Transl Sci Proc. 2023;2023:497-504.
39.Walters
听NL锘�, Lindsey-Mills
听ZT锘�, Brangan
听A锘�,
听et al. 听Facilitating family communication of familial hypercholesterolemia genetic risk: assessing engagement with innovative chatbot technology from the IMPACT-FH study.听锘� 听PEC Innov. 2023;2:100134. doi:锘�
40.Al-Hilli
听Z锘�, Noss
听R锘�, Dickard
听J锘�,
听et al. 听A randomized trial comparing the effectiveness of pre-test genetic counseling using an artificial intelligence automated chatbot and traditional in-person genetic counseling in women newly diagnosed with breast cancer.听锘� 听Ann Surg Oncol. 2023;30(10):5990-5996. doi:锘�
41.Kaphingst
听KA锘�, Kohlmann
听W锘�, Chambers
听RL锘�,
听et al; BRIDGE research team. 听Comparing models of delivery for cancer genetics services among patients receiving primary care who meet criteria for genetic evaluation in two healthcare systems: BRIDGE randomized controlled trial.听锘� 听BMC Health Serv Res. 2021;21(1):542. doi:锘�
42.Del Fiol
听G锘�, Kohlmann
听W锘�, Bradshaw
听RL锘�,
听et al. 听Standards-based clinical decision support platform to manage patients who meet guideline-based criteria for genetic evaluation of familial cancer.听锘� 听JCO Clin Cancer Inform. 2020;4:1-9. doi:锘�
43.Mowery
听DL锘�, Kawamoto
听K锘�, Bradshaw
听R锘�,
听et al. 听Determining onset for familial breast and colorectal cancer from family history comments in the electronic health record.听锘� 听AMIA Jt Summits Transl Sci Proc. 2019;2019:173-181.
44.Bradshaw
听RL锘�, Kawamoto
听K锘�, Kaphingst
听KA锘�,
听et al. 听GARDE: a standards-based clinical decision support platform for identifying population health management cohorts.听锘� 听J Am Med Inform Assoc. 2022;29(5):928-936. doi:锘�
45.Nazareth
听S锘�, Nussbaum
听RL锘�, Siglen
听E锘�, Wicklund
听CA锘�. 听Chatbots & artificial intelligence to scale genetic information delivery.听锘� 听J Genet Couns. 2021;30(1):7-10. doi:锘�
46.Rural-urban commuting area codes. Economic Research Service; US Department of Agriculture. Updated September 25, 2023. Accessed June 12, 2023.
47.Daniel
听DS锘�, Whiting
听K锘�, Curry
听M锘�, Lavery
听JA锘�, Larmarange
听J锘�. 听Reproducible summary tables with the gtsummary package.听锘� 听R J. 2021;13(1):570-580. doi:锘�
48.Farrington
听CP锘�, Manning
听G锘�. 听Test statistics and sample size formulae for comparative binomial trials with null hypothesis of non-zero risk difference or non-unity relative risk.听锘� 听Stat Med. 1990;9(12):1447-1454. doi:锘�
49.The R Project for Statistical Computing. Accessed March 1, 2024.
50.Underkofhler
听K锘�, Wanchek
听T锘�, Thomas
听M锘�,
听et al. 听Modeling and comparison of costs and high-risk variant detection between a genetic counseling chatbot and a traditional genetic counseling pathway.听锘� 听Gynecol Oncol. 2023;176(suppl 1):S64. doi:锘�
51.Sato
听A锘�, Haneda
听E锘�, Suganuma
听N锘�, Narimatsu
听H锘�. 听Preliminary screening for hereditary breast and ovarian cancer using a chatbot augmented intelligence genetic counselor: development and feasibility study.听锘� 听JMIR Form Res. 2021;5(2):e25184. doi:锘�
52.Uhlmann
听WR锘�, Hoskovec
听J锘�, Freivogel
听M锘�. 听40 years and beyond for the National Society of Genetic Counselors: reflections on genetic counseling practice.听锘� 听J Genet Couns. 2020;29(6):888-893. doi:锘�
53.Hamilton
听JG锘�, Symecko
听H锘�, Spielman
听K锘�,
听et al. 听Uptake and acceptability of a mainstreaming model of hereditary cancer multigene panel testing among patients with ovarian, pancreatic, and prostate cancer.听锘� 听Genet Med. 2021;23(11):2105-2113. doi:锘�
54.Wang
听C锘�, Lu
听H锘�, Bowen
听DJ锘�, Xuan
听Z锘�. 听Implementing digital systems to facilitate genetic testing for hereditary cancer syndromes: an observational study of 4 clinical workflows.听锘� 听Genet Med. 2023;25(5):100802. doi:锘�
55.Arun
听BK锘�, Peterson
听SK锘�, Sweeney
听LE锘�,
听et al. 听Increasing referral of at-risk women for genetic counseling and BRCA testing using a screening tool in a community breast imaging center.听锘� 听颁补苍肠别谤. 2022;128(1):94-102. doi:锘�
56.Wernke
听K锘�, Bellcross
听C锘�, Gabram
听S锘�, Ali
听N锘�, Stanislaw
听C锘�. 听Impact of implementing B-RST to screen for hereditary breast and ovarian cancer on risk perception and genetic counseling uptake among women in an academic safety net hospital.听锘� 听Clin Breast Cancer. 2019;19(4):e547-e555. doi:锘�
57.Hoskins
听KF锘�, Tejeda
听S锘�, Vijayasiri
听G锘�,
听et al. 听A feasibility study of breast cancer genetic risk assessment in a federally qualified health center.听锘� 听颁补苍肠别谤. 2018;124(18):3733-3741. doi:锘�
58.Reghunath
听A锘�. 听Expression of Empathy in Social Virtual Bots Used for Genetic Counseling. Uppsala University; 2021.
59.Swisher
听EM锘�, Rayes
听N锘�, Bowen
听D锘�,
听et al. 听Remotely delivered cancer genetic testing in the Making Genetic Testing Accessible (MAGENTA) trial: a randomized clinical trial.听锘� 听闯础惭础 Oncol. 2023;9(11):1547-1555. doi:锘�
60.Wang
听A锘�, Qian
听Z锘�, Briggs
听L锘�, Cole
听AP锘�, Reis
听LO锘�, Trinh
听QD锘�. 听The use of chatbots in oncological care: a narrative review.听锘� 听Int J Gen Med. 2023;16:1591-1602. doi:锘�
61.Aggarwal
听A锘�, Tam
听CC锘�, Wu
听D锘�, Li
听X锘�, Qiao
听S锘�. 听Artificial intelligence-based chatbots for promoting health behavioral changes: systematic review.听锘� 听J Med Internet Res. 2023;25:e40789. doi:锘�
62.Offit
听K锘�, Tkachuk
听KA锘�, Stadler
听ZK锘�,
听et al. 听Cascading after peridiagnostic cancer genetic testing: an alternative to population-based screening.听锘� 听J Clin Oncol. 2020;38(13):1398-1408. doi:锘�
63.Rana
听HQ锘�, Garber
听JE锘�. 听Right-sizing models of genetic cancer predisposition testing.听锘� 听闯础惭础 Oncol. 2023;9(11):1555-1556. doi:锘�
64.Webster
听EM锘�, Ahsan
听MD锘�, Perez
听L锘�,
听et al. 听Chatbot artificial intelligence for genetic cancer risk assessment and counseling: a systematic review and meta-analysis.听锘� 听JCO Clin Cancer Inform. 2023;7:e2300123. doi:锘�
65.Kurian
听AW锘�, Abrahamse
听P锘�, Furgal
听A锘�,
听et al. 听Germline genetic testing after cancer diagnosis.听锘� 听闯础惭础. 2023;330(1):43-51. doi:锘�
66.White
听BK锘�, Martin
听A锘�, White
听JA锘�. 听User experience of COVID-19 chatbots: scoping review.听锘� 听J Med Internet Res. 2022;24(12):e35903. doi:锘�
67.Wutz
听M锘�, Hermes
听M锘�, Winter
听V锘�, K枚berlein-Neu
听J锘�. 听Factors influencing the acceptability, acceptance, and adoption of conversational agents in health care: integrative review.听锘� 听J Med Internet Res. 2023;25:e46548. doi:锘�
68.Chavez-Yenter
听D锘�, Goodman
听MS锘�, Chen
听Y锘�,
听et al. 听Association of disparities in family history and family cancer history in the electronic health record with sex, race, Hispanic or Latino ethnicity, and language preference in 2 large US health care systems.听锘� 听闯础惭础 Netw Open. 2022;5(10):e2234574. doi:锘�
69.McCuaig
听JM锘�, Armel
听SR锘�, Care
听M锘�, Volenik
听A锘�, Kim
听RH锘�, Metcalfe
听KA锘�. 听Next-generation service delivery: a scoping review of patient outcomes associated with alternative models of genetic counseling and genetic testing for hereditary cancer.听锘� 听颁补苍肠别谤s (Basel). 2018;10(11):435. doi:锘�
70.Cohen
听SA锘�, Huziak
听RC锘�, Gustafson
听S锘�, Grubs
听RE锘�. 听Analysis of advantages, limitations, and barriers of genetic counseling service delivery models.听锘� 听J Genet Couns. 2016;25(5):1010-1018. doi:锘�
71.Trepanier
听AM锘�, Allain
听DC锘�. 听Models of service delivery for cancer genetic risk assessment and counseling.听锘� 听J Genet Couns. 2014;23(2):239-253. doi:锘�