Prediction Models for Suicide Attempts and Deaths: A Systematic Review and Simulation

Bradley E. Belsher; Derek J. Smolenski; Larry D. Pruitt; Nigel E. Bush; Erin H. Beech; Don E. Workman; Rebecca L. Morgan; Daniel P. Evatt; Jennifer Tucker; Nancy A. Skopp

doi:10.1001/jamapsychiatry.2019.0174

Armed Forces Health Surveillance Center. 听Numbers and proportions of US military members in treatment for mental disorders over time, active component, January 2000-September 2013.听听惭厂惭搁. 2014;21(5):2-7.

Corr 听WP 听III. 听Suicides and suicide attempts among active component members of the US Armed Forces, 2010-2012; methods of self-harm vary by major geographic region of assignment.听听惭厂惭搁. 2014;21(10):2-5.

Anglemyer 听A, Miller 听ML, Buttrey 听S, Whitaker 听L. 听Suicide rates and methods in active duty military personnel, 2005 to 2011: a cohort study.听听Ann Intern Med. 2016;165(3):167-174. doi:

Pruitt 听LD, Smolenski 听DJ, Bush 听NE, Skopp 听NA, Hoyt 听TV, Grady 听BJ. 听Department of Defense Suicide Event Report: Calendar Year 2015 Annual Report. Washington, DC: Department of Defense; 2016.

Schoenbaum 听M, Kessler 听RC, Gilman 听SE, 听et al; Army STARRS Collaborators. 听Predictors of suicide and accident death in the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS): results from the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS).听听JAMA Psychiatry. 2014;71(5):493-503. doi:

Pruitt 听LD, Smolenski 听DJ, Bush 听NE, Skopp 听NA, Edwards-Stewart 听A, Hoyt 听TV. 听Department of Defense Suicide Event Report: Calendar Year 2016 Annual Report. Washington, DC: Department of Defense; 2017.

听VA National Suicide Data Report: 2005-2015. Washington, DC: US Department of Veterans Affairs; 2018.

听Supporting our veterans during their transition from uniformed service to civilian life.听听Federal Register. 2018;83(9):1513-1514.

Franklin 听JC, Ribeiro 听JD, Fox 听KR, 听et al. 听Risk factors for suicidal thoughts and behaviors: a meta-analysis of 50 years of research.听听Psychol Bull. 2017;143(2):187-232. doi:

10.

Runeson 听B, Odeberg 听J, Pettersson 听A, Edbom 听T, Jildevik Adamsson 听I, Waern 听M.听Instruments for the assessment of suicide risk: a systematic review evaluating the certainty of the evidence.听听PLoS One. 2017;12(7):e0180292. doi:

11.

Carter 听G, Milner 听A, McGill 听K, Pirkis 听J, Kapur 听N, Spittal 听MJ. 听Predicting suicidal behaviours using clinical instruments: systematic review and meta-analysis of positive predictive values for risk scales.听听Br J Psychiatry. 2017;210(6):387-395. doi:

12.

LeFevre 听ML; US Preventive Services Task Force. 听Screening for suicide risk in adolescents, adults, and older adults in primary care: US Preventive Services Task Force recommendation statement.听听Ann Intern Med. 2014;160(10):719-726. doi:

13.

The Joint Commission. 听Detecting and treating suicide ideation in all settings.听听Sentinel Event Alert. 2016;56(56):1-7.

14.

Hogan 听MF, Grumet 听JG. 听Suicide prevention: an emerging priority for health care.听听Health Aff (Millwood). 2016;35(6):1084-1090. doi:

15.

Walkup 听JT, Townsend 听L, Crystal 听S, Olfson 听M. 听A systematic review of validated methods for identifying suicide or suicidal ideation using administrative or claims data.听听Pharmacoepidemiol Drug Saf. 2012;21(suppl 1):174-182. doi:

16.

Wang 听Y, Kung 听LA, Byrd 听TA. 听Big data analytics: understanding its capabilities and potential benefits for healthcare organizations.听听Technol Forecast Soc Change. 2018;126:3-13. doi:

17.

Pokorny 听AD. 听Prediction of suicide in psychiatric patients: report of a prospective study.听听Arch Gen Psychiatry. 1983;40(3):249-257. doi:

18.

MacKinnon 听DR, Farberow 听NL. 听An assessment of the utility of suicide prediction.听听Suicide Life Threat Behav. 1976;6(2):86-91.

19.

Moher 听D, Liberati 听A, Tetzlaff 听J, Altman 听DG; PRISMA Group. 听Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement.听听PLoS Med. 2009;6(7):e1000097. doi:

20.

Predictive modeling algorithms to estimate suicide risk: a systematic review and meta-analysis protocol. PROSPERO. . Accessed February 7, 2019.

21.

Ban 听JW, Emparanza 听JI, Urreta 听I, Burls 听A. 听Design characteristics influence performance of clinical prediction rules in validation: a meta-epidemiological study.听听PLoS One. 2016;11(1):e0145779. doi:

22.

Whiting 听PF, Rutjes 听AW, Westwood 听ME, 听et al; QUADAS-2 Group. 听QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies.听听Ann Intern Med. 2011;155(8):529-536. doi:

23.

Streiner 听DL. 听Diagnosing tests: using and misusing diagnostic and screening tests.听听J Pers Assess. 2003;81(3):209-219. doi:

24.

Choi 听SB, Lee 听W, Yoon 听JH, Won 听JU, Kim 听DW. 听Ten-year prediction of suicide death using Cox regression and machine learning in a nationwide retrospective cohort study in South Korea.听听J Affect Disord. 2018;231:8-14. doi:

25.

Kessler 听RC, Hwang 听I, Hoffmire 听CA, 听et al. 听Developing a practical suicide risk prediction model for targeting high-risk patients in the Veterans Health Administration.听听Int J Methods Psychiatr Res. 2017;26(3):e1575. doi:

26.

McCarthy 听JF, Bossarte 听RM, Katz 听IR, 听et al听Predictive modeling and concentration of the risk of suicide: implications for preventive interventions in the US Department of Veterans Affairs.听听Am J Public Health. 2015;105(9):1935-1942. doi:

27.

Simon 听GE, Johnson 听E, Lawrence 听JM, 听et al. 听Predicting suicide attempts and suicide deaths following outpatient visits using electronic health records.听听Am J Psychiatry. 2018;175(10):951-960. doi:

28.

Kessler 听RC, Warner 听CH, Ivany 听C, 听et al; Army STARRS Collaborators. 听Predicting suicides after psychiatric hospitalization in US Army soldiers: the Army Study To Assess Risk and rEsilience in Servicemembers (Army STARRS).听听JAMA Psychiatry. 2015;72(1):49-57. doi:

29.

Kessler 听RC, Stein 听MB, Petukhova 听MV, 听et al; Army STARRS Collaborators. 听Predicting suicides after outpatient mental health visits in the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS).听听Mol Psychiatry. 2017;22(4):544-551. doi:

30.

Amini 听P, Ahmadinia 听H, Poorolajal 听J, Moqaddasi Amiri 听M. 听Evaluating the high risk groups for suicide: a comparison of logistic regression, support vector machine, decision tree and artificial neural network.听听Iran J Public Health. 2016;45(9):1179-1187.

31.

Barak-Corren 听Y, Castro 听VM, Javitt 听S, 听et al. 听Predicting suicidal behavior from longitudinal electronic health records.听听Am J Psychiatry. 2017;174(2):154-162. doi:

32.

DelPozo-Banos 听M, John 听A, Petkov 听N, 听et al. 听Using neural networks with routine health records to identify suicide risk: feasibility study.听听JMIR Ment Health. 2018;5(2):e10144. doi:

33.

Karmakar 听C, Luo 听W, Tran 听T, Berk 听M, Venkatesh 听S. 听Predicting risk of suicide attempt using history of physical illnesses from electronic medical records.听听JMIR Ment Health. 2016;3(3):e19. doi:

34.

Nock 听MK, Millner 听AJ, Joiner 听TE, 听et al. 听Risk factors for the transition from suicide ideation to suicide attempt: Results from the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS).听听J Abnorm Psychol. 2018;127(2):139-149. doi:

35.

Tran 听T, Luo 听W, Phung 听D, 听et al. 听Risk stratification using data from electronic medical records better predicts suicide risks than clinician assessments.听听BMC Psychiatry. 2014;14:76-84. doi:

36.

Walsh 听CG, Ribeiro 听JD, Franklin 听JC. 听Predicting risk of suicide attempts over time through machine learning.听听Clin Psychol Sci. 2017;5(3):457-469. doi:

37.

Galfalvy 听HC, Oquendo 听MA, Mann 听JJ. 听Evaluation of clinical prognostic models for suicide attempts after a major depressive episode.听听Acta Psychiatr Scand. 2008;117(4):244-252. doi:

38.

Hartl 听TL, Rosen 听C, Drescher 听K, Lee 听TT, Gusman 听F. 听Predicting high-risk behaviors in veterans with posttraumatic stress disorder.听听J Nerv Ment Dis. 2005;193(7):464-472. doi:

39.

Ilgen 听MA, Downing 听K, Zivin 听K, 听et al. 听Exploratory data mining analysis identifying subgroups of patients with depression who are at high risk for suicide.听听J Clin Psychiatry. 2009;70(11):1495-1500. doi:

40.

Motto 听JA, Bostrom 听AG. 听Models of suicide risk: nice persons.听听颁谤颈蝉颈蝉. 1990;11(2):37-47.

41.

Centers for Disease Control and Prevention. Welcome to WISQARS. . Accessed February 6, 2019.

42.

Grimes 听DA, Schulz 听KF. 听Uses and abuses of screening tests.听听尝补苍肠别迟. 2002;359(9309):881-884. doi:

43.

Mulder 听R, Newton-Howes 听G, Coid 听JW. 听The futility of risk prediction in psychiatry.听听Br J Psychiatry. 2016;209(4):271-272. doi:

44.

Nelson 听HD, Denneson 听LM, Low 听AR, 听et al. 听Suicide risk assessment and prevention: a systematic review focusing on veterans.听听Psychiatr Serv. 2017;68(10):1003-1015. doi:

45.

Mustafa 听RA, Wiercioch 听W, Cheung 听A, 听et al. 听Decision making about healthcare-related tests and diagnostic test strategies: paper 2: a review of methodological and practical challenges.听听J Clin Epidemiol. 2017;92:18-28. doi:

46.

Donald Draughon v United State of America, et al. Case No. 14-2264-JAR-GLR. United States District Court, D. Kansas.

47.

听Joint Action Plan for Supporting Veterans During Their Transition From Uniformed Service to Civilian Life. Washington, DC: US Department of Veterans Affairs; 2018.

48.

VA REACH VET initiative helps save veterans lives: program signals when more help is needed for at-risk veterans [press release]. Washington DC: US Department of Veterans Affairs; April 3, 2017.

49.

Tucker 听RP, Tackett 听MJ, Glickman 听D, Reger 听MA. 听Ethical and practical considerations in the use of a predictive model to trigger suicide prevention interventions in healthcare settings听 [published online January 18, 2018]. 听Suicide Life Threat Behav. doi:

Select Your Interests

Save Preferences

Others Also Liked

Review

March 13, 2019

Prediction Models for Suicide Attempts and Deaths: A Systematic Review and Simulation

Bradley E.听Belsher,听PhD^1,2; Derek J.听Smolenski,听PhD, MPH¹; Larry D.听Pruitt,听PhD¹; et al Nigel E.听Bush,听PhD¹; Erin H.听Beech,听MA¹; Don E.听Workman,听PhD^1,2; Rebecca L.听Morgan,听PhD, MPH³; Daniel P.听Evatt,听PhD^1,2; 闯别苍苍颈蹿别谤听罢耻肠办别谤,听笔丑顿¹; Nancy A.听Skopp,听PhD¹

Author Affiliations Article Information

¹Psychological Health Center of Excellence, Defense Health Agency, Silver Spring, Maryland
²Uniformed Services University of the Health Sciences, Bethesda, Maryland
³Department of Clinical Epidemiology and Biostatistics, McMaster University, Hamilton, Ontario, Canada

JAMA Psychiatry. 2019;76(6):642-651. doi:10.1001/jamapsychiatry.2019.0174

Key Points

Question听 Have advances in statistical modeling improved the predictive validity of suicide prediction algorithms sufficiently to render their predictions actionable?

Findings听 In this systematic review of 17 studies including 64 unique suicide prediction models, the models had good overall classification and low positive predictive values. Use of these models would result in high false-positive rates and considerable false-negative rates if implemented in isolation.

Meaning听 At present, the performance of suicide prediction models suggests that they offer limited practical utility in predicting suicide mortality.

Abstract

Importance听 Suicide prediction models have the potential to improve the identification of patients at heightened suicide risk by using predictive algorithms on large-scale data sources. Suicide prediction models are being developed for use across enterprise-level health care systems including the US Department of Defense, US Department of Veterans Affairs, and Kaiser Permanente.

Objectives听 To evaluate the diagnostic accuracy of suicide prediction models in predicting suicide and suicide attempts and to simulate the effects of implementing suicide prediction models using population-level estimates of suicide rates.

Evidence Review听 A systematic literature search was conducted in MEDLINE, PsycINFO, Embase, and the Cochrane Library to identify research evaluating the predictive accuracy of suicide prediction models in identifying patients at high risk for a suicide attempt or death by suicide. Each database was searched from inception to August 21, 2018. The search strategy included search terms for suicidal behavior, risk prediction, and predictive modeling. Reference lists of included studies were also screened. Two reviewers independently screened and evaluated eligible studies.

Findings听 From a total of 7306 abstracts reviewed, 17 cohort studies met the inclusion criteria, representing 64 unique prediction models across 5 countries with more than 14 million participants. The research quality of the included studies was generally high. Global classification accuracy was good (鈮�0.80 in most models), while the predictive validity associated with a positive result for suicide mortality was extremely low (鈮�0.01 in most models). Simulations of the results suggest very low positive predictive values across a variety of population assessment characteristics.

Conclusions and Relevance听 To date, suicide prediction models produce accurate overall classification models, but their accuracy of predicting a future event is near 0. Several critical concerns remain unaddressed, precluding their readiness for clinical applications across health systems.

糖心vlog

Object moved to here.

糖心vlog

Prediction Models for Suicide Attempts and Deaths: A Systematic Review and Simulation

Featured Articles

USPSTF Recommendation Statements

Blogs

糖心vlog

Object moved to here.

糖心vlog

See More About

Select Your Interests

Select Your Interests

Others Also Liked

Citation

Manage citations:

Prediction Models for Suicide Attempts and Deaths: A Systematic Review and Simulation