Jump to comment:

• Suicide prediction following self-harm: implications of the new science of risk modelling and being open to the evidence
  Aida Seyedsalehi, Seena Fazel
  Published on: 20 March 2024
• Suicide prediction following self-harm: are new tools any better?
  Leah Quinlivan, Sarah Steeg and Nav Kapur
  Published on: 23 October 2023

• Published on: 20 March 2024

  Suicide prediction following self-harm: implications of the new science of risk modelling and being open to the evidence

  • Aida Seyedsalehi, DPhil Student University of Oxford
  • Other Contributors:
    • Seena Fazel, Professor of Psychiatry, Forensic Psychiatrist

  The comments by Quinlivan and colleagues provide an opportunity to respond to some common misunderstandings of suicide risk assessment tools, and more broadly, prediction modelling. First, their comments are based on the mistaken assumption that all suicide prediction tools invariably have to classify individuals into low-risk versus high-risk groups. Unlike the earlier tools referred to in the response (all of which are classifiers, i.e. stratify people into risk categories), OxSATS provides probabilistic estimates of suicide risk. The benefits of probability estimation over classification have been discussed widely in the methodological literature,[1,2] and models which produce continuous risk scores are routinely used in other areas of medicine (such as the Framingham and QRISK models for cardiovascular disease risk prediction).

  Second, Quinlivan and colleagues have compared the area under the curve (AUC) of OxSATS to earlier tools and highlighted the discrepancy in the interpretation of the findings. However, this misses the methodological point that what is considered good discrimination performance for a prediction model depends on the clinical area and available alternatives. While very high AUC values (e.g. above 0.90) can be reported for diagnostic prediction,[3] such values are rare in prognostic modelling, where AUC values in the 0.70s are found for best-performing models for incident cardiovascular disease[4] and adverse health outcomes (including mortal...

  Show More

  The comments by Quinlivan and colleagues provide an opportunity to respond to some common misunderstandings of suicide risk assessment tools, and more broadly, prediction modelling. First, their comments are based on the mistaken assumption that all suicide prediction tools invariably have to classify individuals into low-risk versus high-risk groups. Unlike the earlier tools referred to in the response (all of which are classifiers, i.e. stratify people into risk categories), OxSATS provides probabilistic estimates of suicide risk. The benefits of probability estimation over classification have been discussed widely in the methodological literature,[1,2] and models which produce continuous risk scores are routinely used in other areas of medicine (such as the Framingham and QRISK models for cardiovascular disease risk prediction).

  Second, Quinlivan and colleagues have compared the area under the curve (AUC) of OxSATS to earlier tools and highlighted the discrepancy in the interpretation of the findings. However, this misses the methodological point that what is considered good discrimination performance for a prediction model depends on the clinical area and available alternatives. While very high AUC values (e.g. above 0.90) can be reported for diagnostic prediction,[3] such values are rare in prognostic modelling, where AUC values in the 0.70s are found for best-performing models for incident cardiovascular disease[4] and adverse health outcomes (including mortality) in COVID-19.[5]

  Third, their comments fail to note that two models can have similar AUC values despite very different calibration performance. While OxSATS was reasonably well-calibrated in the external validation sample, ‘first-generation scales’ cannot even be assessed on their calibration performance because they have not been tested (and cannot without probability scores). Calibration is especially important when a model is intended to support clinical decision-making, as poorly calibrated predictions can make a model clinically useless or even harmful.[6]

  Finally, it is misleading, in our view, to interpret the positive predictive value (PPV) of a suicide prediction model without considering the specific clinical context. This is because, like other measures of classification, PPV is dependent on the chosen risk cut-off. Usually, there is not one optimal risk threshold for a given prediction model, as the choice of cut-off depends on the specific clinical decision that the model is intended to inform and the relative costs and benefits of true and false positive classifications in that context.[1] For instance, health economics evidence from US primary care has shown that, depending on the target intervention, prediction models for suicide deaths may be cost-effective with PPV values as low as 0.07%, in part due to substantial cost savings associated with preventing one suicide death.[7]

  References:

  1. Wynants L, Smeden M van, McLernon DJ, Timmerman D, Steyerberg EW, Calster BV, et al. Three myths about risk thresholds for prediction models. BMC Med. 2019;17(1):192.

  2. Goorbergh R van den, Smeden M van, Timmerman D, Calster BV. The harm of class imbalance corrections for risk prediction models: illustration and simulation using logistic regression. J Am Med Inform Assn. 2022.

  3. Calster BV, Hoorde KV, Valentin L, Testa AC, Fischerova D, Holsbeke CV, et al. Evaluating the risk of ovarian cancer before surgery using the ADNEX model to differentiate between benign, borderline, early and advanced stage invasive, and secondary metastatic tumours: prospective multicentre diagnostic study. BMJ 2014;349:g5920.

  4. Damen JA, Pajouheshnia R, Heus P, Moons KGM, Reitsma JB, Scholten RJPM, et al. Performance of the Framingham risk models and pooled cohort equations for predicting 10-year risk of cardiovascular disease: a systematic review and meta-analysis. BMC Med. 2019;17(1):109.

  5. Wynants L, Calster BV, Collins GS, Riley RD, Heinze G, Schuit E, et al. Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal. BMJ. 2020;369:m1328.

  6. Calster BV, McLernon DJ, Smeden M van, Wynants L, Steyerberg EW, Bossuyt P, et al. Calibration: the Achilles heel of predictive analytics. BMC Med. 2019;17(1):230.

  7. Ross EL, Zuromski KL, Reis BY, Nock MK, Kessler RC, Smoller JW. Accuracy Requirements for Cost-effective Suicide Risk Prediction Among Primary Care Patients in the US. JAMA Psychiatry. 2021;78(6):642–50.

  Show Less

  Conflict of Interest:
  SF is part of the team that developed OxSATS and other OxRisk calculators.

  • Back to top
• Published on: 23 October 2023

  Suicide prediction following self-harm: are new tools any better?

  • Leah Quinlivan, Research Fellow University of Manchester
  • Other Contributors:
    • Sarah Steeg, Research Fellow
    • Nav Kapur, Professor of Psychiatry and Population Health

  Recent systematic reviews, clinical guidelines, and suicide prevention strategies suggest we should abandon the endeavour of using risk assessment to predict suicide and instead focus on clinical need. (1, 2) Does the recent carefully conducted paper by Fazel and colleagues mean that this advice should be overturned? We think not.

  At the turn of the century several of us were involved in developing risk tools for self-harm. Nearly two decades later, colleagues have used larger samples, novel methodology, and painstaking analysis to produce OxSATs. Of course, interpretation of how good tools are depends on multiple diagnostic accuracy statistics and clinical context. However, one accepted measure of overall performance is the ‘area under the curve’ (AUC).

  What is striking is that the AUC for predicting suicide in the 6 months after self-harm was nearly identical for OxSATs and the first generation scales (0.75 vs 0.71). (3, 4) What is perhaps even more striking is the very different interpretation of the findings. The authors of the recent study suggest OxSATs accurately predicts the risk of suicide whereas Steeg et al. concluded the opposite and suggested scales should not be used to determine treatment.

  What should we make of this discrepancy? In the end it perhaps comes down to what different researchers mean by ‘accurate’. A commonly used measure which may reflect real world utility is the positive predictive value (PPV) - of those rated as at...

  Show More

  Recent systematic reviews, clinical guidelines, and suicide prevention strategies suggest we should abandon the endeavour of using risk assessment to predict suicide and instead focus on clinical need. (1, 2) Does the recent carefully conducted paper by Fazel and colleagues mean that this advice should be overturned? We think not.

  At the turn of the century several of us were involved in developing risk tools for self-harm. Nearly two decades later, colleagues have used larger samples, novel methodology, and painstaking analysis to produce OxSATs. Of course, interpretation of how good tools are depends on multiple diagnostic accuracy statistics and clinical context. However, one accepted measure of overall performance is the ‘area under the curve’ (AUC).

  What is striking is that the AUC for predicting suicide in the 6 months after self-harm was nearly identical for OxSATs and the first generation scales (0.75 vs 0.71). (3, 4) What is perhaps even more striking is the very different interpretation of the findings. The authors of the recent study suggest OxSATs accurately predicts the risk of suicide whereas Steeg et al. concluded the opposite and suggested scales should not be used to determine treatment.

  What should we make of this discrepancy? In the end it perhaps comes down to what different researchers mean by ‘accurate’. A commonly used measure which may reflect real world utility is the positive predictive value (PPV) - of those rated as at high risk, how many go on to have the outcome of interest. In supplemental table 8 of the OxSATS paper the PPV for suicide within six months is 2%. In other words of 100 people rated as at high risk only 2 are truly at high risk. We acknowledge that the PPV is not the only the only way to judge a tool’s performance but we would argue that it is one that matters to clinicians.

  Where should we go from here? Listening to those with lived experience is essential. Service users report that risk scales may detract from therapeutic engagement and lead to exclusion and iatrogenic harms. (5) For many, stigma and poor experiences of care are the more pressing concerns.(6) In the end, there are probably no short-cuts to careful assessment and no easy way to identify those who go on to have adverse outcomes. However, we can agree with Fazel and colleagues on the bigger issue of making high quality, evidence-based interventions available and accessible.

  1. National Institute for Health and Care Excellence. Self-harm: assessment, management and preventing recurrence . NICE Guideline., 2022.
  2. Department of Health and Social Care. Suicide prevention in England: 5-year cross-sector strategy. https://www.gov.uk/government/publications/suicide-prevention-strategy-f..., 2023.
  3. Fazel S, Vazquez-Montes MDLA, Molero Y, et al. Risk of death by suicide following self-harm presentations to healthcare: development and validation of a multivariable clinical prediction rule (OxSATS). BMJ Mental Health 2023;26(1) doi: 10.1136/bmjment-2023-300673
  4. Steeg S, Quinlivan L, Nowland R, et al. Accuracy of risk scales for predicting repeat self-harm and suicide: a multicentre, population-level cohort study using routine clinical data. BMC Psychiatry 2018;18
  5. Graney J, Hunt IM, Quinlivan L, et al. Suicide risk assessment in UK mental health services: a national mixed-methods study. Lancet Psychiatry 2020;7(12):1046-53. doi: 10.1016/s2215-0366(20)30381-3
  6. Quinlivan LM, Gorman L, Littlewood DL, et al. 'Relieved to be seen'-patient and carer experiences of psychosocial assessment in the emergency department following self-harm: qualitative analysis of 102 free-text survey responses. BMJ Open 2021;11(5):e044434. doi: 10.1136/bmjopen-2020-044434 [published Online First: 2021/05/25]

  Show Less

  Conflict of Interest:
  NK leads the suicide work programme of the National Confidential Inquiry into Suicide and Safety. NK and LQ have undertaken quality improvement work for NHS England (NHSE). NK was expert topic advisor to the National Institute for Health and Care Excellence (NICE) self-harm guidelines and is a member of the National Suicide Prevention Strategy advisory group for England. The views of this letter are the personal views of the authors and do not necessarily reflect the views of NHSE, NICE or the Department for Health and Social Care.

  • Back to top