Ruuska et al.’s [1] analysis incorrectly concluded that medical gender reassignment (GR) did not reduce suicide mortality because they had assessed that “the suicide mortality of both those [presenting with gender dysphoria (GD)] who proceeded and did not proceed to GR did not statistically significantly differ from that of controls.” On the contrary, by conventional criteria, the suicide mortality of the non-GR group was significantly higher than controls while that of the GR group was not. Ruuska et al. [1] reported: "Adjusted HRs [hazard rates] for suicide mortality were 3.2 [for non-GR] (95% CI 1.0 to 10.2; p=0.05) and 0.8 [for GR] (95% CI 0.2 to 4.0; p=0.8), respectively." By this finding, those dysphorics who had undergone GR were no more likely to have committed suicide than were general population controls, while those who had not undergone GR were more than three times as likely to have committed suicide. The latter difference is reported with a p-value of .05 and a 95% confidence interval that does not extend below 1.0.

By the prevailing standards of scientific inference, and in virtually any other study, such a finding would be assessed as statistically significant (or perhaps, depending on rounding error, trivially below significance), but Ruuska et al. [1] judged it not to be so. They achieved this anomaly by fiat, announcing: “In order to avoid type 1 error due to multiple testing and the large data size, the cut- off for statistical significance was set at a p<0.01” [1] rather than the usual standard of p<0.05. This is a decision is highly questionable, for two reasons. First, the size of the data analyzed is not nearly large enough to justify it: 18,726 cases (122,358 person-years). Preference for a .01 criterion usually involves data numbering in the hundreds of thousands, at least, and even then rarely precludes recognition of substantively important findings that may have a higher p-value below .05. Of the ten most recent studies using larger similar data—extractions from Finnish registers that are as large or larger than that of Ruuska et al. [1] —with numbers of cases ranging from 24,341 to 5.6 million (the entire Finnish population), or 41.9 million person-years, none rejected findings between .05 and .01 as not being statistically significant. (See Endnote 1) The most methodologically similar study of the same subject as that of Ruuska et al. [1] , Dhejne et al.’s [11] 2011 examination of the Swedish population register, also used the conventional .05 standard. None of these large-data register studies rejected as insignificant a finding with p less than .05 but greater than .01, or a 95% confidence interval that did not include nullity, except for Ruuska et al. [1] Co-authors of the Ruuska et al. [1] study also imposed a restrictive .01 standard for significance on an earlier register study of gender reassignment, [12] but did not do so for similarly-sized register studies of other topics, for example schizophrenia (17,112 cases), [13] and depression (16,842 cases). [14]

More importantly, second, adjusting the significance criterion to reduce the prospect of one type of error inescapably imbalances or biases, by the same amount, the analysis toward the opposite error. In this case, it perversely increases the risk of the very type of error the stricter criterion is trying to avoid. This is because adopting a stricter significance criterion to reduce the risk of Type I errors due to perceived statistical significance raises, by the same amount, the risk of Type I errors due to perceived lack of statistical significance; and the inference that GR did not reduce suicide is based on perceived statistical insignificance. If the goal were not to affirm too easily substantive findings that may be due to measurement imprecision, then for conclusions that depend on similarity between groups rather than difference, the criterion for assessing true difference should be made less strict (perhaps .10), not more strict. A better practice, followed in many studies, might be to avoid unilaterally adjusting the conventional scientific standard of inference, but present findings as more or less certain with reference to that standard depending on their reported actual p-value.

Recognition that suicide risk may be lower with GR than without it in this analysis complicates, but does not negate, Ruuska et al’s [1] substantive observation that this comparison “does not support the claims that GR is necessary in order to prevent suicide.” It complicates it because a similar previous analysis of the same register data involving two of the same authors [12] already found that, when adjusted for year of initial contact with Finland’s gender identity service, follow-up contact with psychiatric care was emphatically no different by medical gender reassignment (GR) status. In that study, the hazard ratio (HR) for those gender dysphorics who had undergone GR—3.8 (95% CI 3.6-4.1)—was virtually identical to that of those who had not done so—3.9 (95% CI 3.6-4.2). Both of these HRs differed significantly from controls but did not differ significantly from each other. Unlike Ruuska et al., [1] they provided strong support for the earlier study’s conclusion that these findings “do not suggest that medical GR interventions resolve psychiatric morbidity among people experiencing gender distress.”[12]

This does not mean that Ruuska et al.’s observation is incorrect. There are at least two plausible explanations for the apparent contradiction of lower suicide mortality following GR despite similar high psychiatric morbidity, compared to GD registrants who did not receive GR. The first explanation reflects the time trajectory of suicide mortality. Dhejne et al. (2011), in the only other national registry study of completed suicide, observed that “survival of transsexual persons [undergoing medical gender reassignment] started to diverge from that of matched controls after about 10 years of follow-up.” [11] Up until 10 years post GR, the suicidal mortality of the Swedish registrants in the Dhejne et al. (2011) study did not differ from that of the general population controls, which is exactly what is observed, at an average follow-up of only 6.5 years, among the Finnish registrants in the Ruuska et al. [1] study. After 10 years, however, Dhejne et al. (2011) found that suicide mortality among Swedish GR recipients increased dramatically, a possibility that is not at all foreclosed for their Finnish counterparts by the findings of Ruuska et al. [1] .

A second explanation is suggested by the fact that, unlike in many jurisdictions, in Finland only 38% of GD diagnosed persons in treatment proceeded to medical GR. This suggests the presence of assessment, screening and monitoring processes that may inhibit some of the negative consequences of psychiatric morbidity, for example by ensuring better social support or medication compliance, than may be the case in other settings.
These two explanations are not mutually exclusive; both may be the case to some extent. Neither of them, moreover, impair in any way the main conclusion of Ruuska et al’s [1] study, that apart from attendant psychiatric morbidity, gender dysphoria in itself does not predict suicide mortality. Both also strongly support the important clinical implication of the study, that it is “of utmost importance to identify and appropriately treat mental disorders in adolescents experiencing gender dysphoria to prevent suicide.”

Endnote 1: The studies are: Bockerman et al.’s study of opioid use and employment rates (41.9 million person-years) [2]; Silenpaa et al.’s estimation of childhood epilepsy incidence (entire population rates); [3] Vaajala et al.’s (2024) study of fear of childbirth (comparing 211,202 cases from the Finnish Birth Register); [4] Ax et al.’s (2024) study of hand surgery and sick leave (24,341 cases); [5] Valtanen et al.’s (2024) comparison of psychiatric medication use by treatment strategy (44,685 cases); [6] Raudasoja et al. (2024), examining corrective surgery after treatment of distal radial fractures (41,418 cases); [7] Terho et al’s. (2024) study of frozen embryo transfer child growth rates (35,894); [8] Tamlander et al’s (2024) analysis of implications of polygenic risk for colorectal cancer screening (453,733 cases); [9] Kolari et al. (2024), examining ADHD medication use among Finnish children (41,920 cases); [10] and Holm et al.’s (2024) study of nonaffective psychosis diagnosis, examining 49,164 persons.

References
1 Ruuska S-M, Tuisku K, Holttinen T, et al. All-cause and suicide mortalities among adolescents and young adults who contacted specialised gender identity services in Finland in 1996–2019: a register study. BMJ Ment Health. 2024;27:e300940.
2 Böckerman P, Haapanen M, Hakulinen C, et al. Prescription opioid use and employment: A nationwide Finnish register study. Drug and Alcohol Dependence. 2021;227:108967.
3 Sillanpää ML, Camfield P, Löyttyniemi E. The changing incidence of childhood epilepsy in Finland. Seizure (London, England). 2024;117:20–7.
4 Vaajala M, Mattila VM, Kuitunen I. Fear of childbirth prolongs interpregnancy interval: A nationwide register-based quantile logistic regression analysis. European Journal of Obstetrics & Gynecology and Reproductive Biology: X. 2024;21:100281.
5 Ax M, Palola V, Ponkilainen V, et al. Duration of sick leave after operated and non-operated distal radial fracture: a Finnish cohort study of 19,995 patients. The Journal of hand surgery, European volume. 2024;49:316–21.
6 Valtanen K, Seikkula J, Kurtti M, et al. Ten-year patterns of psychiatric medications dispensed to adolescent in Finland: Open dialogue-informed practice in Western Lapland as compared to practice in other Finnish regions. Personalized Medicine in Psychiatry. 2024;43–44:100117.
7 Raudasoja L, Vastamäki H, Aspinen S, et al. Distal radial fractures: a nationwide register study on corrective osteotomies after malunion. J Hand Surg Eur Vol. 2024;49:329–33.
8 Terho AM, Tiitinen A, Salo J, et al. Growth of singletons born after frozen embryo transfer until early adulthood: a Finnish register study. Human reproduction (Oxford). 2024;39:604–11.
9 Tamlander M, Jermy B, Seppälä TT, et al. Genome-wide polygenic risk scores for colorectal cancer have implications for risk-based screening. British journal of cancer. 2024;130:651–9.
10 Kolari TA, Vuori M, RÄttÖ H, et al. Incidence of ADHD medication use among Finnish children and adolescents in 2008–2019: a need for practice changes? Scand J Public Health. 2024;14034948231219826.
11 Dhejne C, Lichtenstein P, Boman M, et al. Long-Term Follow-Up of Transsexual Persons Undergoing Sex Reassignment Surgery: Cohort Study in Sweden. PLoS One. 2011;6. doi: 10.1371/journal.pone.0016885
12 Kaltiala R, Holttinen T, Tuisku K. Have the psychiatric needs of people seeking gender reassignment changed as their numbers increase? A register study in Finland. Eur Psychiatry. 2023;66:e93.
13 Holttinen T, Pirkola S, Kaltiala R. Schizophrenia among young people first admitted to psychiatric inpatient care during early and middle adolescence. Schizophrenia research. 2023;252:103–9.
14 Jutta Niemi, Timo Holttinen, Riittakerttu Kaltiala. Adolescents with severe depressive and anxiety – a nationwide register study of inpatient-treated adolescents 1980- 2010. Psychiatria fennica (Online). 2023;54:66–79.

The comments by Quinlivan and colleagues provide an opportunity to respond to some common misunderstandings of suicide risk assessment tools, and more broadly, prediction modelling. First, their comments are based on the mistaken assumption that all suicide prediction tools invariably have to classify individuals into low-risk versus high-risk groups. Unlike the earlier tools referred to in the response (all of which are classifiers, i.e. stratify people into risk categories), OxSATS provides probabilistic estimates of suicide risk. The benefits of probability estimation over classification have been discussed widely in the methodological literature,[1,2] and models which produce continuous risk scores are routinely used in other areas of medicine (such as the Framingham and QRISK models for cardiovascular disease risk prediction).

Second, Quinlivan and colleagues have compared the area under the curve (AUC) of OxSATS to earlier tools and highlighted the discrepancy in the interpretation of the findings. However, this misses the methodological point that what is considered good discrimination performance for a prediction model depends on the clinical area and available alternatives. While very high AUC values (e.g. above 0.90) can be reported for diagnostic prediction,[3] such values are rare in prognostic modelling, where AUC values in the 0.70s are found for best-performing models for incident cardiovascular disease[4] and adverse health outcomes (including mortality) in COVID-19.[5]

Third, their comments fail to note that two models can have similar AUC values despite very different calibration performance. While OxSATS was reasonably well-calibrated in the external validation sample, ‘first-generation scales’ cannot even be assessed on their calibration performance because they have not been tested (and cannot without probability scores). Calibration is especially important when a model is intended to support clinical decision-making, as poorly calibrated predictions can make a model clinically useless or even harmful.[6]

Finally, it is misleading, in our view, to interpret the positive predictive value (PPV) of a suicide prediction model without considering the specific clinical context. This is because, like other measures of classification, PPV is dependent on the chosen risk cut-off. Usually, there is not one optimal risk threshold for a given prediction model, as the choice of cut-off depends on the specific clinical decision that the model is intended to inform and the relative costs and benefits of true and false positive classifications in that context.[1] For instance, health economics evidence from US primary care has shown that, depending on the target intervention, prediction models for suicide deaths may be cost-effective with PPV values as low as 0.07%, in part due to substantial cost savings associated with preventing one suicide death.[7]

References:

1. Wynants L, Smeden M van, McLernon DJ, Timmerman D, Steyerberg EW, Calster BV, et al. Three myths about risk thresholds for prediction models. BMC Med. 2019;17(1):192.

2. Goorbergh R van den, Smeden M van, Timmerman D, Calster BV. The harm of class imbalance corrections for risk prediction models: illustration and simulation using logistic regression. J Am Med Inform Assn. 2022.

3. Calster BV, Hoorde KV, Valentin L, Testa AC, Fischerova D, Holsbeke CV, et al. Evaluating the risk of ovarian cancer before surgery using the ADNEX model to differentiate between benign, borderline, early and advanced stage invasive, and secondary metastatic tumours: prospective multicentre diagnostic study. BMJ 2014;349:g5920.

4. Damen JA, Pajouheshnia R, Heus P, Moons KGM, Reitsma JB, Scholten RJPM, et al. Performance of the Framingham risk models and pooled cohort equations for predicting 10-year risk of cardiovascular disease: a systematic review and meta-analysis. BMC Med. 2019;17(1):109.

5. Wynants L, Calster BV, Collins GS, Riley RD, Heinze G, Schuit E, et al. Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal. BMJ. 2020;369:m1328.

6. Calster BV, McLernon DJ, Smeden M van, Wynants L, Steyerberg EW, Bossuyt P, et al. Calibration: the Achilles heel of predictive analytics. BMC Med. 2019;17(1):230.

7. Ross EL, Zuromski KL, Reis BY, Nock MK, Kessler RC, Smoller JW. Accuracy Requirements for Cost-effective Suicide Risk Prediction Among Primary Care Patients in the US. JAMA Psychiatry. 2021;78(6):642–50.