Article Text

Download PDFPDF

Original research
Comparison of prediction methods for treatment continuation of antipsychotics in children and adolescents with schizophrenia
  1. Soo Min Jeon1,
  2. Jaehyeong Cho2,
  3. Dong Yun Lee3,
  4. Jin-Won Kwon1
  1. 1 BK21 FOUR Community–Based Intelligent Novel Drug Discovery Education Unit, College of Pharmacy and Research Institute of Pharmaceutical Sciences, Kyungpook National University, Daegu, Republic of Korea
  2. 2 Department of Biomedical Systems Informatics, Yonsei University College of Medicine, Seodaemun-gu, Seoul, Republic of Korea
  3. 3 Department of Biomedical informatics, Ajou University School of Medicine, Ajou University, Suwon, Gyeonggi-do, Republic of Korea
  1. Correspondence to Professor Jin-Won Kwon, BK21 FOUR Community–Based Intelligent Novel Drug Discovery Education Unit, College of Pharmacy and Research Institute of Pharmaceutical Sciences, Kyungpook National University, Daegu 41566, Korea (the Republic of); jwkwon{at}


Objective There is little evidence for finding optimal antipsychotic treatment for schizophrenia, especially in paediatrics. To evaluate the performance and clinical benefit of several prediction methods for 1-year treatment continuation of antipsychotics.

Design and Settings Population-based prognostic study conducting using the nationwide claims database in Korea.

Participants 5109 patients aged 2–18 years who initiated antipsychotic treatment with risperidone/aripiprazole for schizophrenia between 2010 and 2017 were identified.

Main outcome measures We used the conventional logistic regression (LR) and common six machine-learning methods (least absolute shrinkage and selection operator, ridge, elstic net, randomforest, gradient boosting machine, and superlearner) to derive predictive models for treatment continuation of antipsychotics. The performance of models was assessed using the Brier score (BS), area under the receiver operating characteristic curve (AUROC) and area under the precision-recall curve (AUPRC). The clinical benefit of applying these models was also evaluated by comparing the treatment continuation rate between patients who received the recommended medication by models and patients who did not.

Results The gradient boosting machine showed the best performance in predicting treatment continuation for risperidone (BS, 0.121; AUROC, 0.686; AUPRC, 0.269). Among aripiprazole models, GBM for BS (0.114), SuperLearner for AUROC (0.688) and random forest for AUPRC (0.317) showed the best performance. Although LR showed lower performance than machine learnings, the difference was negligible. Patients who received recommended medication by these models showed a 1.2–1.5 times higher treatment continuation rate than those who did not.

Conclusions All prediction models showed similar performance in predicting the treatment continuation of antipsychotics. Application of prediction models might be helpful for evidence-based decision-making in antipsychotic treatment.

  • Child & adolescent psychiatry
  • Schizophrenia & psychotic disorders

Data availability statement

Data are available upon reasonable request. This study used Health Insurance Review and Assessment Service database (M20190117533). Requests to access these datasets should be directed to HIRA; Official website of HIRA:; Contact information of data access committee: +82-33-739-1083.

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See:

Statistics from

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Summary box

What is already known about this subject?

  • Antipsychotics are a mainstay of early-onset schizophrenia treatment. However, there is no clear evidence on how to determine the most appropriate antipsychotic medication for each patient, especially for paediatric patients.

What are the new findings?

  • Most machine learning algorithms did not outperform the conventional logistic regression in predicting the treatment continuation of antipsychotics in children and adolescents with schizophrenia.

  • Regardless of the modelling algorithms, the recommendation of antipsychotics for each patient by prediction models had a clinical benefit in terms of treatment continuation.

How might it impact clinical practice in the foreseeable future?

  • The use of prediction models might help identify appropriate antipsychotic treatments in children and adolescents with schizophrenia.


Schizophrenia is a challenging mental health disorder that is characterised by a wide range of psychological symptoms such as hallucinations, delusions and extremely disordered thinking and behaviour that impair daily functioning.1 Early-onset schizophrenia (EOS) is generally defined as schizophrenia onset before the age of 18 years. In childhood, the incidence of schizophrenia is extremely rare, but the incidence increases steeply in adolescence.2 Compared with adulthood-onset schizophrenia, EOS has more severe negative symptoms, premorbid adjustment, impaired cognitive function and worse treatment efficacy.3 4 As schizophrenia is a lifelong mental health disorders, the symptom severity of EOS could deteriorate over time and persist into adulthood,3 4 which causes significant and long-lasting health, social and financial burdens. Therefore, the clinical management of EOS deserves separate attention.

Antipsychotics are a keystone of pharmacological treatment for schizophrenia. To date, most clinical guidelines recommended atypical antipsychotics as the first-line pharmacological treatment for patients with schizophrenia at any age.5 However, due to the different pharmacological mechanisms of each antipsychotic and the clinical heterogeneity of schizophrenia (disease severity, symptoms and comorbidities),4 6 it is difficult to make a decision about which medication could maximise the treatment outcome for each individual patient. Consequently, it is necessary to attempt multiple antipsychotics as a trial-and-error approach for most patients before identifying an effective antipsychotic,7 which causes delayed treatment of early phase and exposes patients to unwarranted risk of side effects. Given the challenges in antipsychotic treatment for schizophrenia, a prediction model for identifying the treatment response before initiating treatment would be a beneficial approach to increase treatment effectiveness and decrease the risk of side effects.

Although some previous studies have developed prediction models to determine the treatment response of antipsychotics,8–12 there were some concerns regarding those studies. First, most only applied the conventional regression method without considering other approaches, such as machine learning methods, which have the potential to improve prediction.8 9 Second, few studies included children or adolescents. The difference between adult-onset schizophrenia and EOS has been reported in several previous studies, and the treatment response of children and adolescents in the gradual process of development in the dopamine and neurotransmitter system may differ from that of adults with a stable system.13 Third, most studies used the complicated predictors that clinicians cannot easily access, such as genetics type10 and imaging data.11 Furthermore, to our best knowledge, only one study evaluated the clinical benefit (improvement of treatment continuation for 1 year) of this kind of prediction model in selecting antipsychotic treatment for schizophrenia.12


This study was conducted to compare several prediction modelling algorithms for predicting the treatment continuation of antipsychotics using the nationwide population-based claims data of children and adolescents with schizophrenia. The focus of this study was on risperidone and aripiprazole, the two most frequently prescribed medications in this population.14 In addition, we preliminary evaluated whether these prediction models could improve the treatment continuation of antipsychotics when applied to personalised treatment recommendation.


Data source

We used the database provided by the Health Insurance Review and Assessment (HIRA) Service, a government-affiliated agency that reviews the accuracy of national health insurance claims, which are mandatory for all Korean populations. The HIRA database includes variables that identify individual characteristics, such as age, sex and economic vulnerability (ie, approximately 2.8% of the total population who qualify for medical insurance advantage beneficiaries). Clinical data include information billed by healthcare providers such as diagnoses, prescriptions, procedures and devices. The diagnosis was identified by the international classification of disease, 10th revision (ICD-10) codes. There were no missing values in this database.

This study was approved by the Institutional Review Board of Kyungpook National University (IRB number: KNU 2018–0141), and the requirement for informed consent was waived. The Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis guideline was used to develop and validate the prediction model in this study.15

Study participants

First, data of all patients who were prescribed antipsychotics and diagnosed with schizophrenia (ICD-10: F20–F29) between 2008 and 2017 were collected. Each patient was required to have a look-back period of at least 2 years to identify the new users of antipsychotics for the first episode of schizophrenia. Patients who had any history of schizophrenia diagnosis or antipsychotic prescription in the first 2 years (2008/2009) were excluded because their look-back period was insufficient. The first date of antipsychotic prescription was defined as the index date for each patient. Moreover, patients who were not included in the age range of 2–18 years at the index date were excluded. To reduce the heterogeneity of disease severity within populations,16 patients who were under psychiatric hospitalisation at the index date or prescribed multiple/long-acting injection-type antipsychotics at the index date were excluded. Finally, patients who used risperidone or aripiprazole as the first-line antipsychotic treatment were selected. Most patients (approximately 75%) in our data set used one of these two medications, and these two medications are the only Korea Food & Drug Administration-approved atypical antipsychotics for treating schizophrenia in children and adolescents17 (online supplemental figure S1).

Supplemental material

Figure 1

Treatment continuation for overall populations in the test data set. (A) backward stepwise LR, (B) LASSO, (C) ridge, (D) elastic net, (E) random forest, (F) GBM and (G) SuperLearner. GBM, gradient boosting machine; LASSO, least absolute shrinkage and selection operator; LR, logistic regression.


Treatment continuation is an important proxy measure of antipsychotic effectiveness, reflecting drug efficacy, safety and tolerability for both patients and clinicians.18 Hence, the treatment continuation was measured as the outcome of interest in this study. The treatment continuation was defined as the continuous treatment of their first-line antipsychotic medications for at least 1 year without stopping over 60 days of missed prescription or any event of all-cause treatment discontinuation. All-cause treatment discontinuation included the occurrence of any of the following events during the follow-up period: suicide (X60–X84, Y87), psychiatry hospitalisation, emergency department visit with any mental disorder and switch to or addition of other antipsychotics. Patients who met all these criteria of treatment continuation were assigned as 1; all others were assigned as 0.


Approximately 60 predictors, including patient’s demographic factors (sex, age and type of insurance) and clinical characteristics (inpatient or outpatient visit, diagnosis and prescription records), were measured for 1 year before the index date. For risperidone and aripiprazole, respectively, each variable was constructed as a binary or continuous predictor (online supplemental tables S1 and S2). Continuous predictors were normalised using min–max scaling, which is a simple method to rescale the values of the predictor range of 0 to 1. Data normalisation is the transformation of predictors to a fixed range, which is essential for preparing data for prediction modelling.19 The predictors observed less than 10 times for either medication were not included in the model development.20 There were no missing values for any predictors because we used claims data.

Development and validation of prediction models

We developed two separate prediction models for the 1-year treatment continuation of patients with risperidone and of those with aripiprazole. The respective clinical benefit of the recommended antipsychotic treatment agent (ie, risperidone or aripiprazole) was investigated based on these individual prediction models. The following seven different prediction algorithms were investigated in this process: (1) multivariable logistic regression (LR), (2) least absolute shrinkage and selection operator (LASSO), (3) ridge, (4) elastic net, (5) random forest, (6) gradient boosting machine (GBM) and (7) SuperLearner. Although LR could be categorised into machine learning algorithms, it was considered as a conventional prediction model in this study because it does not require a complex process of tuning hyperparameters compared with other algorithms.

The data set for each medication was randomly divided into training (75%) and test (25%) data sets. The training data set was used to tune the final prediction model, and the remaining test data set was only used to evaluate the performance of the finalised model. The multivariable LR was fitted using the backward stepwise algorithm based on the Akaike information criterion. In machine learning methods, the grid search method was used to identify the regularisation parameter of LASSO/ridge/elastic net and some hyperparameters of random forest/GBM. In this process, we selected the final hyperparameters that maximised the area under the receiver operating characteristic curve (AUROC) in the fivefold CV of the training data set. SuperLearner, an ensemble method, was generated by combining six tuned models (backward stepwise LR, LASSO, ridge, elastic net, random forest, and GBM) into a single prediction model; the optimal weight for combining these models was determined using the fivefold CV process. The detailed methods used to develop each model are described in online supplemental method S1.

Statistical analysis

Demographics of study population

The data of patients using risperidone and aripiprazole (demographics, medical history and medications) were presented as number (%) for categorical predictors and mean (SD) for continuous predictors. The differences between the training set and test data set were compared using independent two-sample t tests (for continuous variables) and χ2 tests (for categorical variables).

Performance evaluation of candidate algorithms

The performance of the finalised prediction models was evaluated in the test data set of each medication. To evaluate the performance of predicting algorithms using these two approaches, the following eight metrics were used: Brier score (BS), AUROC area under the precision-recall curve (AUPRC), sensitivity, specificity, accuracy, f1-score and the Matthews correlation coefficient (MCC) (online supplemental figure S2). Among these metrics, BS, AUROC and AUPRC were selected as the main metrics to evaluate the algorithms’ performance. Moreover, the importance of predictors was calculated as the relative value to the highest value (1.00) of each model based on their beta coefficient (backward stepwise LR, LASSO, ridge and elastic net) or the decrease in accuracy/Gini score (random forest and GBM).

The finalised prediction models (for each medication) were used to estimate their predicted treatment continuation for the specific medication, and then the medication with a relatively high probability of treatment continuation was labelled as the optimal medication. If the patient received the identical medication with the labelled optimal medication, they were classified as ‘recommended,’ and others were classified as ‘non-recommended’. The Kaplan-Meier survival analysis and log-rank test were used to analyse the difference in treatment continuation in the recommended group versus the non-recommended group (online supplemental figure 3).

All p values <0.05 were considered to be statistically significant, and all analyses were conducted using SAS Enterprise Guide V.6.1 (SAS Institute, Cary, North Carolina) and R software V.3.6.1 (R foundation, Vienna, Austria).


Table 1 shows the characteristics of study participants and the proportion of 1-year treatment continuations for risperidone and aripiprazole. There were 5109 patients in the risperidone cohort and 3393 patients in the aripiprazole cohort, and approximately 14.0% of them were continuously prescribed their initial medication for 1 year. The mean age of patients in the risperidone and aripiprazole cohorts was 13.8 (SD 3.4) and 14.1 (SD 3.4) years, respectively. Male patients were more prevalent (risperidone 57.3%; aripiprazole 62.5 %), and most patients were covered by the national health insurance (risperidone 85.0%; aripiprazole 93.8%). For 1 year before the index date, selective serotonin reuptake inhibitors (risperidone 26.8%; aripiprazole 34.5%) and antianxiety medications (risperidone 29.1%; aripiprazole 32.3%) were the top two psychotropic medications prescribed to patients. Depression was the most frequent diagnosis among the comorbidities of mental health disorder (risperidone 24.6%; aripiprazole 34.9%). The prescription of other medications and the utilisation of the healthcare system showed similar distribution in both cohorts. Overall, the distribution of characteristics between the training data set and test data set showed no statistical difference (online supplemental tables S3 and S4).

Table 1

Baseline characteristics of patients treated with risperidone and aripiprazole

Table 2 shows the performance of the machine learning and backward stepwise LR models for predicting the treatment continuation of both risperidone and aripiprazole in the test data set. Machine learning algorithms demonstrated better performance than the conventional stepwise LR in predicting the treatment continuation of both risperidone and aripiprazole; however, the performance of other machine learning algorithms also showed a close value to these metrics (online supplemental figure S4 and table S5). GBM was the best modelling algorithm in predicting the treatment continuation of risperidone, with the lowest BS value (0.121) and the highest AUROC (0.686) and AUPRC (0.269) values, closely followed by SuperLearner (BS 0.121; AUROC 0.685; AUPRC 0.262). In the prediction of aripiprazole, GBM, SuperLearner and random forest exhibited the best performance in terms of BS (GBM 0.114), AUROC (SuperLearner 0.688) and AUPRC (random forest 0.317). Regarding sensitivity, specificity, precision, accuracy, f1-score and MCC, all models demonstrated comparable performance to each other (online supplemental figure S5). Furthermore, although the conventional stepwise LR demonstrated worse performance than the machine learning algorithms, the difference in performance was small. For instance, the range of absolute difference in AUROC was just −0.027 to 0.001 and −0.016 to 0.006 for risperidone and aripiprazole, respectively.

Table 2

Performance of machine learning models in predicting treatment response

Figure 1 shows the clinical benefit of using the prediction models for improving treatment continuation by supporting treatment selection. Patients taking the recommended medication had a favourable outcome in treatment continuation compared with that of patients taking the non-recommended medication. In the overall cohort of the test data set, the proportion of patients with a treatment continuation in the recommended group was approximately 1.2 to 1.5-fold greater than that in the non-recommended group. Furthermore, during the follow-up period, patients in the recommended medication group had a significantly higher survival probability of treatment continuation than those in the non-recommended medication group. This tendency was consistently observed in the subgroup analysis stratified for the observed medications, although the statistical significance was not obtained in the aripiprazole group (online supplemental figure S6).

Table 3 shows the top 10 important predictors in the prediction models (all predictors for each model were presented in the online supplemental tables S6 and S7). Age and utilisation of healthcare system were important predictors in prediction models of both risperidone and aripiprazole. In the prediction models of risperidone, four models consistently considered anticholinergic use and history of attention deficit hyperactivity disorder as important predictors. Lithium use and bipolar disorder diagnosis were considered as important predictors in four of the six prediction models of aripiprazole.

Table 3

Top 10 important predictors of the prediction models for risperidone and aripiprazole*


To our knowledge, this is the first study to develop and validate prediction models for 1-year treatment continuation of antipsychotics in children and adolescents with schizophrenia. Although some studies have reported prediction models for diverse treatment responses of antipsychotics, there is a scarcity of studies particularly conducted in this population. Moreover, the majority of previous studies applied only one prediction algorithm, whereas the present study has applied and compared several modelling algorithms (including both conventional LR and modern machine learning). The clinical benefit of prediction models for antipsychotic treatment was also evaluated.

In the present study, machine learning algorithms provided better performance than the conventional stepwise LR in predicting treatment continuation of antipsychotics. However, the increase in performance was too limited to confirm their outperformance over the conventional LR algorithm. This finding supported previous studies, such as Christodoulou et al, that compared the performance of LR and machine learning algorithms for clinical prediction modelling. They demonstrated no evidence that machine learning has a performance benefit over LR models in predicting clinical risk.21 Machine learning has the advantage of improving performance by controlling both linear and non-linear interactions between outcomes and predictors; however, all these algorithms have disadvantages that require the tuning of complex hyperparameters, a high burden of data and cost and difficulty with interpretation.22 Meanwhile, the conventional regression method has the advantage of being easy to interpret and consuming less time to develop.22 Hence, conventional LR could be an attractive approach for predicting the treatment continuation of antipsychotics because of its advantages if it shows similar performance to machine-learning.

Machine learning is generally considered a better approach for prediction models than conventional LR, but machine learning failed to demonstrate its superiority to conventional LR in this study. There are some possible explanations for this finding. First, the number of sample cases and predictors might be insufficient for machine learning algorithms. Machine learning methods are data-driven algorithms,22 which show better performance than conventional regression models when using massive data consisting of numerous predictors.23 Second, most of the predictors used in our study were clinically meaningful predictors (socioeconomic factors, concomitant use of other psychotropics and comorbidities of mental health disorders), which might limit the ability of machine learning to discover the hidden interaction between predictors and outcomes. Previous studies that used structured data with clinically meaningful predictors, like our study, also reported little difference in performance between algorithms.24 25 Third, as this study was conducted using claims data, most predictors were formed as categorical variables. One previous study suggested that machine learnings are more suitable for continuous variables than for categorical variables because they are non-parametric models and cannot assume linearity for predicting outcome association.26 Furthermore, in the sensitivity analysis considering all predictors as categorical variables, the performance of models was worse than that of the primary analysis (online supplemental table S8).

Furthermore, the patients in the recommended group were more likely to continue their first-line treatment than those in the non-recommended group. The ratio of treatment continuation rate at the last follow-up between the recommended and nonrecommended groups ranged from 1.20 to 1.50. In a recent study conducted by Wu et al, mostly on adult patients (mean age 36.7 years, SD 14.3) in Taiwan, they developed a prediction model for each antipsychotic medication using the machine learning method.12 They reported a similar range of increasing treatment success rate (treatment continuation for 1 year) from 1.16-fold to 1.48-fold when the individualised treatment rule was applied according to prediction models compared with the non-applied treatment.12 Due to the heterogeneity of data source, study definition and patient demographics, it is difficult to directly compare the result of the present study with their study. However, considering the consistent result with that study, the present result provided additional empirical evidence that the prediction model has the potential to be a useful tool to improve treatment response in clinical practice.

This study has several limitations. First, as schizophrenia was determined using the ICD-10 code in our study, the possibility of misdiagnosis might not be excluded, especially for EOS. Vernal et al have shown that the diagnosis of EOS has generally good validity for the paediatric populations in claims data but found a slightly higher number of false positives for diagnosis in the outpatient setting.27 Second, patients might discontinue or change the antipsychotic treatment regimen because of worsening of symptoms or the onset of other mental health disorders, not because they showed a poor response to their first-line antipsychotic agent. To overcome this limitation, several events indicating the worsening of psychiatric symptoms were included to identify poor treatment results.12 28 Third, it was impossible to include all predictors that might improve the accuracy of the prediction model, such as the type of schizophrenia, disease severity and residential area, due to the lack of data. Fourth, it is possible there are other machine learning algorithms that might outperform conventional regression. Finally, there might be potential bias because of the lack of external validation in populations with different demographic profiles.29 However, the purpose of our study was preliminary research to develop and compare the performance of prediction models for antipsychotic treatment response. This limitation regarding the generalisability may be addressed in future studies through external validation in various clinical settings.30

Clinical implications

Machine learning algorithms provided significantly limited performance improvement over conventional LR for predicting the 1-year treatment continuation of antipsychotics in children and adolescents with schizophrenia. Given the difficulties of hyperparameter tuning and understanding the result of machine learning, conventional LR might still be an attractive approach for constructing prediction models. The application of prediction models might be a helpful decision tool for antipsychotic treatment at the individual patient level.

Data availability statement

Data are available upon reasonable request. This study used Health Insurance Review and Assessment Service database (M20190117533). Requests to access these datasets should be directed to HIRA; Official website of HIRA:; Contact information of data access committee: +82-33-739-1083.

Ethics statements

Patient consent for publication


Supplementary materials

  • Supplementary Data

    This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.


  • Contributors SMJ, JC and J-WK had full access to all study data and take responsibility for the integrity of the data and the accuracy of the data analysis. SMJ and J-WK were responsible for the study concept and design. All authors were involved in the acquisition and interpretation of the data. SMJ and JC drafted the manuscripts. All authors critically revised the manuscripts. All authors provided their final approval of the version to be published and agreed to be accountable for all aspects of the work. SMJ and JC contributed equally. J-WK is responsible for the overall content as the gurantor.

  • Funding This study was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (NRF-2018R1D1A1B07043416). This research was supported by the BK21 FOUR Community-Based Intelligent Novel Drug Discovery Education Unit, College of Pharmacy and Research Institute of Pharmaceutical Sciences, Kyungpook National University. The supporting agencies had no role in the design and conduct of the study; collection, management, analysis and interpretation of the data; preparation, review or approval of the manuscript; and decision to submit the manuscript for publication.

  • Competing interests None declared.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.