Search strategy

This systematic review was completed and reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines.13 As this was an evidence synthesis of existing research, ethical approval was not required; however, we fully complied with the Declaration of Helsinki on medical research.

Aiming to identify publications on mental health training for non-specialist groups worldwide, we searched for terms related to mental health, training, community and evaluation in the following electronic databases on 31 May 2017: PubMed, PsycINFO, CINAHL (using EBSCOHost interface), Cochrane and Web of Science. We included controlled vocabulary terms for each database and searched for articles published from January 2008 to May 2017 (inclusive). The search strategy (table 1) was designed after careful examination of key studies in the literature, and by following the Participants, Interventions, Comparators and Outcomes (PICO) process for evidence-based practice.14 The full search strategy for the PubMed database is provided as an example in the online supplementary material.

We included studies reported in English, meeting the following criteria in line with the PICO design:

• Participants: Following WHO guidance for increasing mental healthcare capacity through task-shifting,12 we included studies in which trainees were non-specialist healthcare workers (eg, generalist medical practitioners, nurses, general community mental healthcare workers and non-medical volunteers). Studies focusing on specialists (eg, psychiatrists) and medical students were excluded as these groups may have received specialist training in addition to a short training course. In line with WHO guidance, we were interested in the efficacy of programmes that could be readily administered without extensive training. We therefore wanted to ensure that this potential confounding factor was removed from our search strategy.

• Intervention: Studies describing the training course format and outcome in general mental health were included. Duration or format were not used as selection criteria. We excluded studies providing training to care for specific subpopulations (eg, children, veterans and/or specific ethnic groups), for one specific mental illness (eg, depression alone) and those covering substance abuse (eg, alcoholism) or mental illnesses secondary to other medical conditions (eg, HIV/AIDS). A further search term, related to ‘primary care’, was instead used to identify courses that focused on integration of mental health into primary care in line with WHO guidelines.

• Comparison: Studies were not required to have a control comparison group, due to the exploratory nature of the review.

• Outcomes: We included studies that evaluated training course outcomes via quantitative or qualitative methods, or a combination of both. We excluded studies that did not provide any evaluation data.

References identified through the search strategy were uploaded into EndNote (X7, Thomson Reuters). After deduplication, titles and abstracts were independently double-screened following the eligibility criteria. Studies meeting the inclusion criteria were obtained as full text articles and independently double-screened by two reviewers using the same criteria. Entries that matched between the two reviewers were included. Unmatched entries were only included following resolution through discussion.

Data extraction

Standardised, piloted data extraction sheets were developed to ensure consistency between studies. Data were extracted by one reviewer and independently double-checked by another. Additional quality control of a random sample was carried out by a third reviewer. Data extracted for each study included, where possible, primary care factors (country of origin, World Bank economic status, number and type of trainees), training factors (types of disorder included, method of training, duration and type of course and frequency of training) and outcome factors (outcomes measured, method and timing of evaluation). Any disagreements were resolved through discussion.

Methodological assessment

We followed the schema established by Liu et al 15 for assessing methodological quality of mental health training courses in Africa, to allow wider comparisons within the field. This framework is based on a combination of validated methods, including the Newcastle-Ottawa Scale,16 Grading of Recommendations Assessment, Development and Evaluation17 and Methodological Index for Non-Randomised Studies.18 It examines the selection (five criteria) and evaluation methods (five criteria) in each study. Studies are given one point for each of the criteria they satisfy. Authors AC, GL and DV undertook this assessment and resolved any disagreements through discussion; TVB performed the quality control.

Classification of training courses and outcomes

Course trainees were categorised according to WHO classifications of healthcare workers.19 Since this only includes healthcare workers, we added three further categories, namely: volunteers, mental health consumers/carers and non-medical staff. The latter included police officers, farm inspection officers, disaster relief staff, educators and housing outreach workers. Studies identified and included these groups as first-line contacts for communities in distress or those which are difficult to reach.

In terms of content, courses were classified as ‘specific’ if they addressed one particular aspect of mental healthcare (eg, a specific management or counselling technique), and ‘general’ if they covered general psychiatry. A third category, ‘emergency mental health’, covered courses teaching mental health first aid and mental health in disaster settings. Additionally, we screened courses to identify if they had specifically used the mhGAP guide to create training modules.

Following Liu et al,15 interventions were classified as ‘didactic’ when they were exclusively made up of lectures and as ‘interactive’ when they included active trainee participation such as role play, small-group work, case discussions or clinical skills. ‘Mixed sessions’ included both didactic and interactive elements. We also used the schema adapted from Kirkpatrick20 to classify types of evaluative outcome into one or more of seven areas: (a) satisfaction with training (evaluation of reaction), (b) change in attitude towards the importance of mental health, (c) change in confidence, (d) change in knowledge, (e) change in clinical skills (evaluation of learning), (f) change in clinical practice (evaluation of behaviour) and (g) change in patient outcomes (evaluation of results).21 For the purpose of this systematic review, we defined skill as the ability to perform a task well, usually gained by training or experience.22 We then reported how this skill was measured. We deliberately followed similar classification strategies to Liu et al 15 to encourage establishment of a systematic method of review in this area, allowing cross-comparison between reviews.

Patient and public involvement

There was no patient or public involvement in this review, this was a synthesis of existing published data.

Findings

Our initial search identified 17 877 results after deduplication (n=3600). Screening of abstracts for PICO eligibility criteria resulted in inclusion of 47 papers from reviewer 1 and 64 papers from reviewer 2. Studies were discussed by reviewers to agree on validity of inclusion. Papers describing the same study were evaluated and excluded if they added no new information. A total of 30 studies were ultimately included, of which one was unobtainable. A random selection of papers was quality-controlled. Full PRISMA search strategy flow shown in figure 1.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Preferred Reporting Items for Systematic Reviews and Meta-Analyses search strategy.

Country and economic status

This systematic review identified that training interventions were implemented in 16 countries (figure 2): four in the UK, three each in Australia, Canada and India, two each in China, Malawi, Nigeria and Zimbabwe and one each in Iraq, Kenya, Nepal, Norway, Sierra Leone, Sri Lanka, USA and Pacific Small Island States. Countries were classified according to World Bank Economic Status (source: World Bank). Under this classification, six training courses took place in low-income settings, seven in lower-middle-income settings, two in upper-middle-income and 13 in high-income settings. Pacific Small Island States was categorised as an ‘aggregates’ nation. International organisations were involved in the implementation of two of the courses: The Catholic Agency for Overseas Development provided medication and funded counsellors’ salaries for the course in Sierra Leone, and the International Medical Corps appointed mental health advisors to oversee training in Iraq.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2

Global distribution of training courses for included studies.

Studies were independently assessed by three reviewers using methodological criteria outlined by Liu et al 15 (table 2). On comparison of findings, differences were resolved through discussion. Two areas proved challenging to assess; first, an agreed threshold for ‘sufficient’ detail for selection of the training sample, and second an agreed threshold for ‘representative’ selection of the evaluation sample. To clarify, the ‘training sample’ were the participants selected as trainees for each course, and the ‘evaluation sample’ consisted of the subgroup of trainees selected to participate in feedback/evaluation. In many cases, the evaluation samples were convenience samples, based on who was available and willing to provide feedback, rather than a representative group.

The median score of the studies in the methodological evaluation was five. A training sample of over 30 people was recruited in 22 (76%) studies, while 17 (59%) used a cohort that was representative of the target population. Selection of the training sample was adequately described in 17 (59%) studies. Only six (21%) trials used a control cohort, of which five used randomisations (four at clinic level and two by individual participants).

Selection of the evaluation sample was well characterised in 26 (90%) studies, but only 19 (66%) fully reported their evaluation and ensured evaluation samples were representative. Preintervention assessment was carried out in 19 (66%) studies and only 13 (45%) included long-term evaluation. The six studies that used a control cohort all used more detailed assessment tools than simple questionnaires, such as blinded reviewer scoring of competence of simulated patient consultations, rate of accurate clinic detection of mental disorders, data on diagnoses made by participants and direct observation of health worker skills. Therefore, the high-quality studies differentiated themselves through randomisation and moving beyond evaluation through the standard preintervention and postintervention questionnaire.

Classification and number of trainees

Community health workers were the most common type of trainee (table 3), featuring in more than half of interventions: 16 (55%). A total of 10 courses (34%) trained nurses, 7 (24%) trained general medical practitioners, 7 (24%) trained social workers and/or counsellors, 2 (7%) trained health service managers and 1 (3%) trained paramedics and clerical support workers. Seven courses (24%) trained non-medical staff, two (7%) trained volunteers and one (3%) trained service users and carers. In 12 interventions (41%), more than one type of trainee participated. Of these, five courses (17%) trained two different types of participants, two (7%) trained three types of participants, four (14%) trained four types of participants and one (3%) trained five types of participants. The latter course was particularly diverse, with trainees drawn from five different backgrounds, including physicians, nurses, social workers, paramedics and police officers. The number of trainees varied widely between interventions, ranging from just three to over 3500.

Course content

Training course curricula varied (table 3): 15 courses (52%) covered a ‘general’ curriculum, of which one also taught mental health first aid, one additionally addressed stigma and one included both. Of these general courses, two (7%) followed the same 5-day curriculum, namely the Kenya Medical Training College mental health primary care training toolkit created in Kenya and subsequently adapted for other countries. Eleven courses (38%) taught a ‘specific’ aspect of mental healthcare using a variety of previously established psychotherapies (eg, cognitive behavioural therapy), or focused on the development of teamwork skills via the New Ways of Working Framework, Access to Mental Health in Primary Care Programme, Rural Mental Health Inter-Professional Training Programme and Friendship Bench Programme. These teamwork development programmes were specifically created for the training interventions, most of which were tailored to the sociocultural background of the country in which they were implemented. Moreover, three courses (10%) focused on emergency mental health, of which two taught mental health first aid and one taught mental health in natural disasters.

In terms of teaching methods, five courses (17%) used didactic methods and six (21%) used interactive methods, though the majority of courses (62%) used a combination of methods providing an immersive learning experience. One course (3%) also offered a choice of teaching methods, based on participants’ favoured learning styles. In this case, trainees were more likely to drop out of self-directed learning than small group teaching. To provide access for remote trainees, two (7%) courses used video-conferencing.

Course lengths varied ranging from 1 day to spread across 2 years. More than half the courses (62%) ranged in length from 1 day to 2 weeks, and nine courses (31%) lasted between 2 weeks and 2 years. Length of training could not be determined for two courses (7%). Of the 29 courses identified by this study, 15 (52%) ran training over a continuous period, and 13 (45%) courses were sessional spread over a longer period. Course structure could not be determined for one course (3%).

Frequency of training

Twelve studies (41%) incorporated data from the same course run on multiple occasions in different localities (to improve access for trainees). The total numbers trained across these courses are listed in table 3. A further eight studies (28%) reviewed courses which had already been evaluated elsewhere and then adapted to incorporate changes. It was difficult to determine total numbers trained over time for these courses. Of note, one study was a follow-up randomised clinical trial for the Friendship Bench Project in Zimbabwe, as recommended in an earlier evaluation of the same project.

Evaluation methods

The majority of courses (66%) used a preintervention and postintervention design (table 3). Eleven courses (38%) also collected evaluation data at later time-points postcourse to assess longer-term changes, four were (14%) randomised controlled trials and one (3%) was a controlled trial. A total of 10 courses (34%) collected outcome measures after the intervention only. Of these, three (10%) collected data at repeated time points postintervention and one (3%) was a randomised controlled trial. One course (3%) was designed for data collection while the course was ongoing, comprising written feedback gathered from participants at the end of each training session.

The type of data collected and tools used for data collection varied enormously across interventions. The majority of courses (52%) collected quantitative data alone, while three courses (10%) collected qualitative data alone, and 11 courses (38%) collected both. The evaluation methods varied greatly with the majority of courses using written tools in the form of questionnaires or clinical vignettes. Further, focus groups or interviews with trainees were commonly used to establish the outcome of training courses. Some other courses examined case records or clinical notes of encounters to collect evaluation data, in several cases comparing clinical notes to patient status determined by previously validated screening tools, such as the General Health Questionnaire, Self-Rating Questionnaire and Structured Clinical Interview for Diagnostic and Statistical Manual of Mental Disorders (DSM-IV) for depression. In addition, a few courses used views of third parties as evaluation data (eg, course facilitator’s field notes, or subjecting trainees to observation by blinded psychiatrists who watched simulated video-taped consultations or clinical encounters with real patients).

Evaluation outcomes

Course evaluation measures also varied (table 4). The most commonly measured outcome (52%) was change in trainees’ attitude towards mental health. Of these 15 courses, 13 found an improvement in attitude with six reporting significant improvements, five found a qualitative improvement and two found an absolute improvement from baseline. One course found no significant change in trainees’ attitude preintervention and postintervention, and one course was an observational study testing significant difference in knowledge, attitude and clinical practice across trainee demographics, years of practice, practice setting and so on. The second most common outcome measured (45%) was knowledge. Of these, 10 courses found an improvement in knowledge postintervention, with six reporting significant improvement and four an absolute improvement. One course measured postintervention knowledge only, reporting it as ‘impressive’, one course reported no significant improvement and one was the observational study reported above. Clinical practice and clinical skills were measured by 11 courses (38%). Measurement of clinical practice was largely qualitative in nature, and suggested positive change in practice following training. Three courses (10%) attempted to quantify change in clinical practice, of which two found a significant improvement and one found no change. Clinical skills were assessed by 11 courses (38%). Of these, seven found a statistically significant improvement in clinical skills, two found a qualitative improvement, and two no improvement from baseline. Change in confidence was assessed by nine courses (31%), with seven finding statistically significant improvement in confidence and two an absolute improvement from baseline. Clinical outcome was assessed by six courses (21%) which all showed positive outcomes. Finally, nine courses (31%) assessed trainees’ satisfaction with the course. All received positive feedback from trainees, except the use of video-conferencing to facilitate remote learning. Trainees often offered helpful suggestions for improvement for future courses.

WHO policy uptake and direction of future research

A total of six studies (21%) referenced the WHO Mental Health Gap Action Plan (WHO, 2008) as their guiding principle, and five of these specifically used the mhGAP Intervention Guide to design training modules. A further nine studies (31%) used other works of the WHO in their studies; in particular, the WHO Disability Assessment Schedule version 2.023 to assess the outcomes of training, and the WHO Primary Care Guidelines for Mental Health.24 One study (3%) was funded by WHO Department of Mental Health and Substance Abuse.

Four studies (14%) detailed plans for ongoing training and two studies (7%) were run as pilot studies for a future more comprehensive version of the training course. Most studies suggested themes for future research, including the need for larger and more diverse training samples, more objective outcomes and more robust evidence in the form of randomised trials.