Article Text
PDF
Abstract
Because of the rapidly increasing number of randomised controlled trials (RCTs) and meta-analyses in many fields, there is an urgent need to step up from meta-analyses to higher levels of aggregation of outcomes of RCTs. Network meta-analyses and umbrella reviews allow higher levels of aggregation of RCT outcomes, but cannot adequately cover the evidence for a whole field. The ‘Meta-Analytic Research Domain’ (MARD) may be a new methodology to aggregate RCT data of a whole field. A MARD is a living systematic review of a research domain that cannot be covered by one PICO. For example, a MARD of psychotherapy for depression covers all RCTs comparing the effects of all types of psychotherapy to control conditions, to each other, to pharmacotherapy and combined treatment. It also covers all RCTs comparing treatment formats, the effects in different target groups, subtypes of depression and secondary outcomes. Although the time and resources needed to build a MARD are considerable, they offer many advantages, including a comprehensive and consistent overview of a research field and important meta-analytic studies that cannot be conducted with conventional methods. MARDs are a promising method to step up the aggregation of RCTs to a next level and it is highly relevant to work out the methods of this approach in a more detailed way.
- adult psychiatry
- depression & mood disorders
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Because of the exponential increase in randomised trials over the past 50 years, the need for statistical integration of the results from multiple trials in meta-analyses has increased considerably. Meta-analyses aim to provide a robust overview of the efficacy of an intervention,1 they stand on the top of the hierarchy of evidence on interventions2 and are key components of evidence-based medicine.3
However, the number of meta-analyses has also increased exponentially over the past decades,2 and in more and more fields a higher level of aggregation is needed to get an overview of the effects of available interventions. For example, one umbrella review identified 247 meta-analyses of 5157 randomised controlled trials (RCTs) on psychotherapies.4 Another umbrella review of psychotherapies and pharmacotherapies for mental disorders included 102 meta-analyses of 3782 RCTs.5
There are several methods to step up to a higher level of aggregation. One method is the network meta-analysis (NMA), in which multiple interventions can be compared with each other and with common comparators.6 In an NMA, it is possible to examine comparative effects of multiple interventions for one disorder or condition and they can include hundreds of RCTs.7 However, an NMA still typically focuses on only one disorder or condition for one patient group.
Another method is the ‘umbrella review’, a ‘systematic review of systematic reviews’.2 8 Because an umbrella review can include multiple meta-analyses in one field, its scope is broader than that of an NMA and it does not have to be focused on one condition or patient group. However, umbrella reviews do not necessarily cover a whole field. There may be subfields that are not covered by a meta-analysis and are therefore not included in the umbrella review. They also include different meta-analyses with different methods, designs and outcomes, and are therefore typically very heterogeneous and typically only narratively describe a field. They are also always somewhat ‘lagging behind’, because of the end date for the searches of included meta-analyses, which in turn also have end dates for their searches.
In this paper, we describe another type of meta-analytic research which is broader than one living systematic review, meta-analysis or NMA, and gives a better and more complete overview of a field than an umbrella review: the ‘Meta-Analytic Research Domain’ (MARD).
What is a MARD?
A MARD is a living systematic review focusing on a specific research area, which is broader than what can be covered by one (network) meta-analysis.
It cannot be covered by one PICO (PICO stands for Participants, Intervention, Comparator, Outcome), as is the case for conventional living systematic reviews and meta-analyses, but it includes multiple PICOs that together cover a whole specific field. As in any living systematic review, the searches are done on a regular basis.9 In practice, some umbrella reviews and NMAs are already broader than one PICO, and some umbrella reviews also extract data on the level of the individual studies, so there is a grey area between umbrella reviews and MARDs. Table 1 gives an overview of the differences between umbrella reviews and MARDs. The concept of the MARD is related to so-called ‘evidence ecosystems’, which are also living systematic reviews in a specific area.10 However, evidence ecosystems are still focusing on one specific (network) meta-analysis of interventions for one clinical condition, while a MARD can cover a broader area and include multiple PICOs.
Comparison of umbrella reviews and MARDs
One example is our MARD on psychological treatment for depression.11 In this MARD, we include any RCT on psychological treatments of depression, in which participants from any age (eg, children, adolescents, adults, older adults) are recruited from any setting (eg, community, inpatients, outpatients) and represent multiple target groups (eg, women with perinatal depression, adults with somatic disorders and so on). We include any type of psychotherapy, delivered through any format (eg, face-to-face, Internet-based, telephone) and compared with any type of comparator (eg, inactive controls, another psychotherapy, pharmacotherapy, combined treatment). The searches are updated every year. We extract data on the participants, the interventions, the design of the study and risk of bias. We have now included more than 850 trials (www.metapsy.org). Over the past 15 years, we have published (network) meta-analyses on several different kinds of psychotherapy compared with control groups, compared with each other, with pharmacotherapy and with combined treatment (for an overview see Cuijpers11). We also published meta-analyses on different subgroups, like children and adolescents, older adults, inpatients and people with comorbid general medical disorders. We have examined delivery formats, length of treatment, digital interventions, number of sessions, secondary outcomes, like quality of life, social support and anxiety, and more methodological characteristics of studies, like publication bias and other risks of bias. Apart from all these ‘regular’ meta-analyses, we have also published systematic overviews of the results of the individual meta-analyses, which give a more or less complete overview of the field.11 12 The methods of the (network) meta-analyses conducted in this MARD are not different from other meta-analyses, but the difference is that together they cover a broad area of research, resulting in consistent study inclusion, data extraction, risk of bias methods and type of quality of evidence appraisal.
There are comparable MARDs on treatments of suicide,13 anxiety disorders,14 post-traumatic stress disorder15 and mental health problems in children and adolescents.16 Each of these includes several hundreds of randomised trials.
Advantages and dangers
MARDs have several important advantages. They give a broad overview of a field with consistent study inclusion, data extraction and risk of bias assessment, and are therefore superior to umbrella reviews, which include reviews with varying methodologies. MARDs also provide an overview of limitations and gaps in knowledge, and make it possible to see emerging trends in the field. MARDs also make it possible to conduct meta-analyses that cannot be conducted in other ways. For example, conventional meta-analyses and living systematic reviews of psychotherapies are not capable of examining secondary outcomes, because abstracts often do not refer to such outcomes and searches would only come up with a limited set of relevant trials. A MARD makes it possible to simply go through all the subsets of trials that potentially include such studies.17 Because MARDs examine a whole field of research, they are also important for meta-research (‘research on research’),18 because they allow to examine the methods and practice of the whole research field. MARDs allow ‘rapid’ meta-analyses on specific questions because no new searches have to be done and the data are already available. Such rapid analyses of subsets are useful for researchers, but also for developers of treatment guidelines and for clinicians and patients who would like to know the effects of a specific treatment, in a specific population for a specific outcome.
There are also disadvantages and dangers of MARDs. The biggest disadvantage of MARDs is that they require considerable resources and time from researches to build and maintain, as well as to find funders who are willing to pay for this over longer periods of time. In addition, a MARD can easily become dominant in a field, which may result in less scientific flexibility of analysing the research field. Furthermore, because data are always available, it is important to register new meta-analyses based on the data of the MARD in time, because there is a risk of exploring the data and only report findings that are ‘interesting’.
The exact methods for MARDs have not yet been worked out completely. How broad or narrow can the scope of a MARD be? Should it necessarily only include RCTs or can it also include open trials and observational studies? How should risk of bias be assessed? How can the results of the meta-analyses published within a MARD best be summarised in an overall overview? It is very important to further work out these methodologies.
Moving open science forward
The scientific community is at the dawn of a new open science paradigm pursuing ‘data-intensive scientific discovery’ where ‘all of the science literature is online, all of the science data is online, and they interoperate with each other’.19 MARDs are not meant to produce mere research outputs but rather provide a unique resource to test new hypotheses, enabling new scientific insights and driving innovation. As science becomes more data intensive and collaborative, MARDs will gain critical importance.
Meta-analyses have been called ‘the grandmother of the ‘big data’ and ‘open science’ movements’,1 because they include and integrate data from all available trials. MARDs have the potential to move open science one step forward. By making the data of a MARD open access, the whole field can benefit from that. A MARD gives a complete overview of the state of the art in a specific field, and in principle other researchers do not have to do new searches in bibliographic databases, extract data, calculate effect sizes or assess risk of bias of included studies, because that has already been done in the MARD. Considering the massive production of unnecessary, misleading and conflicted meta-analyses,20 MARDs can prevent unnecessary work and waste of resources. In the Metapsy project (www.metapsy.org), we have moved this one step further, by making meta-analytic data on psychotherapy for depression open. In addition, researchers can select online a subsample of studies and run a meta-analysis on this subsample through a Web app, without any additional software. Is cognitive behaviour therapy effective in older adults? Does group therapy work in perinatal depression? The shiny app allows to run sophisticated and always up-to-date meta-analyses online giving the answers to these questions. This will certainly result in a reduced number of redundant meta-analyses, because all data are available online and only the most important meta-analyses will be published that really present new knowledge.
Conclusions
Because of the rapidly increasing number of RCTs and meta-analyses in many fields, there is an urgent need to step up from meta-analyses and living systematic reviews to higher levels of aggregation of outcomes of RCTs. MARDs, living systematic reviews of research domains that cannot be covered by one PICO, are one of the most promising methods to realise this. Although the time and resources needed to build a MARD are considerable, they offer many advantages, including a comprehensive and consistent overview of a research field and important meta-analytic studies that cannot be conducted with conventional methods. MARDs are a promising method to step up the aggregation of RCTs to a next level and it is highly relevant to work out the methods of this approach in a more detailed way.
Ethics statements
Patient consent for publication
Ethics approval
Not applicable.
Footnotes
Contributors PC and EK had the idea for this paper. PC wrote the first draft. The content was generated in discussions among all authors. All authors read and approved the final version of the paper.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.