Review Article
Uncertainty method improved on best–worst case analysis in a binary meta-analysis

https://doi.org/10.1016/j.jclinepi.2004.09.013Get rights and content

Abstract

Background

Most systematic reviewers aim to perform an intention-to-treat meta-analysis, including all randomized participants from each trial. This is not straightforward in practice: reviewers must decide how to handle missing outcome data in the contributing trials.

Objective

To investigate methods of allowing for uncertainty due to missing data in a meta-analysis.

Study Design and Setting

The Cochrane Library was surveyed to assess current use of imputation methods. We developed a methodology for incorporating uncertainty, with weights assigned to trials based on uncertainty interval widths. The uncertainty interval for a trial incorporates both sampling error and the potential impact of missing data. We evaluated the performance of this method using simulated data.

Results

The survey showed that complete-case analysis is commonly considered alongside best–worst case analysis. Best–worst case analysis gives an interval for the treatment effect that includes all of the uncertainty due to missing data. Unless there are few missing data, this interval is very wide. Simulations show that the uncertainty method consistently has better power and narrower interval widths than best–worst case analysis.

Conclusion

The uncertainty method performs consistently better than best–worst case imputation and should be considered along with complete-case analysis whenever missing data are a concern.

Section snippets

Handling missing binary data in a meta-analysis

The intention-to-treat approach is generally recommended to minimize bias in the design, conduct, and analysis of randomized controlled trials of effectiveness [1]. Most systematic reviewers aim to perform an intention-to-treat meta-analysis, including all randomized participants from each trial, but in practice this is not always straightforward. It is often apparent in published reports of randomized controlled trials that a number of participants have not been included in the analysis, even

Best–worst case analysis

Figure 1a shows a meta-analysis from a review comparing the effectiveness of two antimalarial drug treatments, artemether-lumefantrine (AL) and mefloquine plus artesunate (MA) [6]. The complete-case analysis is shown for each of the three trials (Lefevre 2001, van Vugt 2000, and van Vugt 1998; cited in [6]), along with a pooled estimate and confidence interval calculated using Woolf's inverse variance method [7]. The percentage of subjects with missing data within each trial is also shown.

Allowing for uncertainty due to missing binary data

In the context of missing survey responses, Molenberghs et al. [9] have described uncertainty as arising not only from imprecision due to sampling error, but also from ignorance due to incomplete data. They suggest the use of a region of uncertainty—a larger region, in the spirit of a confidence region but designed to capture the combined effects of imprecision and ignorance. A natural estimate of the region of uncertainty is the union of confidence regions for an estimated effect across a full

Simulation study

Using simulation, we investigated the attributes of the uncertainty method in comparison to complete-case and best–worst case analysis, in situations with and without an underlying treatment effect, and with informative and noninformative missing data. (Informative missing data occurs when observation of the outcome is dependent upon its value.)

We considered the scenario of a meta-analysis of three trials, each with the same sample size, and with an event rate of 0.25 in the control arm.

Simulation results

We first consider the situation with no underlying treatment effect (OR = 1) and informative missing data generated using MOR = 2 (Table 3). The full data analysis consistently provides close to 95% coverage, as expected, because there are no missing data in this analysis. Coverage of the complete-case method is close to 95% with small sample size or few missing data, but decreases as sample size and the proportion of missing data increase, indicating increasing bias. With 150 participants per arm

Discussion

The intention-to-treat principle is usually advocated for individual clinical trials. If some outcomes are unknown, no method of analysis can provide a pure intention-to-treat estimate. There has been much research and debate on the handling of missing data in an intention-to-treat analysis [1], [2]. Preferred approaches can be identified for some specific situations [2], [10], [11], [12]. For example, if the missing data can be assumed to be noninformative, then complete-case analysis will be

References (20)

  • C.C. Wright et al.

    Intention-to-treat approach to data from randomized controlled trials: a sensitivity analysis

    J Clin Epidemiol

    (2003)
  • P.P. Glasziou

    Meta-analysis adjusting for compliance: the example of screening for breast cancer

    J Clin Epidemiol

    (1992)
  • S. Hollis et al.

    What is meant by intention to treat analysis? Survey of published randomised controlled trials

    BMJ

    (1999)
  • W. Shih

    Problems in dealing with missing data and informative censoring in clinical trials

    Curr Control Trials Cardiovasc Med

    (2002)
  • M. Egger et al.

    Systematic reviews in healthcare: meta-analysis in context

    (2001)
  • D.L. Sackett et al.

    Evidence-based medicine: how to practice and teach EBM

    (1997)
  • J. Deeks et al.

    Analysing and presenting results

  • A.A.A. Omari et al.

    Artemether-lumefantrine for treating uncomplicated falciparum malaria

  • B. Woolf

    On estimating the relation between blood group and disease

    Ann Hum Genet

    (1955)
  • S. Hollis

    A graphical sensitivity analysis for clinical trials with non-ignorable missing binary outcome

    Stat Med

    (2002)
There are more references available in the full text version of this article.

Cited by (0)

View full text