Subnavigation

• RHL home
• Adolescent sexual and reproductive health
• Fertility regulation
• Gynaecology, infertility and cancers
• HIV
• Improving clinical practice
• Newborn health
• Pregnancy and childbirth
• Sexually transmitted infections

Randomization to protect against selection bias in health-care trials

RHL Commentary by Seuc A

1. INTRODUCTION

There is considerable convincing logic and evidence of good results associated with randomization of allocation and its adequate concealment in health-care trials (1, 2). However, this has not prevented skeptics to question the value of randomization (3–5). Clearly, there is a fine but clear distinction between challenging the overuse and misuse of randomization in health-care trials and criticism of over-rating of randomized clinical trials in comparison with observational studies. Many authors have emphasized the complementary roles of experimental and observational studies, recommending against the unnecessary idealization of the randomized controlled trials as the “gold standard” (6–8) for research. Randomization is used in trials to reduce selection bias and increase comparability among groups with various confounding factors (known and unknown). This review (9) does not challenge the logic of the concept of randomization, but rather tries to assess the practical impact of randomization on health-care trials (and its adequate concealment) in terms of bias reduction.

2. METHODS OF THE REVIEW

The review authors planned to include systematic reviews and meta-analyses based on health-care trials; single case-studies, and systematic reviews or meta-analysis that incidentally reported a comparison of interest; they excluded simulation studies. The comparison of random allocation to non-random allocation involved: (i) randomized trials; (ii) non-randomized trials with concurrent controls; and (iii) non-equivalent control group design studies. Studies using historical controls and classical observational studies were excluded. The comparison of adequate concealment of allocation to inadequate or unclear concealment of allocation included only controlled trials with some sort of random assignment.

Subgroup analyses were planned separately for the comparison of randomized versus non-randomized studies, and studies with adequate versus inadequate concealment of allocation. Additionally, subgroup analyses were done and analysed considering if the intervention and/or the condition was fixed or allowed to vary across studies. In all cases the outcome was defined as the magnitude and direction of estimates of effect and imbalances in prognostic factors

The search was conducted using standard methods. The review authors recorded the reported relationship between randomization (or concealment of allocation) on the one hand, and estimates of effect on the other, and where possible converted the reported relationship into the relative over- or under-estimation of the relative risk reduction using the results of randomized trials (or randomized trials with concealed allocation) as reference. The risk of bias was assessed for each of the systematic reviews and meta-analyses included in the review.

The review authors made significant efforts to make the groups compared (randomized versus non-randomized trials, and adequate versus inadequate allocation concealment) as similar as possible in terms of type of patients, interventions, and follow-up; this is the main reason why the 2011 version of the review excluded as many as 22 studies that were included in the 2007 and 2008 versions of the review.

3. RESULTS OF THE REVIEW

Eighteen studies met the inclusion criteria, with a total of 1714 health-care trials plus 74 meta-analyses. The 18 studies reported 24 comparisons distributed as follows: (i) one comparison of randomized versus non-randomized studies of the same intervention and the same condition; (ii) one comparison of randomized versus non-randomized studies across different interventions for the same condition; (iii) nine comparisons of randomized versus non-randomized studies across different interventions and conditions; (iv) one comparison of adequate concealed versus inadequate concealed studies across different interventions for the same condition; and (v) 12 comparisons of adequate versus inadequate concealed trials across different interventions and conditions. All 18 studies had assessed impact on estimate of effect, and only one additionally had assessed the impact on imbalances in prognostic factors. All 18 studies were classified by the review authors as being at high risk of bias.

In summary, the results of the comparison of randomized versus non-randomized studies and of trials with adequate versus inadequate concealment of allocation were not consistent. In some instances non-randomized studies yielded larger estimates of effect and in other instances the opposite was found. When there were differences in the comparison of adequate versus inadequate concealment of allocation, most often trials with inadequate concealment yielded larger estimates of effects. In general, the review results were not able to predict the magnitude, or even the direction, of possible selection bias.

4. DISCUSSION

4.1. APPLICABILITY OF THE RESULTS

The review authors conclude that overall the evidence supports the logical arguments for using randomization of allocation and its concealment in health-care trials. However, they point out that most often non-randomized and randomized trials appear to have similar results, and that the bias (when present) can go in either direction.

In this respect it should be noted that the review authors found only one study/comparison for the same intervention and the same condition (comparing randomized versus non-randomized trials); as they rightly acknowledge, “comparisons across different interventions and conditions … are of questionable value”. In general, it could be expected that bias resulting from non-randomization and/or inadequate concealment of allocation depends both on the intervention and the condition.

The review authors also acknowledge that a possible cause of the inconsistent results observed is publication bias, which “would be the case if randomized trials were more likely to be published … than non-randomized studies (with historical controls)… “. I do agree and think this problem limits the applicability of the review results. The lack of proper randomization and adequate concealment of allocation makes it very difficult for a trial to reach visibility, not least publication in a peer-reviewed journal. In order to “succeed”, these trials need to thoroughly justify these limitations and in all other aspects of design and execution be almost “perfect”, which make these trials quite unrepresentative of all trials conducted without randomization and adequate concealment of allocation, no matter how comprehensive was the search for such trials. This should partly explain why this review does not find a consistent bias when randomized trials (with adequate concealment of allocation) are compared with non-randomized trials (with inadequate concealment of allocation).

Other studies have shown similar results. For example, Pildal et al. (10) assessed the impact of allocation concealment on conclusions drawn from (meta-analyses of) randomized trials, and found that the bias was smaller and less consistent than expected. Finally, as pointed out by Deeks (11), "meta-epidemiological techniques tend not to provide useful information on the sources of and degrees of bias in non-randomized studies owing to the existence of meta-confounding and lack of systematic or predictable bias in the results of non-randomized studies".

4.2. IMPLEMENTATION OF THE INTERVENTION

This review assesses the practical impact (in terms of bias) of a widespread accepted “intervention” introduced in 1948 (12), i.e. randomization of allocation and its adequate concealment. It does not propose a new procedure/intervention, and the overall conclusion is that these methods are beneficial. Therefore, there are no issues related to the implementation of a "new" intervention.

4.3. IMPLICATIONS FOR RESEARCH

Myths are not good for the advancement of science (11), but I am skeptical about the usefulness of meta-epidemiological techniques to fully clarify the impact of randomization and its concealment in bias reduction. The concepts that this review is challenging are logical, intuitive, and intimately connected to the basics of experimental research (1), and it seems nobody is really trying to erode them so far from a philosophical standpoint. In any case, future reviews would do better if they were able to include more comparisons keeping the same intervention and the same condition, as also pointed out by the review authors.

The indiscriminate over-rating of randomized clinical trials in comparison to observational studies is a different and real problem that is affecting research, researchers, and patients alike. Tackling this problem requires a high dose of common sense and context–objective analyses; research can make a very significant contribution to this as a variety of attractive methodological tools could be developed for the design of observational studies (for causal effects) to parallel the design of randomized experiments, as some have done not long ago (7, 8).

References

• Cox DR. Randomization for concealment. James Lind Library Bulletin: commentaries on the history of treatment evaluation 2009; www.jameslindlibrary.org; accessed: 25 July 2011.
• Liu PY, Anderson G, Crowley JJ. Observational studies and randomized trials. Correspondence to the New England Journal of Medicine 2000;343:1195.
• Manchikanti L, Pampati V. Research designs in interventional pain management: is randomization superior, desirable or essential? Pain Physician 2002;5:275-284.
• Black N. Why we need observational studies to evaluate the effectiveness of health care. BMJ 1996;312:1215-1218.
• Abel U, Koch A. The role of randomization in clinical studies: myths and beliefs. Journal of Clinical Epidemiology 1999;52:487–497.
• Concato J. Observational versus experimental studies: what’s the evidence for a hierarchy? The Journal of the American Society for Experimental Neurotherapeutics 2004;1:341–347.
• Rosenbaum PR. Observational study. Volume 3. In: Everitt BS & Howell DC, Eds. Encyclopedia of Statistics in Behavioral Science. Chichester: John Wiley & Sons, Ltd.; 2005.
• Rubin DB. The design versus the analysis of observational studies for causal effects: parallels with the design of randomized trials. Statistics in Medicine 2007;26:20–36.
• Odgaard-Jensen J, Vist GE, Timmer A, Kunz R, Akl EA, Schünemann H, et al. Randomisation to protect against selection bias in healthcare trials. Cochrane Database of Systematic Reviews 2011;Issue 4. Art. No.: MR000012; DOI: 10.1002/14651858.MR000012.pub3.
• Pildal J, Hrobjartsson A, Jørgensen KJ, Hilden J, Altman DG, Gøtzsche PC. Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials. International Journal of Epidemiology 2007;36:847–857.
• Deeks JJ, Dinnes J, D’Amico R, Sowden AJ, Sakarovitch C, Song F, et al. Evaluating non-randomized intervention studies. Health Technology Assessment 2003;7(27).
• Medical Research Council. Streptomycin treatment of pulmonary tuberculosis. BMJ 1948;ii: 769-782.

This document should be cited as: Seuc A. Randomization to protect against selection bias in health-care trials: RHL commentary (last revised: 1 January 2012). The WHO Reproductive Health Library; Geneva: World Health Organization.