Randomised trials are designed to provide an unbiased estimate of treatment effect in the population studied but can still be prone to a number of biases both within the individual trial and when considering all relevant randomised evidence.
Within a single trial the risk of bias has been well studied (eg: Cochrane Risk of Bias tool). But a special issue arises related to post-hoc decision rules in the analysis due to changes in the trial progress and emerging evidence from other trials. Under what circumstances is it appropriate to modify the planned analysis without biasing the results?
A second issue relates to combining the results of all relevant randomised clinical trials. Over 30 years ago he demonstrated that a review confined to just published trials may be biased, if the unpublished trials (which were more likely to have been negative) were not included. A systematic review of all trials (published or not) is preferred and, if not feasible, then a review confined to just the prospectively registered trials. This strategy has motivated the efforts internationally to register all trials prospectively with the trial registry being the principle source of trials for inclusion.
A third and related question relates to the post-hoc nature of the systematic review. Often reviews of the current evidence are partly based on knowledge of some or many of the trial results to be included. Further, unlike a single randomised trial where a primary outcome is defined, the same set of trials (or subsets of them) may be used to address many overlapping questions each in a different forum. Hence the problem of multiplicity may be addressed in a single trial but remains a big challenge for combined analyses of trials and especially prone to post-hoc decision rules. Prospective meta-analysis provide a strategy to address these challenges and provides similar strengths to a single large scale clinical trial.
For each of these issues several examples (over many years) will be presented to illustrate approaches as well as some potential pitfalls.