Is one study ever enough?

The simple answer is ‘hardly ever’. Very seldom will one fair treatment comparison yield sufficiently reliable evidence on which to base a decision about treatment choices.

However, this does sometimes happen. Such rare single studies include one showing that taking aspirin during a heart attack reduces the risk of premature death; [1] another making clear that giving steroids to people with acute traumatic brain injury is lethal (see below); and a third identifying caffeine as the only drug known to prevent cerebral palsy in children born prematurely.

Usually, however, a single study is but one of several comparisons addressing the same or similar questions. So evidence from individual studies should be assessed alongside evidence from other, similar studies.

One of the pioneers of fair tests of treatments, the British statistician Austin Bradford Hill, said in the 1960s that reports of research should answer four questions:

  • Why did you start?
  • What did you do?
  • What did you find?
  • And what does it mean anyway?

These key questions are equally relevant today, yet they are too often inadequately addressed or overlooked completely. The answer to the last question – what does it mean? – is especially important since this is likely to influence decisions about treatment and future research.

Take the example of a short, inexpensive course of steroid drugs given to women expected to give birth prematurely. The first fair test of this treatment, which was reported in 1972, showed a reduced likelihood of babies dying after their mothers had received a steroid. A decade later more trials had been done, but these were small and the individual results were confusing, because none of them had taken systematic account of previous, similar studies. Had they done so, it would have been apparent that very strong evidence was emerging favouring a beneficial effect of the drugs. In fact, because this was not done until 1989, most obstetricians, midwives, paediatricians and neonatal nurses had meanwhile not realized the treatment was so effective. As a result, tens of thousands of premature babies had suffered and died unnecessarily.[2]

To answer the question ‘what does it mean?’, the evidence from a particular fair treatment comparison must be interpreted alongside evidence from the other, similar fair comparisons. Reporting new test results without interpreting them in the light of other relevant evidence, reviewed systematically, can delay identification of both useful and harmful treatments, and lead to unnecessary research.

  • Ichalmers

    Merge the two paras above ‘Synthesizing information from research’