Outline

• [Background]
• Summary points
• Evaluating systematic reviews
  • Question 1: Can you find an important clinical question which the review addressed?
  • Question 2: Was a thorough search done of the appropriate databases and were other potentially important sources explored?
• Meta-analysis for the non-statistician
• Explaining heterogeneity
• REFERENCES
• GRAPHICS
  • Figure 1
  • Table 1
  • Table 2
  • Table 3
  • Figure 4
  • Figure 2
  • Figure 3

This is the ninth in a series of 10 articles introducing non-experts to finding medical articles and assessing their value

Summary points

A systematic review is an overview of primary studies that used explicit and reproducible methods

A meta-analysis is a mathematical synthesis of the results of two or more primary studies that addressed the same hypothesis in the same way

Although meta-analysis can increase the precision of a result, it is important to ensure that the methods used for the review were valid and reliable

Remember the essays you used to write as a student? You would browse through the indexes of books and journals until you came across a paragraph that looked relevant, and copied it out. If anything you found did not fit in with the theory you were proposing. you left it out. This, more or less, constitutes the methodology of the journalistic review-an overview of primary studies which have not been identified or analysed in a systematic (standardised and objective) way.

In contrast, a systematic review is an overview of primary studies which contains an explicit statement of objectives, materials, and methods and has been conducted according to explicit and reproducible methodology (**).

Figure 1. Methodology for a systematic review of randomised controlled trials

Some advantages of the systematic review are given in box 1. When a systematic review is undertaken, not only must the search for relevant articles be thorough and objective, but the criteria used to reject articles as "flawed" must be explicit and independent of the results of those trials. The most enduring and useful systematic reviews, notably those undertaken by the Cochrane Collaboration, are regularly updated to incorporate new evidence. (2)

Box 1 Advantages of systematic reviews (3)

Many, if not most, medical review articles are still written in narrative or journalistic form. Professor Paul Knipschild has described how Nobel prize winning biochemist Linus Pauling used selective quotes from the medical literature to "prove" his theory that vitamin C helps you live longer and feel better. (3,4) When Knipschild and his colleagues searched the literature systematically for evidence for and against this hypothesis they found that, although one or two trials did strongly suggest that vitamin C could prevent the onset of the common cold, there were far more studies which did not show any beneficial effect.

Experts, who have been steeped in a subject for years and know what the answer "ought" to be, are less able to produce an objective review of the literature in their subject than non-experts. (5,6) This would be of little consequence if experts' opinions could be relied on to be congruent with the results of independent systematic reviews, but they cannot. (7)

Evaluating systematic reviews

Question 1: Can you find an important clinical question which the review addressed?

Question 2: Was a thorough search done of the appropriate databases and were other potentially important sources explored?

Box 2 Checklist of data sources for a systematic review

Question 3: Was methodological quality assesed and the trials weighted accordingly?

One of the tasks of a systematic reviewer is to draw up a list of criteria, including both generic (common to all research studies) and particular (specific to the field) aspects of quality, against which to judge each trial (see box 3 (**)). However, care should be taken in developing such scores since there is no gold standard for the "true" methodological quality of a trial (12) and composite quality scores are often neither valid nor reliable in practice. (13,14) The various Cochrane collaborative review groups are developing topic-specific methodology for assigning quality scores to research studies. (15)

Box 3 Assigning weight to trials in a systematic review

Question 4: How sensitive are the results to the way the review has been done?

Carl Counsell and colleagues "proved" (in the Christmas 1994 issue of the BMJ) an entirely spurious relationship between the result of shaking a dice and the outcome of an acute stroke. (16) They reported a series of artificial dice rolling experiments in which red, white, and green dice represented different therapies for acute stroke. Overall, the "trials" showed no significant benefit from the three therapies. However, the simulation of a number of perfectly plausible events in the process of meta-analysis-such as the exclusion of several of the "negative" trials through publication bias, a subgroup analysis which excluded data on red dice therapy (since, on looking back at the results, red dice appeared to be harmful), and other, essentially arbitrary, exclusions on the grounds of "methodological quality"-led to an apparently highly significant benefit of "dice therapy" in acute stroke.

If these simulated results pertained to a genuine medical controversy, how would you spot these subtle biases? You need to work through the "what ifs". What if the authors of the systematic review had changed the inclusion criteria? What if they had excluded unpublished studies? What if their "quality weightings" had been assigned differently? What if trials of lower methodological quality had been included (or excluded)? What if all the patients unaccounted for in a trial were assumed to have died (or been cured)?

An exploration of what ifs is known as a sensitivity analysis. If you find that fiddling with the data in various ways makes little or no difference to the review's overall results, you can assume that the review's conclusions are relatively robust. If, however, the key findings disappear when any of the what ifs changes, the conclusions should be expressed far more cautiously and you should hesitate before changing your practice in the light of them.

Question 5: Have the numerical results been interpreted with common sense and due regard to the broader aspects of the problem?

Any numerical result, however precise, accurate, "significant," or otherwise incontrovertible, must be placed in the context of the painfully simple and often frustratingly general question which the review addressed. The clinician must decide how (if at all) this numerical result, whether significant or not, should influence the care of an individual patient. A particularly important feature to consider when undertaking or appraising a systematic review is the external validity or relevance of the trials that are included.

Meta-analysis for the non-statistician

These days, the results of meta-analyses tend to be presented in a fairly standard form, such as is produced by the computer software MetaView. ** is a pictorial representation (colloquially known as a "forest plot") of the pooled odds ratios of eight randomised controlled trials which each compared coronary artery bypass grafting with percutaneous coronary angioplasty in the treatment of severe angina. (17) The primary (main) outcome in this meta-analysis was death or heart attack within one year.

[not reproducable]
Figure 2. Pooled odds ratios of eight randomised controlled trials of coronary artery bypass grafting against percutaneous coronary angioplasty, shown in MetaView format Reproduced with authors' permission. (17)

The horizontal line corresponding to each of the eight trials shows the relative risk of death or heart attack at one year in patients randomised to coronary angioplasty compared to patients randomised to bypass surgery. The "blob" in the middle of each line is the point estimate of the difference between the groups (the best single estimate of the benefit in lives saved by offering bypass surgery rather than coronary angioplasty), and the width of the line represents the 95% confidence interval of this estimate. The black line down the middle of the picture is known as the "line of no effect," and in this case is associated with a relative risk of 1.0.

If the confidence interval of the result (the horizontal line) crosses the line of no effect (the vertical line), that can mean either that there is no significant difference between the treatments or that the sample size was too small to allow us to be confident where the true result lies. The various individual studies give point estimates of the relative risk of coronary angioplasty compared with bypass surgery of between about 0.5 and 5.0, and the confidence intervals of some studies are so wide that they do not even fit on the graph. Now look at the tiny diamond below all the horizontal lines. This represents the pooled data from all eight trials (overall relative risk of coronary angioplasty compared with bypass surgery = 1.08), with a new, much narrower, confidence interval of this relative risk (0.79 to 1.50). Since the diamond firmly overlaps the line of no effect, we can say that there is probably little to choose between the two treatments in terms of the primary en! d point (death or heart attack in the first year). Now, in this example, every one of the eight trials also suggested a non-significant effect, but in none of them was the sample size large enough for us to be confident in that negative result.

Note, however, that this neat little diamond does not mean that you might as well offer coronary angioplasty rather than bypass surgery to every patient with angina. It has a much more limited meaning-that the average patient in the trials presented in this meta-analysis is equally likely to have met the primary outcome (death or myocardial infarction within a year), whichever of these two treatments they were randomised to receive. If you read the paper by Pocock and colleagues (17) you would find important differences in the groups in terms of prevalence of angina and requirement for further operative intervention after the initial procedure.

Explaining heterogeneity

The definitive test for heterogeneity involves a slightly more sophisticated statistical manoeuvre than holding a ruler up against the forest plot. The one most commonly used is a variant of the chi sup 2 (chi square) test, since the question addressed is whether there is greater variation between the results of the trials than is compatible with the play of chance. Thompson (18) offers the following rule of thumb: a chi sup 2 statistic has, on average, a value equal to its degrees of freedom (in this case, the number of trials in the meta-analysis minus one), so a chi sup 2 of 7.0 for a set of eight trials would provide no evidence of statistical heterogeneity. Note that showing statistical heterogeneity is a mathematical exercise and is the job of the statistician, but explaining this heterogeneity (looking for, and accounting for, clinical heterogeneity) is an interpretive exercise and requires imagination, common sense, and hands-on clinical or research! experience.

(**) shows the results of ten trials of cholesterol lowering strategies. The results are expressed as the percentage reduction in risk of heart disease associated with each reduction of 0.6 mmol/l in serum cholesterol concentration. From the horizontal lines which represent the 95% confidence intervals of each result it is clear, even without knowing the chi sup 2 statistic of 127, that the trials are highly heterogeneous. Correcting the data for the age of the trial subjects reduced this value to 45. In other words, much of the "incompatibility" in the results of these trials can be explained by the fact that embarking on a strategy which successfully reduces your cholesterol level will be substantially more likely to prevent a heart attack if you are 45 than if you are 85.

Figure 3. Reduction in risk of heart disease by strategies for lowering cholesterol. Reproduced with permission from Chalmers and Altman (18)

Clinical heterogeneity, essentially, is the grievance of Professor Hans Eysenck, who has constructed a vigorous and entertaining critique of the science of meta-analysis. (19) In a world of lumpers and splitters, Eysenck is a splitter, and it offends his sense of the qualitative and the particular to combine the results of studies which were done on different populations in different places at different times and for different reasons.

Eysenck's reservations about meta-analysis are borne out in the infamously discredited meta-analysis which showed (wrongly) that giving intravenous magnesium to people who had had heart attacks was beneficial. A subsequent megatrial involving 58 000 patients (ISIS-4) failed to find any benefit, and the meta-analysts' misleading conclusions were subsequently explained in terms of publication bias, methodological weaknesses in the smaller trials, and clinical heterogeneity. (20,21)

The articles in this series are excerpts from How to read a paper: the basics of evidence based medicine. The book includes chapters on searching the literature and implementing evidence based findings. It can be ordered from the BMJ Publishing Group: tel 0171 383 6185/6245; fax 0171 383 6662. Price [pound sign]13.95 UK members, [pound sign]14.95 non-members.

Thanks to Professor Iain Chalmers for advice on this chapter.

References

2. Bero L. Rennie D. The Cochrane Collaboration: preparing, maintaining, and disseminating systematic reviews of the effects of health care. JAMA 1995;274:1935-8. [Back to Summary points]

3. Chalmers I, Altman DG, eds. Systematic reviews. London: BMJ Publishing Group. 1995. [Back to Summary points, GRAPHICS:Table 1]

4. Pauling L. How to live longer and feel better. New York: Freeman, 1986. [Back to Summary points]

5. Oxman AD, Guyatt GH. The science of reviewing research. Ann NY Acad Sci 1993;703:125-31. [Back to Summary points]

6. Mulrow C. The medical review article: state of the science. Ann Intern Med 1987;106:485-8. [Back to Summary points]

7. Antman EM, Lau J, Kupelnick B. Mosteller F, Chalmers TC. A comparison of results of meta-analyses of randomised controlled trials and recommendations of clinical experts. JAMA 1992;268:240-8. [Back to Summary points]

8. Koudstaal P. Secondary prevention following stroke or TLA in patients with non-rheumatic atrial fibrillation: anticoagulant therapy versus control. Cochrane Database of Systematic Reviews. Oxford: Cochrane Collaboration, 1995. (Updated 14 February 1995.) [Back to Evaluating systemati..:Question 1: Can you ..]

9. Greenhalgh T. Searching the literature. In: How to read a paper. London: BMJ Publishing Group. 1997:13-33. [Back to Evaluating systemati..:Question 2: Was a th..]

10. Knipschild P. Some examples of systematic reviews. In: Chalmers I. Altman DG. Systematic reviews. London: BMJ Publishing Group. 1995:9-16. [Back to Evaluating systemati..:Question 2: Was a th..]

11. Knipschild P. Searching for alternatives: loser pays. Lancet 1993;341:1135-6. [Back to Evaluating systemati..:Question 2: Was a th..]

12. Oxman A. ed. Preparing and maintaining systematic reviews. In: Cochrane Collaboration handbook, section VI. Oxford: Cochrane Collaboration. 1995. (Updated 14 July 1995.) [Back to Evaluating systemati..:Question 2: Was a th..]

13. Emerson JD, Burdick E, Hoaglin DC, Mosteller F, Chalmers TC. An empirical study of the possible relation of treatment differences to quality scores in controlled randomized clinical trials. Controlled Clin Trials 1990;11:339-52. [Back to Evaluating systemati..:Question 2: Was a th..]

14. Moher D. Jadad AR, Tugwell P. Assessing the quality of randomized controlled trials: current issues and future directions. Int J Health Technol Assess 1996;12:195-208. [Back to Evaluating systemati..:Question 2: Was a th..]

15. Garner P, Hetherington J. Establishing and supporting collaborative review groups. In: Cochrane Collaboration handbook, section II. Oxford: Cochrane Collaboration. 1995 (Updated 14 July 1995.) [Back to Evaluating systemati..:Question 2: Was a th..]

16. Counsell CE, Clarke MJ, Slattery J. Sandercock PAG. The miracle of DICE therapy for acute stroke: fact or fictional product of subgroup analysis? BMJ 1994;309:1677-81. [Back to Evaluating systemati..:Question 2: Was a th..]

17. Pocock SJ, Henderson RA, Rickards AF, Hampton JR, Sing SB III. Hamm CW. et al. Meta-analysis of randomised trials comparing coronary angioplasty with bypass surgery. Lancet 1995;346:1184-9. [Back to Meta-analysis for th.., GRAPHICS:Figure 2]

18. Thompson SG. Why sources of heterogeneity in meta-analysis should be investigated. In: Chalmers I, Altman DG. Systematic reviews. London, BMJ Publishing Group. 1995:48-63. [Back to Explaining heterogen.., GRAPHICS:Figure 3]

19. Eysenck HJ. Problems with meta-analysis. In: Chalmers I, Altman DG. Systematic reviews. London: BMJ Publishing Group. 1995:64-74. [Back to Explaining heterogen..]

20. Magnesium, myocardial infarction, meta-analysis and mega-trials. Drug Ther Bull 1995;33:25-7. [Back to Explaining heterogen..]

21. Egger M, Davey Smith G. Misleading meta-analysis: lessons from "an effective, safe, simple" intervention that wasn't. BMJ 1995;310:752-4. [Back to Explaining heterogen..]

TORONTO

HOME

---