Current issues of ACP Journal Club are published in Annals of Internal Medicine


Editorial

Numbers needed to treat derived from meta-analysis: A word of caution

PDF

ACP J Club. 2003 Mar-Apr;138:A11. doi:10.7326/ACPJC-2003-138-2-A11

Related Content in the Archives
• Letter: Numbers needed to treat derived from meta-analysis: a word of caution



In clinical trials, treatment effects from binary outcomes, such as “alive” or “dead,” can be presented in various ways (e.g., relative risk reduction [RRR] and absolute risk reduction [ARR]) (1, 10). (See Glossary for definitions and calculations). Alternatively, the number needed to treat (NNT) is an expression of the number of patients who need to be treated to prevent one additional adverse event (2-4). Mathematically, the NNT equals the reciprocal of the ARR. Many journals now report results from clinical trials using the NNT, along with 95% confidence intervals (CIs) (5).

Since its introduction (3), a debate has ensued whether reporting NNTs from meta-analyses is misleading (4, 6, 7). ACP Journal Club has devoted attention to this debate, and methodologic questions have been raised in various review discussions and commentaries (8, 9). The objective of this editorial is to provide ACP Journal Club readers with caveats and suggestions when applying NNTs from a meta-analysis to an individual patient. We highlight 2 problems. First, NNTs from a meta-analysis are subject to variation in risk differences among the studies included in the meta-analysis as well as in baseline risks. Second, applying NNTs to an individual requires adjustment for their baseline risk.

NNTs derived from meta-analyses are affected by variations in risk differences among the studies, as well as baseline event rates in control groups of randomized controlled trials (1, 2). Summary estimates of NNTs assume constant risk differences between trials, a problematic assumption because of inevitable variation in baseline event rates between trials, differences in outcomes considered, effects of secular trends on disease risk, and differences in clinical setting as well as duration of follow-up (i.e., time horizon) (2, 10). In primary prevention of chronic disease, such as cardiovascular disease, the effect of time trends will become noticeable.

Several approaches have been introduced to derive or present NNTs from summary risk estimates of meta-analysis. ACP Journal Club readers have become familiar with standard summary presentations of meta-analytic research that provides NNTs based on pooled risk differences. However, in meta-analysis, the assumption underlying the pooling of any effect estimate is that measures are reasonably homogeneous across trials and that any variation can be attributed to random chance. In the case of the NNT, this means that absolute risk differences should be constant across studies. Growing empirical evidence suggests that relative risks and odds ratios in most instances provide more homogeneous estimates than absolute risk differences (1). For this reason, clinicians should use NNTs that are derived from pooled estimates of relative risks instead of absolute risk differences.

Application of NNTs from meta-analyses to individual patients also requires attention to baseline risks, subject to the following conditions: knowledge of the baseline risk of the study patients and the estimated risk of the individual patient to whom the NNT will be applied, and information on the time horizon of the study (e.g., the median follow-up time).

We illustrate these points using the meta-analysis of randomized controlled trials investigating the effect of n-3 polyunsaturated fatty acids on clinical endpoints in patients with coronary heart disease as an example (11, 12). The meta-analysis found a 19% (CI 10% to 27%) RRR of overall mortality favoring the use of n-3 polyunsaturated fatty acids. In the ACP Journal Club structured summary of this meta-analysis, the NNT derived from a pooled risk difference, and based on an average baseline risk of 9.5%, is 73 (CI 49 to 147) (11). However, this estimate does not reflect the high variability of baseline risk of the included trials. As discussed by the authors of the original meta-analysis (12), the baseline risk of included trials ranged from 2% to 22%. Based on the pooled relative risk estimate, the corresponding NNT for an average follow-up of 18 months is 263 (CI 185 to 500) for a baseline risk of 2% and 24 (CI 17 to 45) for a baseline risk of 22%. This example illustrates that if risk differences and baseline event rates in meta-analysis vary considerably across trials, the NNT derived from pooled absolute risk differences is not very informative and may be misleading if used to assess the benefit of a given intervention for an individual patient. Only if the baseline risk does not vary considerably across trials may the NNT derived from pooled risk differences be applied to determine the potential benefit for an individual patient.

Cook and Sackett (4) and Straus (13) have proposed a simple method to apply estimates of RRRs from single trials or a meta-analysis to individual baseline risk. They suggest comparing the patient’s baseline risk with that of a typical patient in the published trial. If the baseline risk of an individual patient is a factor f compared with the baseline risk of a typical study patient and the relative risk is constant, the ARR for the patient is scaled according to the same factor f. The estimated NNT corresponding to patients at the revised baseline risk is therefore simply the study NNT multiplied by f. Thus, in our example, if a high-risk patient is judged to have baseline risk 11 times higher than that of a low-risk patient, then f = 11 and the corresponding NNT is 24 × 11 = 264. Confidence intervals can also easily be obtained by multiplying the limits of the corresponding confidence interval from the original study with the factor f. However, as we have mentioned, this method should not be used with NNTs that are derived from pooled estimates of absolute risk differences. For example, if a clinician used an NNT of 73 based on absolute risk differences and a weighted control event rate of 9.5% to assess the benefit of n-3 polyunsaturated fatty acids for a patient with a baseline risk of 2%, the resulting NNT would be 347 (compared with 263 based on relative risk estimates). Therefore, clinicians should be wary when interpreting NNTs derived from pooled risk differences of a meta-analysis if baseline risks or absolute risk differences vary across trials. In this case, it makes no sense to use an NNT for the “average” single trial patient or the “average” meta-analysis patient.

Acknowledgments

We thank Matthias Egger and Brian Haynes for their critical review and contributions to this paper.

Arthur Marx, MD, MPH
Basel Institute for Clinical Epidemiology
Basel, Switzerland

Heiner C. Bucher, MD, MPH
Basel Institute for Clinical Epidemiology
Basel, Switzerland


References

1. Deeks JJ, Altman DG. Effect measures for meta-analysis of trials with binary outcomes. In: Egger M, Davey Smith G, Altman DG, eds. Systematic Reviews in Health Care. 2d ed. London: BMJ Publishing Group; 2001:313-35.

2. Ebrahim S. Numbers needed to treat derived from meta-analyses: pitfalls and cautions. In: Egger M, Davey Smith G, Altman DG, eds. Systematic Reviews in Health Care. 2d ed. London: BMJ Publishing Group;2001:386-99.

3. Laupacis A, Sackett DL, Roberts RS. An assessment of clinically useful measures of the consequences of treatment. N Engl J Med. 1988;318:1728-33. [PubMed ID: 3374545]

4. Cook RJ, Sackett DL. The number needed to treat: a clinically useful measure of treatment effect. BMJ. 1995;310:452-4. [PubMed ID: 7873954]

5. Altman DG. Confidence intervals for the number needed to treat. BMJ. 1998;317:1309-12. [PubMed ID: 9804726]

6. Smeeth L, Haines A, Ebrahim S. Numbers needed to treat derived from meta-analyses–sometimes informative, usually misleading. BMJ. 1999;318:1548-51. [PubMed ID: 10356018]

7. Lubsen J, Hoes A, Grobbee D. Implications of trial results: the potentially misleading notions of number needed to treat and average duration of life gained. Lancet. 2000;356:1757-9. [PubMed ID: 11095272]

8. Guyatt GH, Cook DJ, Jaeschke R. How should clinicians use the results of randomized trials? [Editorial]. ACP J Club. 1995;122:A12-3. [PubMed ID: 7704477]

9. Glasziou P, Guyatt GH, Dans AL, et al.Applying the results of trials and systematic reviews to individual patients [Editorial]. ACP J Club. 1998;129:A15-6. [PubMed ID: 9825009]

10. McAlister FA. Commentary: relative treatment effects are consistent across the spectrum of underlying risks…usually. Int J Epidemiol. 2002;31:76-7. [PubMed ID: 11914298]

11. Simon JA.Review: n-3 polyunsaturated fatty acids reduce fatal MI, sudden death, and overall mortality in coronary heart disease. ACP J Club. 2002;137:49. [PubMed ID: 12207427]

12. Bucher HC, Hengstler P, Schindler C, Meier G. N-3 polyunsaturated fatty acids in coronary heart disease: a meta-analysis of randomized controlled trials. Am J Med. 2002;112:298-304. [PubMed ID: 11893369]

13. Straus SE. Individualizing treatment decisions. The likelihood of being helped or harmed. Eval Health Prof. 2002;25:210-24. [PubMed ID: 12026754]