Incorporating economic analysis into clinical practice guidelines: a guide for usersPDF
ACP J Club. 2002 Nov-Dec;137:A11. doi:10.7326/ACPJC-2002-137-3-A11
A large health plan has asked you to help them develop a clinical practice guideline for colon cancer screening. The plan currently covers annual fecal occult blood testing (FOBT) and flexible sigmoidoscopy every 5 years as screening methods. Member rates for both types of screening are very low (hence the impetus for the guideline). At the behest of several local gastroenterologists, the plan is also considering whether to cover colonoscopy every 10 years for average-risk persons ≥ 50 years of age. Part of the process involves reviewing the cost-effectiveness literature, because the chief executive officer of the health plan is skeptical that colonoscopy represents a wise use of their ever-tightening budget. You are familiar with U.S. Preventive Services Task Force Guidelines for colon cancer screening. After reviewing several recent cost-effectiveness studies, however, you find them daunting in terms of methodologic complexity, terminology, and representation of outputs (a confusing array of large numerical tables and graphs that strongly resemble something you studied in first-year college economics). Is it possible to translate these studies into something you understand, believe reflects sound clinical practice, and believe would be useful to the health plan for their decision making?
The situation just described, an apparent disconnect between cost-effectiveness studies and clinical practice guidelines and the confusion it causes for readers, is not unusual. Economic analyses are rarely included in guidelines, mostly because each discipline has different and often conflicting views of what constitutes “best practice” (1, 2). This exclusion is unfortunate, because both guidelines and cost-effectiveness studies offer important information to help us practice more effective and efficient health care. Decision makers today have a need to synthesize and interpret these studies rapidly and efficiently. This editorial offers suggestions to help clinicians understand cost-effectiveness studies and to use them in developing guidelines. Its approach is similar to the methods for defining and answering clinical queries used by those who practice evidence-based medicine: defining the clinical question, searching for evidence, and evaluating the quality of evidence.
What is the clinical problem that is the subject of the guideline?
In this case we focus on screening options for adults ≥ 50 years of age who do not have symptoms of and who are at average risk for colorectal cancer.
What interventions are being considered? What is the default intervention, if any?
Guidelines are designed to reduce variation in care, with the goal of using treatments that have been shown to improve desirable outcomes or minimize undesirable ones. Likewise, cost-effectiveness studies most often focus on comparing new with existing care. In the case of colon cancer screening, the health plan already covers flexible sigmoidoscopy and FOBT, therapies with good evidence of efficacy. It is thus reasonable to consider the cost-effectiveness of colonoscopy compared with the following alternatives: no screening, flexible sigmoidoscopy alone, FOBT alone, and sigmoidoscopy plus FOBT.
Search the literature for cost-effectiveness studies
Searching for high-quality economic articles can be laborious, and the outcomes are not always satisfactory (3), primarily because of overuse of the term cost-effectiveness in the literature. Searching for this term will retrieve many articles, but most are not formal economic analyses. To address this problem, I suggest starting by using the medical subject heading (MeSH) for cost-effectiveness analysis, that is, “cost-benefit analysis” (when searching in MEDLINE) with other terms that are specific to the clinical situation and technologies being compared (as outlined in 1 and 2 above). For colon cancer screening in PubMed (www.ncbi.nlm.nih.gov/entrez/query.fcgi), start with “cost-benefit analysis” as well as the content terms “screening, colonoscopy, sigmoidoscopy, and fecal occult blood.” Although this approach is not perfect, it improves the specificity of the search and substantially reduces the volume of articles compared with using the last 4 terms alone. In this case, we retrieve 24 articles. Reviewing the abstracts, it appears that several are formal cost-effectiveness analyses of all 3 screening methods. For this exercise, we select 3 recent analyses (4-6). These will be a good start for the next phase of the evaluation.
What is the bottom line of each study for the interventions being considered?
The bottom line of most economic analyses is an incremental cost-effectiveness ratio—that is, the ratio of the difference in costs over the difference in outcomes for the interventions being compared. While starting with the conclusion does not help us determine the quality of the methods, it allows us to address the question: “If this were true, could it influence how I practice (or write or implement the guidelines)?” The economic analysis could be influential, for example, if it states that a little-used therapy is highly cost-effective, or conversely, that a widely advocated therapy has poor cost-effectiveness. If your answer to the question is “no,” then stop reading. If the answer is “yes,” we need to go on and review the methods to answer another question: “Is it likely to be true?”
In this case, the 3 studies (4-6) consider the relative values of colonoscopy, flexible sigmoidoscopy, and FOBT as screening tools for persons ≥ 50 years of age at average risk for colorectal cancer. Comparisons worth considering include the value of each compared with no screening and the incremental value of colonoscopy compared with either flexible sigmoidoscopy or FOBT. Because colonoscopy is more costly than the other approaches (at least initially) it is important to ask what one gets for this additional expenditure. The results are shown in Table 1.
Table 1. Cost-effectiveness of colorectal cancer screening as reported in selected articles*
|Sonnenberg (4)||Frazier (5)||Vijan (6)|
|Annual FOBT vs no screening||$9705||$12 667||$7756|
|FS every 5 y vs no screening||$36 509||$12 571||$14 668|
|Colonoscopy vs no screening||$10 983||$20 418||$7019|
|Colonoscopy vs annual FOBT||$11 382||$76 552||$6243|
|Colonoscopy vs FS every 5 y||Dominates‡||$52 903||Dominates‡|
|Colonoscopy vs FS every 5 y plus annual FOBT||Not available||$561§||$332 630§|
*FOBT = fecal occult blood test; FS = flexible sigmoidoscopy.
†Baseline compliance: Sonnenberg 100%, Frazier 60%, Vijan 75%.
‡Dominates means that the second procedure listed was more costly and less effective than the first procedure.
§FOBT plus FS is more expensive and more effective than colonoscopy.
Eyeballing these results tells us 2 things. First, all 3 studies conclude that compared with no screening, all strategies are reasonably cost-effective when using a (rather arbitrary, but widely cited) threshold of $50 000/life-year gained. Second, the articles reach very different conclusions regarding the incremental value of colonoscopy compared with FOBT and flexible sigmoidoscopy. Sonnenberg (4) and Vijan (6) and their colleagues state that compared with FOBT and flexible sigmoidoscopy alone, colonoscopy provides good value; Frazier and colleagues (5) find that colonoscopy is only marginally cost-effective.
What factors “drive” the outcome?
Cost-effectiveness analyses are by nature synthetic, integrating data from multiple sources to derive the result. Reviewing each source (there can be dozens) for quality and accuracy is impractical for most decision makers. To help with this issue, ask: “What factors drive the outcome, that is, would have the greatest influence on the bottom line of the study if they were changed from their baseline value?” This issue, formally termed sensitivity analysis, is a logical next step for evaluating the quality of a cost-effectiveness study. As a rule of thumb, 2 to 4 factors usually have the greatest effect on the outcome, especially the cost and efficacy of the intervention. If the cost-effectiveness ratio changes from favorable to unfavorable (or vice versa) at plausible values for influential factors, this reduces confidence in the conclusions.a
Most cost-effectiveness studies have a sensitivity analysis section that addresses this issue. Unfortunately, it is difficult to identify the most influential factors for the 3 screening methods in the sensitivity analysis sections of the 3 studies. The 2 mentioned consistently are procedure cost and compliance. These are useful because they are seldom specifically evaluated in clinical trials and are likely to vary from setting to setting. Table 2 summarizes the sensitivity analysis for these variables.
Table 2. Sensitivity analysis of screening vs no screening for cost and compliance*
|Variable||Cost per life year gained vs no screening (1998 dollars)|
|Cost||$9705†||$12 667‡||$14 668§||$36 509†||$7019||$20 418‡|
|Compliance||$5530§||$32 500†||$5800‡||$22 161§||$10 983†||$26 000†|
*Results are the most and least cost-effective outcomes among all 3 studies at the
stated factor extremes (in parentheses). Direct comparability is limited because of
variations in other factors and modeling techniques. Estimates at factor extremes
were not available for all studies.
†= reference 4.
‡= reference 5.
§= reference 6.
It seems that varying procedure cost or compliance across a probable range is unlikely to have an extremely adverse effect on the cost-effectiveness of the 3 strategies.
How valid are the most influential variables at their default values?
If the bottom line of the study is highly dependent on specific factors, then these should be scrutinized closely. It is particularly important that influential data are based on the highest-quality trials available, preferably randomized controlled trials (RCTs). Of note, the ability of colonoscopy to prevent colorectal cancer or mortality has not been measured in a randomized screening trial. Two articles base their results on 1 case-control study (7), suggesting that screening colonoscopy reduces the risk for dying of colon cancer. None of these articles would pass the screening criteria for cost-effectiveness studies used by the ACP Journal Club (See Purpose and Procedure), which requires ≥ 1 of the studies included in the analysis of effectiveness to be an RCT. One could easily justify ignoring these articles at this point if the purpose was simply to keep up to date. However, our purpose is to address a policy problem (about screening), and for this the lack of strong evidence should be considered in reaching a decision about colonoscopy. The effectiveness of FOBT, on the other hand, is supported by 3 large RCTs showing benefit in reducing colorectal cancer–related mortality (8-10). Flexible sigmoidoscopy is intermediate in effectiveness, with 1 small RCT and 2 case–control studies showing benefit (11-13).
Interpretation: How should the economic studies influence the guidelines?
If we are reasonably confident that the economic studies are valid and cover the major clinical choices facing the organization, next we need to consider what role (if any) they should play in shaping the guideline. These include issues that extend beyond the scope of the studies, such as implementation costs and the availability of clinicians trained in colonoscopy. Other issues, such as the first-year budget effect of adopting a particular policy, may or may not be available from the economic studies (in this case they are not). All should be considered (and weighted) in conjunction with the economic and clinical data.
The economic studies suggest that given the assumptions about efficacy, compliance, and cost, all methods—including colonoscopy—are relatively cost-effective for colorectal cancer screening compared with no screening. From an evidence-based medicine perspective, the weak supporting clinical evidence of the efficacy of colonoscopy is problematic. The clinical and economic outcomes of FOBT are more certain because efficacy rates are drawn from RCTs, and sensitivity analyses show that the outcomes are probably maintained across a reasonable range of costs and compliance. The same applies to a lesser extent for flexible sigmoidoscopy.
How much “worse” would colonoscopy be from a cost-effectiveness standpoint if the efficacy was not as assumed? Unfortunately, this is not addressed in the economic studies. The studies together suggest that incremental gain of switching from FOBT or flexible sigmoidoscopy to colonoscopy is highly uncertain, ranging from “dominant” to cost-ineffective. As with clinical studies, such a degree of uncertainty should make one pause before adopting a new standard of care for an organization. Thus, the review of the cost-effectiveness and clinical evidence do not support adopting colonoscopy as a covered screening procedure at this time. At this point it is reasonable to note that “absence of proof is not proof of absence.” It could well be that colonoscopy is more effective than FOBT or flexible sigmoidoscopy, even taking into account increased morbidity. However, the onus of proof rests with those who believe this is the case.
Meanwhile, the evidence and cost-effectiveness analyses support screening with FOBT (or flexible sigmoidoscopy when affordable) and justify efforts to increase utilization of this program. As it happens, success with such a program would also increase the use of colonoscopy to evaluate “positive” findings by FOBT.
Economic analyses have much to offer in the creation of clinical practice guidelines. They often “put the pieces together” between therapy and outcome and give a sense of health value for money, this latter issue being an unavoidable fact of life in today’s environment. Economic analyses should not dictate clinical practice guideline policies, but they can inform them by identifying the evidence that is available and helping to identify cost-inefficient strategies and making explicit the tradeoffs between benefit and expenditure for others.
aOne-way sensitivity analyses generally understate the uncertainty in the cost-effectiveness ratio. One should check to see if the article presents confidence intervals around the cost-effectiveness ratio. New techniques that can estimate confidence intervals are being used more frequently today.
Scott D. Ramsey, MD, PhD
Fred Hutchinson Cancer Research CenterUniversity of Washington
Seattle, Washington, USA
1. Wallace JF, Weingarten SR, Chiou CF, et al. The limited incorporation of economic analyses in clinical practice guidelines. J Gen Intern Med. 2002;17:210-20. [PubMed ID: 11929508]
2. Ramsey SD. Economic analyses and clinical practice guidelines: why not a match made in heaven? J Gen Intern Med. 2002;17:235-7. [PubMed ID: 11929511]
3. Sassi F, Archard L, McDaid D. Searching literature databases for health care economic evaluations: how systematic can we afford to be? Med Care. 2002;40:387-94. [PubMed ID: 11961473]
4. Sonnenberg A, Delco F, Inadomi JM. Cost-effectiveness of colonoscopy in screening for colorectal cancer. Ann Intern Med. 2000;133:573-84. [PubMed ID: 11033584]
5. Frazier AL, Colditz GA, Fuchs CS, Kuntz KM. Cost-effectiveness of screening for colorectal cancer in the general population. JAMA. 2000;284:1954-61. [PubMed ID: 11035892]
6. Vijan S, Hwang EW, Hofer TP, Hayward RA. Which colon cancer screening test? A comparison of costs, effectiveness, and compliance. Am J Med. 2001;111:593-601. [PubMed ID: 11755501]
7. Muller AD, Sonnenberg A. Prevention of colorectal cancer by flexible endoscopy and polypectomy. A case-control study of 32, 702 veterans. Ann Intern Med. 1995;123:904-10. [PubMed ID: 7486484]
8. Mandel JS, Bond JH, Church TR, et al. Reducing mortality from colorectal cancer by screening for fecal occult blood. Minnesota Colon Cancer Control Study. N Engl J Med. 1993;328:1365-71. [PubMed ID: 8474513]
9. Hardcastle JD, Chamberlain JO, Robinson MH, et al. Randomised controlled trial of faecal-occult-blood screening for colorectal cancer. Lancet. 1996;348:1472-7. [PubMed ID: 8942775]
10. Kronborg O, Fenger C, Olsen J, Jorgensen OD, Sondergaard O. Randomised study of screening for colorectal cancer with faecal-occult-blood test. Lancet. 1996;348:1467-71. [PubMed ID: 8942774]
11. Thiis-Evensen E, Hoff GS, Sauar J, et al. Population-based surveillance by colonoscopy: effect on the incidence of colorectal cancer. Telemark Polyp Study I. Scand J Gastroenterol. 1999;34:414-20. [PubMed ID: 10365903]
12. Selby JV, Friedman GD, Quesenberry CP Jr, Weiss NS. A case-control study of screening sigmoidoscopy and mortality from colorectal cancer. N Engl J Med. 1992;326:653-7. [PubMed ID: 1736103]
13. Newcomb PA, Norfleet RG, Storer BE, Surawicz TS, Marcus PM. Screening sigmoidoscopy and colorectal cancer mortality. J Natl Cancer Inst. 1992;84:1572-5. [PubMed ID: 1404450]