Performance Measurement in Community-Acquired Pneumonia: Consequences Intended and Unintended
- 1Northeastern Ohio University College of Medicine, Rootstown
- 2Summa Health System, Akron, Ohio
- 3Hackensack University Medical Center, Hackensack
- 3University of Medicine and Dentistry of New Jersey–New Jersey Medical School, Newark, New Jersey
- Reprints or correspondence: Dr. Thomas M. File Jr., 75 Arch St., Ste. 105, Akron, OH 44304 (filet{at}summa-health.org).
During the past decade, there have been extensive efforts toward improving the treatment of patients in hospitals. The result, in part, has been the publication of guidelines and the development of quality-of-performance measures. Quality measurement has been of particular interest for health care policy makers, consumer groups, media, payers, and regulatory organizations. Because antimicrobial agents are commonly used in hospitals and account for upwards of 30% of hospital pharmacy budgets [1], it is important that these agents be assessed for appropriate use. Furthermore, it has been recognized that ⩽50% of antimicrobial use is inappropriate [2]. Such use not only adds to considerable unnecessary cost-to-patient treatment, but also potentially exposes patients to adverse events (i.e., Clostridium difficile) and is the leading risk factor for antimicrobial resistance.
In this issue of Clinical Infectious Diseases, Schouten et al. [3] present a randomized trial to evaluate a “tailored” approach to improve the antibiotic management of patients who are hospitalized with community-acquired pneumonia (CAP) or acute exacerbation of chronic bronchitis (AECB). The authors used an intervention strategy that targeted specific performance measures that were deemed most in need of improvement. The choice to include CAP is important, because it is a common diagnosis that leads to hospitalization and has been the focus of numerous guidelines and performance measures. Most of the measures for the treatment of CAP were made on the basis of evidence that linked specific processes of care and patient outcomes (although much of the evidence was based on retrospective, observational studies) [4]. Schouten and colleagues also chose to include patients who were admitted for AECB, although they acknowledge that there are limited data relating to performance measures for this diagnosis.
As a result of their prior study [5], Schouten and colleagues recognized the need to choose performance measures that were most important to the local institution and to develop strategies of evaluation that were tailored to the needs of the institution. Importantly, they identified local barriers to implementation and demonstrated that a flexible, tailored approach can lead to better performance of specific measures [3]. Although this approach may be ideal, it is not practical for national comparisons. In addition, as Schouten and colleagues mention, the approach has limited applicability, because it is labor and time intensive.
Schouten et al. [3] recommend a multifaceted approach, similar to one that was recommended previously by Gross and Pujat [6] and one that was recommended by the Infectious Diseases Society of America and the Society for Healthcare Epidemiology of America in the recently published Antimicrobial Stewardship Guidelines [7]. The value of a multifaceted approach has been challenged by those who think that identification of a single intervention that will be successful is more important [8]. Measures to consider for implementation of a multifaceted approach include obtaining local buy-in that is facilitated by local opinion leaders, involving all stakeholders, disseminating the plans, and convening local consensus conferences. Academic detailing is helpful for persons who are reluctant to implement a multifaceted approach. The use of computer-aided decision support and computerized provider order-entry systems, as well as midlevel providers, will aid in facilitating improvement. Finally, reminders, audits, and feedback complete the more successful approaches to implementation [9].
We support the concept of using performance measures to improve treatment, but we would like to emphasize that much more data are needed to assess the relevance of performance measures to clinically important patient outcomes, particularly because hospitals are evolving into pay-for-performance environments. We need to appreciate that randomized, controlled trials are often efficacy trials that exclude many categories of patients. In the “real world,” where all patients are included, the effectiveness of a specific treatment may not be as successful as in efficacy trials. Importantly, there should be objective evidence that adherence to specific recommendations results in better outcomes for the patient. Donabedian [10] refers to process measures that are created on the basis of studies in which outcomes improved as “process-oriented outcome measures.” In the Center for Medicare Medicaid Services/Premier Hospital Quality Incentive Demonstration Project, there are multiple examples that document the positive outcomes that result from implementing certain process measures [11].
Schouten et al. [3] chose to evaluate 5 specific measures from a list of several that were recommended by their local practice guidelines. However, only one of these measures (antimicrobial choice) was shown to be associated with a better outcome for the patient (e.g., decrease in mortality). The others were associated with decreased duration of hospital stay and decreased cost (switches in therapy) or were not convincingly shown to have a direct clinical benefit (e.g., obtaining sputum cultures). Although we feel that Gram staining and culture of sputum samples can be useful for selected patients, we also acknowledge that the utility of sputum studies has been the subject of much controversy [12]. Even Shouten and colleagues acknowledge that the publication of some reports concerning this issue may have adversely influenced the use of these tests during their study. Many of the measures that were evaluated by Shouten and colleagues were based on recommendations that were based on “expert opinion” (i.e., duration of therapy for AECB) rather than on strong evidence from clinical studies. We are also concerned that the authors combined data for CAP with AECB, because the level of evidence for applying performance measures to outcomes for AECB is so limited [13].
For performance measures to be clinically relevant, they need to be evaluated for possible unintended consequences and updated as new findings become available. Physicians in the United States are well-accustomed to the use of performance indicators for treatment of CAP, as measured by the Center for Medicare Medicaid Services and the Joint Commission on Accreditation of Healthcare Organization [14]. These performance measures are reviewed quarterly by a panel of clinical professionals who are knowledgeable about the management of pneumonia (Pneumonia Expert Panel), along with representatives from the Joint Commission on Accreditation of Healthcare Organization and the Center for Medicare Medicaid Services, as part of the National Pneumonia Medicare Quality Improvement Project. As a result of this review process, 2 measures were recently found to be potentially associated with overuse of antimicrobials—precisely an outcome that is opposite to the intended purpose of the performance measures for appropriate antimicrobial use. First, observational studies of the effect of requiring blood cultures for all patients who are admitted to the hospital have demonstrated that false-positive culture results increase the overuse of vancomycin and increase the duration of hospital stay by ∼1 day; in addition, blood cultures may have limited benefit for patients infected with CAP who are admitted to the general ward [15]. Because of this, the performance measure was changed to focus on patients who are admitted to the intensive care unit within 24 h of hospitalization [16]. Second, the timing of administration of the first dose of antimicrobials has been controversial [17–19]. The narrow 4-h window may have actually been associated with the overuse of antimicrobials, because clinicians may use antimicrobials before a firm diagnosis of pneumonia can be established to achieve a high rate of performance. Thus, at the time this commentary was written, the Pneumonia Expert Panel had voted to recommend to change the window time of the administration of the first dose of antimicrobials to 6 h. We mention these examples to illustrate how performance measures need to undergo periodic evaluation and be subject to revision on the basis of the current, best available recommendations.
We also recommend that studies be performed to assess the effect of the combined use of the quality measures in bundles, as a composite score or as an all-or-none score. Although each of the performance measures presently in place have been shown to have an effect because of studies which have measured each one individually, there are limited data to measure the collective effect on clinical outcomes of all of the measures as a composite. One recent study evaluated a composite of 10 performance measures (including 6 for acute myocardial infarction, 3 for heart failure, and 3 for CAP) and found only a modest correlation with hospital risk-adjusted mortality rates [20]. Because CAP is one of the most common conditions that is treated with antimicrobials in hospitals, it is vital to conduct such studies. We recognize that the challenge here is to appropriately apply risk-adjustment methods for any outcome (such as mortality rate), but unless this is evaluated, we cannot adequately know if we are doing the best for our patients.
In conclusion, if performance measures are to be used as the basis for public reporting—as is the trend—we feel that the reporting of actual rates should have a realistic limit, instead of 100% compliance, to avoid “gaming” the system. There is very real pressure for hospitals and physicians to drive their actions on the basis of these measures rather than on what is best for an individual patient. Deviation from the performance measures is acceptable if the reason is well documented in the chart, because specific performance measures cannot include all host and epidemiological contexts. We feel that a better approach is to target an evidence-based benchmark threshold for each indicator, which reflects good care. Reporting hospitals by a threshold target may reduce indiscriminate antibiotic administration by eliminating the pressure to reach 100%. Primum non nocere.
Acknowledgments
Potential conflicts of interest. T.F. and P.G. are both members of the Pneumonia Expert Panel of the National Pneumonia Medicare Quality Improvement Project.
- Received December 27, 2006.
- Accepted December 28, 2006.
- © 2007 Infectious Diseases Society of America
References
This Article
- ExtractFree
- » Full Text (HTML)Free
- Full Text (PDF)Free
Classifications
-
- Articles and Commentaries
Services
Citing Articles
Google Scholar
PubMed
Related Content
Share
- Email this article
-
CiteULike
Connotea
Delicious
Facebook
Google+
Mendeley
Twitter
Published on behalf of
IDSA Members: For your free access to this journal, log in via the IDSA members area.
The Journal
Impact Factor: 8.186
5-Yr impact factor: 7.898
For The Media
For Authors
- Services for authors
- Self-archiving Policy
- IDSA Conflict of Interest policy
- Instructions for Authors
- ICMJE Form
- Submit Now!
Open access options for authors visit Oxford Open
This journal enables compliance with the NIH Public Access Policy
Corporate Services
Alerting Services
Related IDSA Publications
