Skip Navigation

A Systematic Review of the Methods Used to Assess the Association between Appropriate Antibiotic Therapy and Mortality in Bacteremic Patients

  1. Jessina C. McGregor1,
  2. Shayna E. Rich2,
  3. Anthony D. Harris2,4,
  4. Eli N. Perencevich2,4,
  5. Regina Osih3,
  6. Thomas P. Lodise Jr.5,
  7. Ram R. Miller2, and
  8. Jon P. Furuno2
  1. 1Oregon State University College of Pharmacy, Portland
  2. 2Department of Epidemiology and Preventive Medicine, Baltimore
  3. 3Department of Medicine, University of Maryland School of Medicine, Baltimore
  4. 4Veterans Affairs Maryland Health Care System, Baltimore
  5. 5Albany College of Pharmacy, Albany, New York
  1. Reprints or correspondence: Dr. Jessina C. McGregor, Oregon State University College of Pharmacy, 3303 SW Bond Ave., Mail Code CH12C, Portland, OR 97239 (mcgregoj{at}ohsu.edu).

Abstract

Studies of the association between inappropriate antibiotic therapy and mortality among bacteremic patients have generated conflicting findings. We systematically reviewed these studies to identify methodological heterogeneity that may explain the lack of agreement. We identified 51 articles that met the inclusion criteria, and we extracted the following data: study design, definition and measurement of variables, and statistical methods. Only 8 studies (16%) defined inappropriate antibiotic therapy as that which was inactive in vitro against the isolated organism(s) and not consistent with current clinical practice recommendations and distinguished between empiric and definitive treatment. Thirty-four studies (67%) measured the severity of illness, but only 6 (12%) specified the time at which it was measured. The methodological recommendations suggested in this article are intended to improve the validity and generalizability of future research. In brief, future studies should define “inappropriate” therapy on the basis of in vitro susceptibility data, should separately evaluate empiric and definitive therapy, and should control for the baseline severity of illness.

Bloodstream infections are associated with considerable morbidity, mortality, and health care costs [1, 2]. Furthermore, increasing antibiotic resistance has resulted in fewer treatment options and has made empiric therapy selection more difficult [3]. Clinicians strive to empirically select antibiotic therapy that is appropriate (i.e., antibiotics with in vitro bactericidal or bacteriostatic activity against the infecting agent[s]). However, existing research has not provided consistent evidence of an association between appropriate antibiotic therapy and mortality [4]. Although aggressive empiric therapy with broad-spectrum antibiotics may appear to be an attractive treatment strategy, it can result in increased costs, adverse events, and increased selective pressure for antibiotic resistance [5,67].

To make accurate scientific inferences from this body of research, confounding and selection or information biases must be excluded as alternative explanations for associations between the administration of inappropriate therapy and patient mortality [8]. This article presents our systematic review of studies that have assessed the association between inappropriate antibiotic therapy for the treatment of bacteremia and subsequent mortality. Heterogeneity in the study designs or analyses is summarized, and we propose recommendations for future studies.

Methods

The PubMed database was used to query the National Library of Medicine for English-language articles published before 1 January 2006 with the following medical subject heading terms: “anti-bacterial agents” AND “septicemia” AND (“hospital mortality” OR “outcome assessment (healthcare)” OR “survival rate”). Each publication was required to be a study limited to adults, to have included appropriate antibiotic therapy for bacteremia as an independent variable, and to have included mortality as a dependent variable. Two authors (J.C.M. and S.E.R.) independently reviewed each publication to identify those that met the inclusion/exclusion criteria, and a third author (J.P.F.) resolved any ties. Additional sources that had been cross-referenced from our PubMed search results were included if they met the criteria above.

Methodological flaws may lead to biased and inconsistent study results. Thus, the suitability of the study design to measure an association between inappropriate antibiotic therapy and mortality was evaluated for each study on the basis the following 4 domains: (1) definition and measurement of the independent variable (i.e., appropriate antibiotic therapy), (2) definition and measurement of the dependent variable (i.e., mortality), (3) confounding and inclusion/exclusion criteria, and (4) statistical power.

Definition and measurement of appropriate antibiotic therapy. Appropriate antibiotic therapy was defined as a primary exposure of interest if the study clearly identified it as such in the study objectives; studies were permitted to have more than 1 primary exposure of interest. Even if it was not specifically stated, we considered appropriate antibiotic therapy to be a primary variable of interest if it was forced into the final statistical analysis. We reviewed how appropriate antibiotic therapy was measured, and we assessed whether the definition accounted for the antibiotic resistance profile of the organism(s) isolated from the index blood culture. We determined whether the therapy dose and route were evaluated for consistency against current clinical practice recommendations, and we assessed the time point at which the antibiotic therapy was evaluated for appropriateness.

The term “empiric therapy” refers to antibiotics that are administered during the period prior to the receipt of blood culture and antibiotic susceptibility test results, whereas the term “definitive therapy” refers to the antibiotic therapy given subsequent to receipt of these results. In practice, physicians often receive partial culture results (e.g., Gram stain results) before they receive the final report. For this review, we defined empiric therapy as all nondefinitive therapy (i.e., all therapy given prior to the receipt of final culture and susceptibility results). Furthermore, we determined whether the authors differentiated between empiric and definitive therapy and whether they knew precisely when organism and susceptibility results had become known. For articles that contained no mention of the aforementioned classifications, we considered therapy to be empiric if it had been provided up to 72 h after culture specimen collection.

Definition and measurement of patient mortality. For each study, the definition of the dependent variable (mortality) was evaluated to identify potential discrepancies.

Confounding variables and inclusion/exclusion criteria. We evaluated which covariates were measured and considered as potential confounding factors for the association between antibiotic therapy and mortality. The primary potential confounder of interest for this review was severity of illness. We ascertained the methods used to measure severity of illness, including the time at which it was assessed, as well as whether the authors statistically adjusted for severity of illness. We also determined whether comorbidity, septic shock, and the source of bacteremia were assessed.

For the inclusion and exclusion criteria, we assessed whether individuals who experienced repeated bacteremic episodes during the study period entered into the study multiple times. We also assessed whether polymicrobial bacteremic episodes were included or excluded from the study.

Statistical power. We calculated simple bivariable power calculations for each of the studies on the basis of a test of the difference in the observed proportions of death between patients who received appropriate and inappropriate antibiotic therapy. The α value was set to 0.05. In studies in which multiple analyses were performed, we report results for the highest-powered analysis.

Results

We identified 471 articles in the National Library of Medicine database that met the search criteria. After reviewing these publications and those that were cross-referenced, we identified 51 publications that met all study criteria and that were therefore included in our systematic review [9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,5859]. Eighteen of the studies were conducted in Europe, 12 were conducted in the United States, 10 were conducted in Asia, 4 were conducted in the Middle East, 2 were conducted in Australia, 2 were conducted in South America, and 1 was conducted in Africa; 2 of the studies were multinational. Ten of the studies were conducted at multiple institutions [12, 17, 25, 26, 40, 44, 46, 47, 56, 58]. Forty-four studies were hospital-wide [11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,2627, 29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,4950, 52, 53, 55, 56, 58], 4 were conducted only among patients in the intensive care unit (ICU) [9, 10, 28, 59], 2 were conducted among patients with cancer [51, 54], and 1 was conducted among geriatric patients [57].

Definition and measurement of appropriate antibiotic therapy. Although all 51 publications included in this review assessed appropriate antibiotic therapy as an independent variable, it was a primary exposure of interest in only 25 (49%) of the studies (table 1). The definition of appropriate therapy was provided in 46 studies (90%) (table 2). Thirty-four studies (74%) specifically defined therapy as being empiric or definitive therapy. However, only 11 studies (24%) reported data on the receipt of culture and antibiotic susceptibility results, whereas the 23 remaining studies (50%) defined empiric therapy as antibiotic treatment that had been administered after the index blood culture result was obtained but not exceeding some maximum time point afterwards. Thus, in the latter group, the therapy assessed may have represented empiric therapy and a short period of definitive therapy. No study reported the mean or median time from culture specimen collection to the availability of susceptibility test results. In 5 studies, we could not determine whether the time of the final susceptibility test report was available to the investigators. Only 2 publications assessed the time to appropriate therapy as a risk factor for mortality [10, 41].

Table 1
Table 1

Summary of the reviewed articles: studies of the association between appropriate antibiotic therapy and mortality in patients with bacteremia.

Table 2
Table 2

Differences in the definition of appropriate antibiotic therapy.

The manner in which antibiotic therapy was deemed to be appropriate or inappropriate also varied. Forty-one studies (87%) defined appropriate therapy as therapy that included an antibiotic that possessed in vitro activity for the isolated organism. Only 13 studies (28%) assessed both the adequacy of dosing and the route of administration as part of their definition of appropriate therapy, whereas an additional 7 studies (15%) assessed either dosing or the route of administration. Overall, only 8 studies (16%) included all 3 of the following components in their definition: in vitro activity against the isolated organism(s), consistency with current clinical practice recommendations (i.e., appropriateness of route and/or dose), and classification of treatment as either empiric or definitive.

Definition and measurement of patient mortality. The mortality construct measured varied between studies (table 1). Bacteremia-attributable mortality was the outcome of interest in 18 studies (35%), 22 studies (43%) measured all-cause mortality within a defined range of time (e.g., 30-day mortality), and 16 studies (31%) measured in-hospital mortality. Six studies did not define mortality. Note that some studies measured mortality in multiple ways.

Confounding variables and inclusion/exclusion criteria. Severity of illness was measured in 34 studies (67%), and 23 studies (68%) adjusted for it in the final analysis (table 3). Fifteen studies used the McCabe-Jackson score, 14 used the APACHE II score, and 3 used either the Simplified Acute Physiology Score (SAPS) or SAPS II [60,61,6263].

Table 3
Table 3

Differences in the measurement of patient severity of illness.

Only 7 studies (21%) that measured severity of illness detailed the time at which the variable was assessed during the patient's hospitalization. None of the studies clearly stated that severity of illness was measured before the index blood culture specimen collection or prior to the onset of bacteremia. Severity of illness was measured in 2 studies at the time that the blood culture specimen was obtained and by another 2 studies at the time of the positive blood culture result; an additional 2 studies used the APACHE II score at the time of ICU admission. The final study reported measuring the APACHE II score at the onset of bacteremia but did not define this time point in relation to the collection of the blood culture sample or receipt of culture results. None of the 15 studies that used the McCabe-Jackson score specified the time at which the severity of illness was assessed.

Patient comorbidity was assessed in 44 studies (86%), and only 1 study used an aggregate measure of comorbidity (i.e., the Charlson comorbidity index) [64]. Thirty-five studies (69%) assessed the primary source of bacteremia. Seventeen studies adjusted for the presence of septic shock, an intermediate variable, in their final analyses. Only 1 study used propensity score methodology to adjust for confounding [26].

Twenty-eight studies (55%) limited their study population to patients with bacteremia due to a single bacterial species. Eighteen studies (35%) included patients with polymicrobial bacteremia. Thirteen studies (25%) included patients more than once if they had multiple episodes of bacteremia during the study period.

Statistical power. Statistical power calculations were calculated for 41 studies (table 1). For the remaining 10 studies, power calculations could not be performed either because of a lack of necessary data or because of a matched study design (in studies in which appropriate antibiotic therapy was not the primary exposure of interest). Twenty-one studies (41%) had a statistical power of ⩾.70.

Discussion

We systematically reviewed original research on the association between appropriate antibiotic therapy and mortality among bacteremic patients, and we identified significant heterogeneity in the study methodologies. Appropriate therapy and mortality were not consistently measured, and key confounding variables—primarily, the severity of illness—were not always measured or adjusted for in the final analysis. Differences in variable definitions and in the selection of confounding factors to control for may partially explain inconsistencies in the observed associations between appropriate therapy and mortality. Therefore, future research in this area should measure all study variables in a manner that best captures the construct that they represent and should statistically adjust for key confounding variables in final analyses.

However, differing methodologies that generate conflicting conclusions do not necessarily invalidate the individual studies. Two studies with conflicting results may have each observed “true” results, if underlying differences in the distributions of other causal variables were present [65]. Nevertheless, when inconsistent or inappropriate methodologies are used, it becomes exceedingly difficult to assess whether true differences exist or whether unrecognized sources of confounding or biases may be affecting the observations. Recommendations for future studies include the major methodological considerations described in detail below and summarized in table 4.

Table 4
Table 4

Key recommendations for future studies of the association between appropriate antibiotic therapy and mortality among bacteremic patients.

The appropriateness of antibiotic therapy should optimally be assessed on a case-by-case basis. Therapy should be considered to be appropriate if the regimen exhibits in vitro activity against the isolated pathogen(s). Furthermore, we recommend that empiric and definitive antibiotic therapy be separately defined, because interventions aimed at increasing the proportion of patients who receive appropriate empiric and definitive therapy would be inherently different. In populations in which inappropriate empiric therapy is associated with increased mortality, clinical guidelines, hospital antibiograms, and consultations with infectious diseases specialists may improve the likelihood that empiric therapy is appropriate. In contrast, interventions aimed at increasing appropriate definitive therapy would involve facilitation of prompt access to the final culture and susceptibility test reports, as well as full comprehension of the implications for treatment.

In addition, because the delay to appropriate therapy may be associated with increased mortality, future studies are needed to evaluate time to appropriate antibiotic therapy as a risk factor for poor patient outcomes. Such studies are emerging in the biomedical literature and may provide evidence in support of the use of rapid microbiological testing procedures that compress the window during which empiric therapy is the only available option [66, 67].

In any epidemiologic investigation, it is critical to assess the presence of potential confounding effects on the primary association of interest; failure to statistically adjust for confounding variables can lead to biased estimates of effect. Identification of confounders requires knowledge of the causal associations being studied. Baseline comparisons of the data are extremely useful for identifying potential confounders among the measured covariates. Although our review cannot cover this topic with the depth warranted by its importance, there are many resources available to provide guidance in this area when planning future research [68,69,70,71,7273]. In our evaluation of the potential confounding variables included in the studies we reviewed, we gave specific attention to severity of illness. We observed that the time at which severity of illness was measured varied greatly; in some cases, this may have resulted in improper adjustment for the confounding effect of severity of illness. In studies that specified precisely when severity of illness was measured, the assessment occurred at any time from hospital admission to ICU admission and at any time from the blood culture specimen collection to when the blood culture result was found to be positive. For assessments made at the time of hospital admission or ICU admission, patients may have experienced differing durations between assessment and the onset of bacteremia. For some patients, this point may occur after the onset of bacteremia, whereas for others, assessment may occur just before the onset of bacteremia or even weeks beforehand. Thus, variation in how severity of illness is measured inevitably results in varying constructs for severity of illness among studies.

We propose that the optimal time to measure severity of illness is immediately before the true onset of bacteremia. Unfortunately, this may be impossible to pinpoint and may not correlate with the time that the blood specimen is obtained for culture. Severity of illness should not be measured after the onset of bacteremia, because this would result in inappropriately controlling for a variable in the causal pathway rather than controlling for the patient's baseline state and risk of mortality [69, 74, 75]. This could result in an underestimation of the measure of effect. Thus, investigators must select a measurement point as close to the theoretical ideal as is feasible. We recommend that severity of illness be measured 48 h before the index blood sample is obtained for culture. This time point—although it is not a perfect match to the theoretical point in time just before the onset of bacteremia onset—represents a feasible estimate that should be easily reproducible. However, use of this time point also has important limitations. First, measurement at 48 h before culture sample collection may still follow the onset of bacteremia and would thus be in the causal pathway. Furthermore, it may be difficult to measure severity of illness at this time point among patients who are admitted to the hospital with bacteremia, because investigators may not have data on the patient's status before admission. Therefore, we also recommend reporting the proportion of patients who were admitted to the hospital with bacteremia and separately analyzing these patients whenever feasible.

To compare and synthesize information obtained from multiple studies, it is important that investigators provide a complete description of the study methods. Studies should describe whether patients with multiple episodes of bacteremia during the study period were included more than once, because a patient whose bacteremic episode is potentially recurrent may be more likely to receive appropriate empiric therapy. Future studies should also indicate whether patients with polymicrobial bacteremia were included and, if so, whether the definition of appropriate therapy pertains to therapy that was appropriate for all isolated organisms.

Although the dependent variable, mortality, was subject to less variation in definition and measurement, some important differences were observed. Not all studies clearly defined how mortality was assessed. Forty-three percent of the studies assessed mortality within a defined time window (e.g., 30-day mortality). Not all of these studies clearly specified whether patients were observed after hospital discharge, if discharge occurred within the window of time that mortality was to be assessed. Furthermore, all of these studies used logistic regression as the method of analysis, and this does not allow for patients to be censored when discharged. Rather, the assumption is made that discharged patients are “survivors,” which could result in a biased estimate of association. The same bias is possible when in-hospital mortality is the outcome used. Thus, greater specificity is needed for the definition of mortality as an outcome. The definition should carefully consider the biologically plausible window of effect. Furthermore, survival analysis techniques (e.g., Cox proportional hazards modeling) should be used whenever study participants must be censored because of loss to follow-up.

Whenever sample size permits, multivariable analyses should be conducted to adjust for confounding. Severity of illness, patient comorbidity, and the presence of polymicrobial bacteremia should be evaluated as potential confounders of the association between appropriate antibiotic therapy and mortality. Although some studies adjusted for the effect of septic shock in their final analyses, researchers should generally not adjust for this condition, because it occurs as a result of bacteremia and lies in the causal pathway to mortality [75,76,7778].

This systematic review identified several major areas of divergent and suboptimal methodologies in research studies that evaluated the effect of appropriate antibiotic therapy on patient mortality among bacteremic patients. Such variation in methodologies may account for the conflicting results in the literature. Without adequately designed research studies in this area, there is little evidence for or against recommendations regarding aggressive empiric therapy with broad-spectrum antibiotics. The recommendations provided here are motivated by a desire to provide guidance for future study design and analysis. By implementing the recommendations discussed herein, the scientific inferences made from future studies will have greater validity, and the commonality in methodologies will make it far simpler to accurately synthesize the body of research conducted in this area.

Acknowledgments

We wish to thank Leslie Norris for her assistance in the literature search and in obtaining manuscripts and Allison Krug for editing this manuscript.

Financial support. National Institutes of Health (1R01A160859-01A1 and 1K23AI001752-01A1, to A.D.H.; and 1K12RR023250-01, to J.P.F.), US Department of Veterans Affairs (RCD-02-026-2 and IIR-05-123-1, to E.N.P.), and Claude D. Pepper Older Americans Independence Center Junior Faculty Award (P60 AG12583, to R.R.M.).

Potential conflicts of interest. All authors: no conflicts.

  • Received November 29, 2006.
  • Accepted April 4, 2007.

References

| Table of Contents