Development of Quality Indicators for the Antibiotic Treatment of Complicated Urinary Tract Infections: A First Step to Measure and Improve Care
- 1Department of Internal Medicine, Division of Infectious Diseases, Tropical Medicine & AIDS, Center for Infection and Immunity Amsterdam, Academic Medical Center, Amsterdam, Nijmegen, The Netherlands
- 2Centre for Quality of Care Research, Radboud University Nijmegen Medical Centre, Nijmegen, The Netherlands
- Reprints or correspondence: Suzanne Geerlings, Academic Medical Center, Rm. F4-217, Meibergdreef 9, 1105 AZ Amsterdam, The Netherlands (s.e.geerlings{at}amc.nl).
Abstract
Background. Appropriateness of antibiotic treatment of urinary tract infection (UTI) is important. The aim of this study was to develop a set of valid, reliable, and applicable indicators to assess the quality of antibiotic use in the treatment of hospitalized patients with complicated UTI.
Methods. A multidisciplinary panel of 13 experts reviewed and prioritized recommendations extracted from a recently developed evidence-based national guideline for the treatment of complicated UTI. The content validity was assessed in 2 consecutive rounds with an in-between discussion meeting. Next, we tested the feasibility, interobserver reliability, opportunity for improvement, and case-mix stability of the potential indicators for a data set of 341 inpatients and outpatients with complicated UTIs who were treated at the urology or internal medicine departments at 4 hospitals.
Results. The panel selected and prioritized 13 indicators. Four and 9 indicators were performed satisfactorily in the urology and internal medicine departments, as follows: performance of urine culture, prescription of treatment in accordance with guidelines, tailoring of treatment on the basis of culture results, and a switch to oral treatment when possible in the urology and internal medicine departments; and selective use of fluoroquinolones, administration of treatment for at least 10 days, prescription of treatment for men in accordance with guidelines, replacement of catheters in patients with UTI, and adaptation of the dosage on the basis of renal function in the internal medicine department.
Conclusion. A systemic evidence- and consensus-based approach was used to develop a set of valid quality indicators. Tests of the applicability of these indicators in practice in different settings is essential before they are used in quality-improvement strategies.
Urinary tract infections (UTIs) are among the most common infections in patients seen in the hospital [1]. Consequently, a significant portion of overall antimicrobial use in hospitals is for treatment of UTIs. Resistance rates for Escherichia coli and other uropathogens to trimethoprim (with and without sulfa-methoxazole) and other antibiotics have been increasing in recent years in many countries of the world [2, 3]. Therefore, guidelines for the treatment of UTIs have to be updated regularly and subsequently implemented [4–6]. Previous studies have demonstrated that adherence to guidelines on antibiotic use improves outcome [7, 8].
The quality of care can be measured by indicators, which can be defined as retrospectively measurable elements of practice performance for which there is evidence or consensus that they can be used to assess the quality of care—and, thus, leads to changes in the quality of care provided [9]. To our knowledge, no quality indicators have been developed for antibiotic use in hospitalized patients with complicated UTI.
The aim of our study was to develop a valid set of applicable indicators to measure the quality of care for antibiotic treatment of complicated UTI. We used our recently written, evidence-based national guidelines as a starting point [6]. The indicator set was tested in 4 hospitals to provide an applicability test for daily practice.
Methods
Phase 1: Development of Indicators
Key recommendations were preselected from a recently developed, evidence-based national guideline on the antimicrobial treatment of complicated UTI in patients seen in the hospital (inpatients and outpatients) (http://www.swab.nl) [6]. On the basis of the available literature, the level of evidence was graded for each recommendation to determine its scientific soundness or the likelihood that improvement for the quality indicator would produce consistent improvements in the quality of care (table 1) [11]. Key recommendations supported by a level of evidence with a grade of A were selected as content validity indicators. To assess the validity of recommendations supported by grade B-D supporting evidence, we used the group judgement of expert opinions with a 3-step modified Delphi approach [12]. Therefore, we assembled a multidisciplinary assessment team with representatives of all involved clinical experts, as described elsewhere [11, 13]. The expert panel consisted of 13 panelists: 2 medical microbiologists, 4 infectious diseases specialists, 2 hospital pharmacists, 2 urologists, 2 nephrologists, and 1 gynecologist. During 3 rating rounds, the expert panel rated the preselected recommendations by judging their relevance with regard to patient health, development of bacterial resistance, and health care cost.
During the first rating round, the preselected recommendations were formatted as a questionnaire, and the panelists were asked to rate each potential indicator using a 9-point Likert scale (with 1 denoting “definitely not appropriate care” and 9 denoting “definitely appropriate care”) and were asked to add additional recommendations for consideration or suggestions and comments. Between the first and second round, a panel meeting took place, during which each panelist was provided with an overview of the first round's ratings. Recommendations rated with an overall median score of 8 and 9, without disagreement, were considered to be face valid and reliable [14] and were thus selected as preliminary indicators. Disagreement was defined as the occurrence of ⩾30% of scores in both the bottom [1–3] and the top [7–9] tertile [15]. Recommendations with an overall median rating of 1–3 were considered to range from invalid to equivocal. Newly added recommendations, recommendations with an overall median rating of 4–6 or of 7 with agreement, and recommendations with an overall median rating of 8 or 9 with disagreement were discussed and reformulated when necessary.
After the meeting, all of the discussed and reformulated recommendations (plus additional recommendations) were formatted again as a questionnaire, and the panelists were asked to rate these again. The final selection was made by asking the panel members to prioritize the potential indicators by selecting the “top 5” most important potential indicators, which were selected in the final set if prioritized by ⩾2 panelists. These prioritized potential indicators were further developed into indicators by defining numerators and denominators [16].
Phase 2: Applicability Testing
Before the indicator set was used in a specific setting (e.g., as part of an improvement project in a specific hospital or in a study including several hospitals aiming at comparing and improving the quality of antibiotic treatment of UTI), the applicability of each indicator had to be tested. So, in each practice setting chosen, the next step was to provide empirical evidence of feasibility, interobserver reliability, opportunity for improvement, and case-mix stability for each indicator (figure 1).
Figure 1
Procedural flow chart showing the development of quality indicators (phase 1) and their applicability testing in practice (phase 2). “Not feasible” means that there were not enough data available to evaluate the indicator. “Not reliable” means that the interobserver reliability was too low. “No room for improvement means” that the performance score of the indicator was already high. “Minimum sample size” means that the indicator could be evaluated in a minimal sample size of patients.
We tested the applicability of the indicator set at 4 (1 university and 3 nonuniversity) medical centers in Amsterdam, The Netherlands: Academic Medical Center, Onze Lieve Vrouwe Hospital, Sint Lucas Andreas Hospital, and BovenIJ Hospital. To assess whether applicability was different in different types of wards, we decided to study the internal medicine and urology departments separately. We included the same categories of patients as in our recently written, evidence-based national guideline [6].
Eligible patients included inpatients and outpatient aged >12 years who were treated with antibiotics for a complicated UTI during the period from January 2005 through July 2006. A patient with an uncomplicated UTI was defined as an immunocompetent, nonpregnant woman who had cystitis, without symptoms of tissue invasion or systemic infection and without any functional or anatomical abnormalities of the urinary tract. All other UTIs were considered to be complicated UTIs [17]. Patients with uncomplicated UTI, nosocomial infection, or infection that occurred after a urological procedure were not described in the guideline and were therefore excluded. The identification of patients was performed using the national diagnosis registration system. Subsequent manual screening took place, with use of medical and nursing records, admission sheets, medication charts, and laboratory and culture results. Both inpatients and outpatients were eligible.
Feasibility. For every indicator, feasibility was defined as the availability of administrative data required to evaluate the indicator. An indicator was considered to be feasible if the data necessary to score the indicator could be abstracted from the available data for >70% of cases [18].
Interobserver reliability. Interobserver reliability was measured to assess the reproducibility between observers and between cases. A second investigator rated 10% of all the records of the participating departments of the 4 medical centers. To assess the agreement between the 2 investigators corrected for chance, κ coefficients were calculated. Indicators for which κ<0.60 were considered to be unreliable [19].
Potential room for improvement. “Potential room for improvement” was defined as the sensitivity of a potential indicator to detect variability and changes in the quality of care between and within cases. A potential indicator with an invariable high performance score does not discriminate quality of care between and within cases. Potential indicators with a median performance score of >85% were defined as having little interdepartmental and interhospital variation and were therefore considered to have little room for improvement [20].
Sample size. The applicability of an indicator is also dependent on the number of patients on which it can be applied. Considering the period of inclusion and the number of wards studied, we considered an indicator to be applicable if it had a minimum sample size of 20 patients for the 4 internal medicine departments and of 15 patients for the urology departments.
Case-mix stability. The relationship between certain patient characteristics and the result for the indicator was analyzed using logistic regression analysis. Indicators that are not “case-mix stable” require comparable patient populations when comparing the quality of care in time or between settings. Indicators for specific subpopulations (i.e., indicators 8–13) (table 2) did not need correction for case mix, because we assumed that the subpopulations already accounted for a more homogenous population for different hospital sizes and settings. We studied the distribution of outcome of the general indicators (i.e., indicators 1–7) (table 2) on the basis of age, sex, the presence of systemic symptoms, and comorbidities (cardiovascular disease and urological medical history) at first clinical presentation.
Table 1
Level of supporting evidence.
Table 2
Applicability of potential indicators for quality of antibiotic use in 341 patients with complicated urinary tract infection.
Results
Phase 1: Development of Indicators
Two infectious diseases specialists (S.E.G. and J.M.P.) and 1 quality-of-care specialist (M.E.L.J.H.) independently preselected 39 key recommendations from the national guideline (table 3). None of the key recommendations were supported by grade A evidence. Of the 13 panelist who agreed to participate, all completed the questionnaires for rounds 1 and 2. Eight panelists (62%) were present at the face-to-face panel meeting. Twelve panelists (92%) responded to the prioritizing round, in which 11 panelists provided a top 5 priority list and 1 panelist provided a top 3 priority list.
Table 3
Rating and adding procedure for the selection of potential quality indicators.
After the first rating round, 26 recommendations were accepted as preliminary indicators. Thirteen recommendations were discussed and reformulated during the panel meeting. One recommendation was added (recommendation 40). After the second rating round, there were no recommendations with an overall median rating of 4–6, and consensus was reached for 7 of the revised and newly added recommendations, resulting in 33 potential indicators for the prioritizing round. A final set of 13 potential indicators was prioritized (table 2).
Phase 2: Applicability Testing
Applicability testing (figure 1) of the set of indicators in the 4 specific hospitals started in July 2006 and ended in November 2006. Seven of the 8 approached departments participated. A total of 341 patients were studied. In the internal medicine departments, nearly all patients (190 of 191) were hospitalized, and the majority of patients (112 of 150) in the urology department came from the outpatient clinics. As to be expected, in the internal medicine department, complicated UTIs included pyelonephritis (48 of 191 patients), urosepsis (50 patients), and UTI in immunocompromised patients (92 patients, consisting of 52 diabetic patients, 20 patients who were receiving immunosuppressive drugs, and 20 patients with kidney disease), whereas complicated UTIs in the urology department mainly included acute or chronic prostatitis (67 of 150 patients) or UTIs in patients with anatomical abnormalities of the urinary tract (108 patients).
One indicator—“start antibiotic therapy intravenously in pregnant women with pyelonephritis”—could not be tested, because there were no pregnant women in the study sample. The applicability of the other 12 indicators was tested and compared between the urology and internal medicine departments (table 2).
Feasibility. In the urology departments, 93% of patients had missing data for indicator 7 (“initiate antibiotics within 4 h after clinical presentation”). Furthermore, indicator 11 (“change catheter after initiating antibiotics”) had moderate feasibility: 33% of patients did not have enough data to assess the indicator. In the internal medicine departments, only indicator 7 was determined to have low feasibility: for 75% of patients, the required data to assess the time of antibiotic administration were missing.
Interobserver reliability. Indicator 6 (“duration of therapy should be at least 10 days”) had a low interobserver reliability (k=0.58) in the urology departments and was therefore rejected. In the internal medicine departments, moderate interobserver reliability (k=0.60) was noted for indicator 7 (“initiate antibiotic treatment within 4 h after clinical presentation”). As described above, this indicator also had low feasibility.
Room for improvement. For the urology departments, indicator 3 (“use fluoroquinolones only as oral therapy or because of anaphylaxis related to β-lactam antibiotics”) yielded a high median performance rate (94%) and had little room for improvement (range for hospitals, 90%–100%). For the internal medicine departments, indicator 10 (“do not prescribe prophylaxis for patients using a catheter”) yielded a high median performance rate (100%) and had little room for improvement (range for hospitals, 83%–100%).
Minimum number of patients. Five indicators (indicators 8 and 10–13) had a sample size of <15 patients for the urology departments and were therefore rejected. The indicator concerning diabetic patients (indicator 12) could be applied to a total of only 6 patients for all 4 internal medicine departments together.
Case-mix stability. At the urology departments, correction for sex was necessary for 3 indicators (indicators 1, 2, and 4). These indicators were more often observed in male patients than in female patients. Two indicators (indicators 1 and 6) required correction for patients with a medical history of urological disease: indicator 1 was more often “positive” (i.e., it was followed more often) when the patient had a medical history of urological disease, but indicator 6 was more often “negative” (i.e., not followed). Two indicators (indicators 4 and 6) needed correction for patients who were known to have cardiovascular diseases: indicator 4 was more often “positive” when the patient had a medical history of cardiovascular disease, but indicator 6 was more often “negative.” Because we assumed that the subpopulations already accounted for a more homogeneous population throughout different hospital sizes and settings, no further corrections for case-mix stability were necessary. At the internal medicine departments, empirical therapy was more often prescribed in accordance with the recommendations of the guideline (indicator 2) for elderly patients than for younger patients.
In sum, as shown in table 2, following the applicability test of each of the 13 indicators, 4 indicators turned out to be applicable for our urology departments, and 9 indicators were applicable for our internal medicine departments. These indicators can currently be used by these departments to measure and improve the quality of antibiotic care for patients with complicated UTI.
Discussion
A rigorous selection procedure was used, combining scientific evidence and expert opinion, to define a valid set of quality indicators regarding antibiotic treatment for complicated UTIs in the hospital. The clinical benefit of applying indicators is that they provide insight into current care. But more importantly, they reveal deficiencies that need to be corrected to reach better outcomes in patients. We developed a set of 13 quality indicators defining optimal UTI care: 7 general indicators, 1 for men, 1 for pregnant women, 2 for people with a urinary catheter, 1 for diabetic patients, and 1 for people with kidney disease.
To our knowledge, this is the first set of quality indicators for the antibiotic use of complicated UTI in the hospital. The same procedure was performed by Schouten et al. [20] in 2005, when they defined a set of 15 indicators for the quality of antibiotic use in hospitalized patients with lower respiratory tract infections. We used a more rigorous selection of recommendations by asking the panel to prioritize 5 recommendations in order, to yield a manageable number of indicators to test for applicability in daily practice.
Before using the indicators in a specific practice setting, the next step should always be to test the indicators in the hospital (or nursing home) where they will be used. In our 4 hospitals, the applicability differed by department: only the following 4 indicators for both the urology and internal medicine departments and 5 indicators for the internal medicine departments alone turned out to be applicable: performance of urine culture, prescription of treatment in accordance with guidelines, tailoring of treatment on the basis of culture results, and switches to oral treatment when possible for both the urology and internal medicine departments; and selective use of fluoroquinolones, duration of treatment of at least 10 days, prescription of treatment for men in accordance with guidelines, changing of the catheter in cases of UTI, and adaptation of dosage on the basis of renal function for the internal medicine department. The remaining indicators could either not be measured or showed no room for improvement in these specific departments.
An applicability test in a different setting (e.g., in other Dutch hospitals or in some hospitals in the United Kingdom or United States) may yield different outcomes, with different implications for the hospitals in question. Our applicability test implied the following: if these 4 hospitals want to measure their care as described with the indicators that currently lack feasibility, they first have to improve the registration of patient information. If current care could not be measured because the minimum number of 15 patients could not be reached within the 18 months of enrollment, the measurement period for these specific indicators should be expanded to enroll enough patients to measure care. Regarding the indicators that showed no room for improvement, the measurement showed that, currently, no deficiencies need to be corrected in these specific hospitals.
Our study had several limitations. We used a rigorous procedure, which combined scientific evidence and expert opinion, to define a valid set of quality indicators. Recommendations were extracted from a Dutch guideline and were therefore based on resistance patterns for uropathogens in The Netherlands; in general, these resistance patterns are more favorable for most antimicrobials, compared with those in other countries in Europe and the United States [21]. Therefore, not all recommendations—especially those concerning the choice of antimicrobial agents—can be generalized for other countries; they have to be fine-tuned for the local situation. In addition, none of the preselected Dutch guideline recommendations had high levels of supporting evidence (i.e., level A evidence) (table 1) that guaranteed quality of care. So, unfortunately, (systematic reviews of) randomized, controlled trials exist for none of the indicators that link indicator performance to improved patient outcome. Therefore, measurement of the impact of the indicators on outcome is an essential next step, to make the effort to develop these indicators more meaningful. However, although no content validity was available for any recommendation, we aimed at high face validity and reliability by using a consensus approach that combined scientific evidence and expert opinion. The composition of the expert panel was carefully chosen, taking into account that the composition of the panel can influence the consensus [22, 23]. We selected experts of all involved disciplines, originating from different geographical regions.
A last limitation concerns the applicability test. Our data collection occurred retrospectively, with use of medical and nursing records, admission sheets, medication charts, and laboratory and culture results. Most Dutch hospitals do not have systematic and robust registration systems. Therefore, application of indicators to measure and improve patient care is currently an elaborate, time-consuming activity, hampering the application of these indicators for monitoring or research purposes.
Future studies must prove whether the final set of 13 clinical indicators can be used for an intervention program to improve the quality of antibiotic use for complicated UTI in the hospital. Tailored interventions based on low performance scores for the indicators with different implementation strategies and with postintervention measurements can demonstrate whether it is possible to change antibiotic use in the hospital.
Acknowledgments
We thank all medical specialists who performed the consensus procedure: medical microbiologists P. M. N. Schneeberger and E. E. Stobberingh; hospital pharmacists P. M. G. Filius and R. Janknegt; infectious diseases specialists S. E. Geerlings, J. M. Prins, P. J. van den Broek, and A. I. M. Hoepelman; urologists M. T. W. T. Lock and E. P. van Haarst; nephrologists R. T. Krediet and L. J. Vleming; and gynecologist A. H. Adriaanse. We also thank M. Prins, epidemiologist, for her statistical support. We acknowledge our gratitude to the medical specialists and staff of the participating hospitals for their contribution to the data collection: in the urology departments, T. M de Reijke (Academic Medical Center), G. van Andel (Onze Lieve Vrouwe Hospital), and E. P. van Haarst (Sint Lucas Andreas Hospital); and in the internal medicine departments, K. Brinkman (Onze Lieve Vrouwe Hospital), J. Veenstra (Sint Lucas Andreas Hospital), and M. G. W. Barnas (BovenIJ Hospital).
Potential conflicts of interest. All authors: no conflicts.
- Received July 2, 2007.
- Accepted October 13, 2007.
- © 2008 by the Infectious Diseases Society of America
References
This Article
- AbstractFree
- » Full Text (HTML)Free
- Full Text (PDF)Free
Classifications
-
- Articles and Commentaries
Services
Citing Articles
Google Scholar
PubMed
Related Content
Share
- Email this article
-
CiteULike
Connotea
Delicious
Facebook
Google+
Mendeley
Twitter
Published on behalf of
IDSA Members: For your free access to this journal, log in via the IDSA members area.
The Journal
Impact Factor: 8.186
5-Yr impact factor: 7.898
For The Media
For Authors
- Services for authors
- Self-archiving Policy
- IDSA Conflict of Interest policy
- Instructions for Authors
- ICMJE Form
- Submit Now!
Open access options for authors visit Oxford Open
This journal enables compliance with the NIH Public Access Policy
Corporate Services
Alerting Services
Related IDSA Publications
