Top

Intensive Care Medicine

Published in:

01-12-2003 | Original

Using hierarchical modeling to measure ICU quality

Authors: Laurent G. Glance, Andrew W. Dick, Turner M. Osler, Dana Mukamel

Published in: Intensive Care Medicine | Issue 12/2003

Abstract

Objective

To determine whether hierarchical modeling agrees with conventional logistic regression modeling on the identity of ICU quality outliers within a large multi-institutional database.

Design

Retrospective database analysis.

Setting and patients

Subset of the Project IMPACT database consisting of 40,435 adult patients admitted to surgical, medical, and mixed surgical-medical ICUs (n =55) between 1997 and 1999 who met inclusion criteria for SAPS II.

Measurements and results

The SAPS II score was customized to this database using conventional logistic regression and using a hierarchical (random coefficients) model. Both models exhibited excellent discrimination (C statistic) and calibration (Hosmer-Lemeshow statistic). The hierarchical and nonhierarchical models hadC statistics of .870 and .865, and HL statistics of 3.71 (p>.88, df=8) and 8.94 (p>.35, df=8), respectively. Since the random effects component of the hierarchical model accounts for between-hospital variability, only the fixed-effects coefficients were used to calculate the expected mortality rate based on the hierarchical model. The ratio and 95% confidence intervals of the observed to expected mortality rate were calculated using both models for each ICU. ICUs whose observed/expected ratio was either less than 1 or greater than 1, and whose 95% confidence interval did not include 1 were labeled as either high-performance or low-performance outliers, respectively. Analysis using κ statistic revealed almost perfect agreement between the two models (nonhierarchical vs. hierarchical) on the identity of ICU quality outliers.

Conclusions

Models obtained by customizing SAPS II using a nonhierarchical and a hierarchical approach exhibit excellent agreement on the identity of ICU quality outliers.

Available only for authorised users

Knaus WA, Zimmerman JE, Wagner DP, Draper EA, Lawrence DE (1981) APACHE-acute physiology and chronic health evaluation: a physiologically based classification system. Crit Care Med 9:591–597CrossRef

Rivera-Fernandez R, Vazquez-Mata G, Bravo M, Aguayo-Hoyos E, Zimmerman J, Wagner D, Knaus W (1998) The APACHE III prognostic system: customized mortality predictions for Spanish ICU patients. Intensive Care Med 24:574–581CrossRef

Le Gall JR, Lemeshow S, Saulnier F (1993) A new Simplified Acute Physiology Score (SAPS II) based on a European/North American multicenter study. JAMA 270:2957–2963CrossRef

Lemeshow S, Teres D, Klar J, Avrunin JS, Gehlbach SH, Rapoport J (1993) Mortality probability models (MPM II) based on an international cohort of intensive care unit patients. JAMA 270:2478–2486CrossRef

Zimmerman JE, Shortell SM, Rousseau DM, Duffy J, Gillies RR, Knaus WA, Devers K, Wagner DP, Draper EA (1993) Improving intensive care: observations based on organizational case studies in nine intensive care units: a prospective, multicenter study. Crit Care Med 21:1443–1451CrossRef

Iezzoni LI (1996) On opening “black boxes” and looking inside. Int J Qual Health Care 8:209–210CrossRef

Iezzoni LI (1991) ‘Black box’ medical information systems. A technology needing assessment. JAMA 265:3006–3007CrossRef

Iezzoni LI (1997) The risks of risk adjustment. JAMA 278:1600–1607CrossRef

Burgess JF Jr, Christiansen CL, Michalak SE, Morris CN (2000) Medical profiling: improving standards and risk adjustments using hierarchical models. J Health Econ 19:291–309CrossRef

10.

Marshall EC, Spiegelhalter DJ. Institutional Performance. In: Leyland AH, Goldstein H, editors (2001) Multilevel modeling of health statistics. Wiley, New York, pp 127–142

11.

Christiansen CL, Morris CN (1997) Improving the statistical approach to health care provider profiling. Ann Intern Med 127:764–768CrossRef

12.

Shahian DM, Normand SL, Torchiana DF, Lewis SM, Pastore JO, Kuntz RE, Dreyer PI (2001) Cardiac surgery report cards: comprehensive review and statistical critique. Ann Thorac Surg 72:2155–2168CrossRef

13.

Moreno R, Matos R (2000) The “new” scores: what problems have been fixed, and what remain? Curr Opin Crit Care 6:158–165CrossRef

14.

Leyland AH, Goldstein H (eds) (2001) Multilvel modeling of health statistics, ch 1. Wiley, New York

15.

Goldstein H, Spiegelhalter DJ (1996) League tables and their limitations: statistical issues in comparisons of institutional performance. J R Stat Soc 159:385–443CrossRef

16.

Glance LG, Osler T, Dick AW (2002) Identifying quality outliers in a large multi-institutional database using customized versions of SAPS II and MPM II. Crit Care Med 30:1995–2002CrossRef

17.

Glance LG, Osler T, Dick A (2002) Rating the quality of ICUs—is it a function of the ICU scoring system? Crit Care Med 30:1969–1975CrossRef

18.

Rapoport J, Teres D, Steingrub J, Higgins T, McGee W, Lemeshow S (2000) Patient characteristics and ICU organizational factors that influence frequency of pulmonary artery catheterization. JAMA 283:2559–2567CrossRef

19.

Diringer MN, Edwards PhD DF (2001) Admission to a neurologic/neurosurgical intensive care unit is associated with reduced mortality rate after intracerebral hemorrhage. Crit Care Med 29:635–640

20.

Cook SF, Visscher WA, Hobbs CL, Williams RL (1999) Preliminary results from a pilot validity study of a new observational database (abstract). Crit Care Med 27:36ACrossRef

21.

Le Gall JR, Lemeshow S, Saulnier F (1993) A new Simplified Acute Physiology Score (SAPS II) based on a European/North American multicenter study. JAMA 270:2957–2963CrossRef

22.

Leyland AH, Goldstein H (2001) Multilevel modelling of health statistics. Wiley, New York

23.

Rabe-Hesketh S, Skrondal A, Pickles A (2002) Estimation of generalized linear mixed models. Stata J 2:1–21CrossRef

24.

Hox J (2002) Multilevel analysis: techniques and applications. Erlbaum, Mahwah

25.

Hadorn DD, Keeler E, Rogers WH, Brook RH (1993) Assessing the performance of mortality prediction models. RAND/UCLA/Harvard Center for Health Care Financing Policy Research, Santa Monica

26.

Hosmer DW, Lemeshow S (2000) Applied logistic regression, 2nd edn. Wiley-Interscience, New York

27.

Hosmer DW, Lemeshow S (1995) Confidence interval estimates of an index of quality performance based on logistic regression models. Stat Med 14:2161–2172CrossRef

28.

Efron F, Tibshirani RJ (1993) An introduction to the bootstrap. Chapman & Hall, New York

29.

Anonymous (2000) To err is human: building a safer health system. National Academy, Washington

30.

Anonymous (2001) Crossing the quality chasm. National Academy, Washington

31.

Iezzoni LI (1997) The risks of risk adjustment. JAMA 278:1600–1607CrossRef

32.

Hurtado MP, Swift EK, Corrigan JM (2001) Envisioning the national health care quality report. National Academy, Washington

33.

Lansky D (2002) Improving quality through public disclosure of performance information. Health Aff (Millwood) 21:52–62

Title: Using hierarchical modeling to measure ICU quality
Authors: Laurent G. Glance
Andrew W. Dick
Turner M. Osler
Dana Mukamel
Publication date: 01-12-2003
Publisher: Springer Berlin Heidelberg
Published in: Intensive Care Medicine / Issue 12/2003
Print ISSN: 0342-4642
Electronic ISSN: 1432-1238
DOI: https://doi.org/10.1007/s00134-003-1959-9

At a glance: The STEP trials

Springer Medicine

Using hierarchical modeling to measure ICU quality

Abstract

Objective

Design

Setting and patients

Measurements and results

Conclusions

At a glance: The STEP trials

Springer Medicine

Abstract

Objective

Design

Setting and patients

Measurements and results

Conclusions

Please log in to get access to this content

Other articles of this Issue 12/2003

Alcohol withdrawal severity is decreased by symptom-orientated adjusted bolus therapy in the ICU

Temporal changes in muscle glutathione in ICU patients

Erratum to: Comparison of effectiveness of two urinary drainage systems in intensive care unit: a prospective, randomized clinical trial

Unusual central venous catheter malposition into the left internal mammary vein: a case report

Optimizing outcomes for older patients treated in the intensive care unit

Impact of adequacy of initial antimicrobial therapy on the prognosis of patients with ventilator-associated pneumonia