Hospital-wide, dynamic, individualized prediction of central line-associated bloodstream infections-development and temporal evaluation of six prediction models

Abstract

Background

Central line-associated bloodstream infections (CLABSI) are preventable hospital-acquired infections. Predicting CLABSI helps improve early intervention strategies and enhance patient safety.

Aim

To develop and temporally evaluate dynamic prediction models for continuous CLABSI risk monitoring.

Methods

Data from hospitalized patients with central catheter(s) admitted to University Hospitals Leuven between 2014 and 2017 were used to develop five dynamic models (a landmark cause-specific model, two random forest models, and two XGBoost models) to predict 7-day CLABSI risk, accounting for competing events (death, discharge, and catheter removal). The models’ predictions were then combined using a superlearner model. All models were temporally evaluated on data from the same hospital from 2018 to 2020 using performance metrics for discrimination, calibration, and clinical utility.

Findings

Among 61629 catheter episodes in the training set, 1930 (3.1%) resulted in CLABSI, while in the test set of 44544 catheter episodes, 1059 (2.4%) experienced CLABSI.

Among individual models, one XGBoost model achieved the highest AUROC of 0.748. Calibration was good for predicted risks up to 5%, while the cause-specific and XGBoost models overestimated higher predicted risks. The superlearner displayed a modest improvement in discrimination (AUROC up to 0.751) and better calibration than the cause-specific and XGBoost models, but worse than the random forest models. The models showed clinical utility to support standard care interventions (at risk thresholds between 0.5–4%), but not to support advanced interventions (at thresholds 15–25%).

Conclusion

Hospital-wide CLABSI prediction models offer clinical utility based on medium-risk thresholds. Clinical utility at present may be limited as the model performance deteriorated over time.

Introduction

Central line-associated bloodstream infection (CLABSI), defined as a bacteraemia occurring at least 48 h after admission in the absence of infection at another site [1], is a priority for prevention due to its association with prolonged hospital stays, increased healthcare costs and elevated morbidity and mortality [2,3,4]. Developing a risk prediction model for CLABSI can assist healthcare professionals, including clinicians and infection control teams, with timely risk assessments to inform preventative interventions and improve patient outcomes.

A previous systematic review [5] assessed existing risk prediction models for CLABSI and found that 15 out of 16 developed models were static, with the remaining one being dynamic. Static models may be less effective, as they fail to capture the changes in a patient’s health throughout their hospital stay. The increasing availability of continuously updated electronic health records (EHR) datasets offers a new opportunity. As demonstrated by other longitudinal studies on EHR data for various clinical outcomes [6,7,8], these datasets enable more frequent measurements that may better reflect the patient’s evolving condition or disease progression. By continuously updating patient information and renewing predictions, dynamic prediction becomes feasible, providing a more nuanced and timely assessment of the patient’s health status [9].

We developed and temporally evaluated dynamic risk prediction models for capturing the 7-day risk of CLABSI among hospitalized patients with central venous catheters, utilizing EHR data from the University Hospitals Leuven. We developed six models: a cause-specific regression model, two random forest (RF), two eXtreme Gradient Boosting (XGB) models, and a superlearner ensemble.

Methods

The study reporting follows the TRIPOD + AI statement [10].

Study design and data

Patient data were extracted from the EHR system of the University Hospitals Leuven for hospital admissions in the period January 2014—December 2020. We included patient admissions with registration of central catheters: centrally inserted central catheter (CICC), tunneled cuffed and non-cuffed central venous catheter, port catheter (Totally Implanted Vascular Access Devices—TIVAD), peripherally inserted central catheter (PICC) and dialysis catheter. No exclusion criteria were applied; all patients from all medical wards were included, from neonates to geriatric patients, including intensive care (ICU) admissions, reflecting the target population for a hospital-wide CLABSI alert system. The training set used for model development included hospital admissions starting between January 2014 and December 2017. The test set used for temporal evaluation included hospital admissions between January 2018 and December 2020. The data extraction and preparation procedure has been described in our previous studies conducted on data collected prior to January 2014 [11, 12] and is repeated.

The following levels of the outcome are considered:

• CLABSI: any laboratory-confirmed bloodstream infection (LCBSI) for a patient with central catheter or within 48 h after the central catheter removal that is not present on admission (detected in the first 48 h from admission), that is not a secondary infection (the same germ detected in another sample type, like urine, sputum, etc. in a time window of ± 17 days from the blood sample collection time), is not a skin contamination (based on the list of skin contaminants) and is not a mucosal barrier injury LCBSI (one positive culture of intestinal microorganism or two positive cultures with Streptococcus Viridans for a patient with neutropenia in a time window of ± 3 days from the blood sample collection time). The CLABSI definition was applied retrospectively on the extracted data following the definition of the Belgian public health institute Sciensano, published in 2019 [1]. While Sciensano does not impose a specific time window for secondary infections, we used 17 days, considering the infection window period of 14 days plus the repeat infection timeframe of 3 days [13].

• Discharge: Hospital discharge or 48 h after catheter removal, whichever happens first. According to the CLABSI definition from the Belgian Public Health Institute Sciensano, the patient remains at risk of CLABSI for 48 h after catheter removal. For conciseness, we refer to this as “discharge” thereafter.

• Death: The first contact with palliative care during admission, transfer to palliative care or patient death, whichever happens first. Patients stop being closely monitored in palliative care and predictions on this ward are not actionable. For conciseness, we refer to this as “death” thereafter.

These three outcomes are considered competing events, which means that once a patient experiences one of these events, they are no longer at risk for the other competing events. The primary event of interest in this study is CLABSI, with discharge and death treated as competing events to prevent biased risk estimates and poor model performance, as shown in a previous UZ Leuven EHR study from 2012 to 2013 [12].

Patient admissions were split into catheter episodes. A catheter episode starts at catheter placement (if placed in ICU) or at the first structured catheter observation (e.g. monitoring observations like dressing status, lumens flushed, bandage change, etc.) during the admission (outside ICU). Exact catheter placement and removal times are only consistently recorded in the ICU. As part of the hospital’s procedures for catheter monitoring, catheter observations are recorded fairly consistently during three daily nurse shifts in structured fields in the EHR system. The continuity of catheter episodes (without data gaps) is ensured when patients get transferred to and from the ICU. For patients admitted with long-term catheters, such as port or tunneled catheters, the catheter episode starts at the first recorded catheter observation. A catheter episode ends when no catheter observation is made for 48 h (as a proxy for catheter removal) or at catheter removal, or when discharge, death or CLABSI occurs. A catheter episode was further split in landmarks (LMs) marking every 24 h: LM0 is the time of the first catheter observation, LM1 is 24h later, and so on (Supplementary Material 1). The risk of CLABSI within the following 7 days is predicted at each landmark throughout the catheter episode.

The complete dataset consisted of 160 baseline and time-varying variables comprising patient demographics, ward transfers, medication, laboratory test results, comorbidities, vital signs and catheter registrations. They were selected out of 302 extracted variables based on a priori information, including insights from analyses on data prior to 2014, clinical expertise, and reliability of variables (i.e. expectation that the variable will be recorded consistently in future data). Whenever multiple measurements were available during a time window, these were aggregated into a single landmark value (e.g.: maximum temperature in last 24 h, Supplementary Material 2).

Missing data were encountered for 66 variables and were imputed using missForestPredict [14, 15] on the training dataset using the complete set of variables. The outcome was not included in the imputation process. The test set was imputed using the imputation model learned on the training set. (Supplementary material 3 provides further information).

Model development and evaluation

We built five different dynamic models that use time-varying predictor information and jointly model CLABSI, death, discharge and “no event” within 7 days as the outcome levels: a regression model, specifically a landmark cause-specific (CS) model, considering death and discharge as competing events for CLABSI, two multinomial RF models and two multinomial XGB models.

The CS model used a limited set of 62 variables, including interaction and polynomial terms (Supplementary Material 2). The limited set of variables was selected from the complete set of variables based on their predictiveness for CLABSI in data prior to 2014 (Supplementary Material 4). One RF and one XGB model were built on the same variables, excluding interaction and the polynomial terms, resulting in 58 variables (RF-LIM and XGB-LIM). Another RF and XGB model were built using the complete set of 160 variables with either explicit data-driven variable selection (RF-ALL) or implicit data-driven variable selection through regularization hyperparameters (XGB-ALL).

Further, we combined the predictions of the five models into a superlearner model, which previously demonstrated improved predictive performance [16, 17]. We used ten-fold cross-validation (called outer CV) to parametrize the superlearner. Within the outer CV we performed nested fivefold cross-validation for tuning the XGB models and used the out-of-bag samples for tuning the RF models. (Supplementary Material 5).

The models compared in this study were listed in Table 1.

Table 1 Different prediction models used in this study

Internal evaluation was performed for the individual models (CS, RF and XGB) by evaluating performance metrics using the outer CV held-out folds. To assess the model performance over time, temporal evaluation was performed for all models including the SL model on the test set (2018–2020). Threshold-independent metrics for discrimination (Area Under the ROC curve) and calibration (assessing how well the predicted CLABSI probabilities align with the observed outcomes) are reported at each landmark in a catheter episode. Calibration plots over all landmarks are presented. Threshold-dependent metrics (net benefit, sensitivity, specificity, positive predictive value, negative predictive value and alert rate) were evaluated on temporal evaluation, for all landmarks combined. We considered two risk thresholds for generating alerts: a medium-risk threshold for which the nursing team can intervene with standard care practice (standard care bundle with enhanced routine maintenance check) and a high-risk threshold for enhanced interventions for additional protection (e.g.: chlorhexidine-impregnated dressings and washcloths or chlorhexidine-coated catheters). Following consensus of different representatives of the model’s users (nurses, clinicians and infection preventionists), for the medium-risk threshold we considered the values: 0.5%, 1%, 2%, 3%, 4%, 5% and for the high-risk threshold 15%, 20%, 25%. Decision curves [18, 19] were also presented to assess clinical utility of intervention decisions at the proposed thresholds.

The code for the project is available at: https://github.com/sibipx/CLABSI_prediction_models/. Model objects are available upon request by contacting the corresponding author. The code to build the RF and XGB models has been run on a high-performance computing cluster using 36 cores and 128 GB RAM for the RF model and an NVIDIA V100 GPU for the XGB model. R version 4.2.1, randomForestSRC [20] 3.2.2, ranger [21] 0.15.1 and xgboost [22] 2.0.0.1 have been used to build the models.

Results

Data characteristics

The training set consisted of 55,910 admissions, 61,628 catheter episodes with complete follow up for all event types (1930 CLABSI, 3206 deaths and 56,492 discharges) and 541,815 landmarks. The test set consisted of 40,994 admissions, 44,544 catheter episodes (1059 CLABSI, 1901 deaths and 41,584 discharges) and 391,353 landmarks. Supplementary material 6 presents the outcome prevalence and incidence. Table 2 presents the patient demographics and catheter information at baseline.

Table 2 Patient characteristics at baseline (start of the catheter episode); median (IQR) for continuous variables, after missing data imputation; n (%) for categorical variables. Patients can have two or more different catheter types simultaneously, therefore catheter type has been coded as a binary rather than a categorical variable with multiple categories

Model evaluation

Considering the individual models, the XGB-ALL model had the highest AUROC on temporal evaluation, reaching a maximum of 0.748 at landmark 8 (i.e. 8 days after the start of the catheter episode). On the other hand, the landmark cause-specific model, which combined separate Cox models for each competing risk with a smoothed baseline hazard, showed the lowest AUROC compared to machine learning approaches (Fig. 1a), despite outperforming other statistical regression models in a previous study based on earlier data [12]. As a result of the model-driven feature selection, the RF-ALL model selected slightly less variables compared to the RF-LIM model yet had slightly better performance (45 versus 58 variables out of which 25 are common). The XGB-ALL model did not perform variable selection. The performance decreased with each year on the test set. In Supplementary Material 7 we present the models evaluated on each year of the test set: 2018, 2019 and 2020.

Fig. 1

Prediction performance at each landmark on the test. AUROC = Area Under the ROC curve; E:O = Expected:Observed. The vertical bars represent confidence intervals; for better visualisation, these are shown only for two models: CS-LIM and SL. The confidence interval bars for the SL model are shifted to the right with a small offset for better visualisation

The calibration plot (Fig. 2a) showed agreement between observed and predicted risks for all models at low predicted risk levels (lower than 5%). The majority of predictions fall within this range due to the low event rate of CLABSI (Fig. 2b). The high density of data in this area makes it easier for all models to learn and accurately predict low-risk cases, leading to higher agreement. The cause-specific and XGBoost models produced overly extreme estimates at high predicted risk levels (Fig. 2). The E:O ratios higher than one indicate that the models tend to overestimate the CLABSI risk (Fig. 1b). The E:O ratio for all models temporally evaluated on each year also showed an increasing trend over year, which suggested more severe overestimation at later years in the test data (Supplementary Material 7).

Fig. 2

Calibration curves using restricted cubic splines (a); prediction boxplots (b); decision analysis curves (c) for all models on temporal evaluation using all landmarks. a The dashed grey line represents the identity function (perfect calibration). c The dashed grey lines mark the proposed thresholds for model evaluation

The superlearner model showed a minimal improvement in discrimination: maximum AUROC of 0.751 at landmark 6. It also displayed better calibration than the cause-specific and XGB models, but its calibration was still inferior to that of the RF models (Fig. 2). The proportional contributions of each model in the superlearner were: 0.02 for CS, 0.06 for RF-LIM, 0.27 for RF-ALL, 0 for XGB-LIM and 0.65 for XGB-ALL.

The models showed no benefit to support specific technological interventions (high-risk thresholds 15%−25%), but they had clinical utility to support standard care interventions for medium-risk thresholds up to 3% for the CS model and up to 4% for the other models (Fig. 2c and Table 3). Nonetheless, the utility of the models differed by ward. For example, the superlearner had utility for all (0.5%−5%) medium-risk thresholds in haematology, traumatology, gynaecology, neonatology, thoracic surgery, neurology, internal medicine, and urology (Supplementary Material 7), while it depended on the specific threshold in other wards.

Table 3 Threshold dependent metrics at different thresholds. The Net Benefit has been multiplied by 100 and represents the Net true positives per 100 landmarks (catheter-days)

The sensitivity, specificity and alert rate at medium-risk thresholds varied widely; the PPV remained below 5% regardless of threshold. The highest ward-specific alert rates at medium-risk thresholds are reached at threshold 0.5% in emergency (99.1%) and ICU (98.9%) and the lowest at threshold 5%: 0.2% in traumatology and gynaecology.

Among the individual models, XGB-ALL had the best discrimination and the best net benefit at medium-risk thresholds. The superlearner showed slightly better discrimination and calibration than XGB-ALL, and superior net benefit at thresholds 3% and 4%, but slightly lower net benefit at the other three thresholds (0.5%, 1%, 2%). From a practical perspective, the superlearner comes with the disadvantage of deploying four models (considering that XGB-LIM got zero weight). Both models include the same number of variables, as XGB-ALL did not perform variable selection.

Temporal shift

All models showed a performance deterioration on temporal evaluation (test set) compared to the internal evaluation (Supplementary Material 7). For example, the median CV AUROC for XGB-ALL reached the maximum value of 0.774 at landmark 3.

To understand the reduced performance in temporal evaluation, we first analysed predictor shifts across training and test data (Supplementary Material 8). Fluctuations in missing data and variable sparsity were observed across years. Additionally, significant shifts in log(D-dimer) distributions negatively impacted temporal evaluation. A lab machine upgrade in May 2019 expanded the maximum measurable D-dimer range from 7,650 (training) to 284,780 (test), leading to inflated predictions for patients with extremely high D-dimer levels. Removing log(D-dimer) from the landmark cause-specific model improved overall AUC and calibration, making it comparable to the XGB models’ calibration.

Furthermore, we analysed the outcome prevalence shift across training and test data (Supplementary Material 8). A declining trend in CLABSI cases was observed across medical specialties like geriatrics and thoracic surgery, along with a continuous hospital-wide decrease in outcome prevalence, despite relatively stable counts of positive cultures and catheter episodes. This reflects improvements in clinical care, likely drive by the gradual implementation of the line care bundle from 2012 to full hospital-wide adoption by 2016.

To assess the impact of shifts in predictor-outcome associations, we fitted landmark cause-specific models using the limited set of variables on yearly data from 2014 to 2020 (Supplementary Material 8). Results showed lower apparent AUCs for 2019 and 2020 at the first six landmarks compared to other years, particularly the apparent AUCs for 2019 worsened after landmark nine. As training the model on test data did not improve the prediction model’s performance, these results suggest the decreasing performance reflect complex changes in the evolving clinical environment that made the prediction problem harder, rather than the issues with the model. Due to time and computerization constraints, only the landmark cause-specific models were fitted on the full training and test data. XGBoost models on the full dataset may be explored in future studies.

Discussion

We developed and evaluated dynamic models for predicting the 7-day risk of CLABSI in patients with central catheters using EHR data. XGBoost models showed the best discrimination and random forest models showed the best calibration. Combining multiple models through a superlearner framework marginally improved discrimination and partially corrected the miscalibration of XGBoost. Temporal evaluation revealed a decrease in model performance compared to internal evaluation. The CLABSI incidence has also decreased over time, which is reflected in the increasing overestimation of predicted risks over time.

The CLABSI models have practical relevance when implemented in an early warning system targeting users like nurses and clinicians on different wards, as well as infection preventionists and the Vascular Access Specialty Teams who oversee the care of all hospitalized patients. We aimed for two implementation thresholds: medium-risk alerts, for which nurses can intervene by verifying the catheter maintenance care bundle and infection preventionists can check the adherence to the care bundle items, and high-risk alerts, which prompt for specific technological interventions (e.g.: dressing type, catheter replacement). The models demonstrated both hospital-wide and ward-specific utility for medium-risk alerts (up to a threshold of 3%), but not for high-risk alerts.

In comparison to previously published models [5], which are mostly static, we developed and temporally evaluated dynamic models. While our systematic review identified one dynamic model [23], it was limited to patients in the cardiac ICU or cardiac ward, whereas our models targeted a hospital-wide population. Additionally, their definition of CLABSI as any new positive blood culture was much broader than the surveillance definition of CLABSI. Moreover, previously published models were at high risk of bias, e.g. by exclusions based on future data, temporal data leakage, or using predictors unavailable at the time of prediction. Furthermore, previous model evaluations have generally been less rigorous and lacked an implementation-driven approach compared to ours.

Our study has limitations. Although we performed a comprehensive data extraction, we did not particularly focus on extracting strong predictors for the competing events included in our models: death and discharge. Further refining the extracted predictors or variable selection strategy might lead to improved model performance. Second, we did not account for variable dynamics in the model. Even with dynamic predictions, a prediction at a given time relies only on the values at the prediction time, or in a fixed time window before the prediction time (typically of 24 h) without incorporating previous values during a catheter episode (e.g.: values 48 h before the prediction time) or trends for individual variables within a catheter episode (e.g.: increase / decrease over time in lab values). In a previous analysis on data prior to 2014, we tested models with two lagged values or with differences between values at consecutive landmarks, but these did not improve the performance (unpublished results). However, this approach may have been limited, and more advanced models like recurrent neural networks, transformers [24], or DeepHit [25], could be explored in the future. Moreover, incorporating more diverse models within the superlearner framework may improve its overall performance. Third, we observed a deterioration in performance on temporal evaluation, reflecting potential shifts in patient care practices, catheter management strategies, patient populations or data recording procedures in the EHR system. We observed changes in patient distribution across hospital wards, and improved registration of laboratory test results, scores and respiratory indicators during the test period (2018—2020), most probably partly attributable to closer patient monitoring during the COVID-19 pandemic. Despite these changes, we found that the models performed similarly in 2018 and in internal validation, but their performance dropped in 2019. Performance in 2019 and 2020 remained comparable, with no clear impact from the COVID-19 pandemic period. Calibration deterioration is more common than discrimination deterioration and can often be corrected by model recalibration [26, 27]. When we refitted the models on data per calendar year separately, we observed a decline in discrimination with each successive year. This suggests that the CLABSI prediction task is becoming increasingly challenging, raising concerns about the utility of these models for data beyond 2020 if this trend continues in the coming years. Lastly, we acknowledge limitations related to fairness in our study. Although we included age and gender as predictors in our models, data on race, ethnicity or socioeconomic status were not available for extraction. This limits our ability to fully explore how the predictive performance varies across different demographic groups. Furthermore, wealthier patients who can afford single rooms may be at a lower risk of hospital-acquired infections, including CLABSI, compared to patients in shared rooms. However, it is difficult to determine whether socioeconomic bias is reflected in our models.

Further research is needed to determine if CLABSI models have practical applicability or are generalizable to other hospitals. Following the CLABSI definition, our models predict infections in patients with central catheters where no other identifiable source of infection can be established; these infections are not always directly attributed to the central catheter. Consequently, interventions like improved line care do not guarantee prevention of infections classified as CLABSI, especially in a hospital where the infection prevention strategy is already of high standard. Implementation studies assessing the impact on clinical decision-making, patient outcomes, user-perceived usefulness or even adverse outcomes (e.g.: unnecessary catheter removals) could help determine if the model adds practical value in healthcare settings. The data extraction from various clinical databases and the proprietary format of catheter observations would hinder generalization to hospitals using a different EHR system. To ensure model compatibility across different EHR systems, it is essential to standardize data extraction for both model development and implementation, and perform clinical concepts mapping, using standards like FHIR [28] and OMOP CDM [29]. Furthermore, we adhered to the Belgian Institute of Public Health’s definition of CLABSI, which might differ slightly from other local or international definitions. However, testing the models’ performance in Belgian hospitals with the same EHR system may prove feasible, after adapting the mapping configuration between extracted items and the clinical concepts (without changing the extraction format) and assessing the quality of the data extraction and preparation.

Conclusion

Our study proposed a number of dynamic models for predicting the 7-day risk of CLABSI in patients with central catheters. The overall performance of the models was modest: the maximum AUROC of 0.751 was achieved by the superlearner model at day 6. Temporal evaluation revealed dataset shifts. The models showed clinical utility for generating alerts to check standard care interventions based on medium-risk thresholds, but not for alerts to consider technological interventions based on high-risk thresholds.