Development of a prediction model for antimicrobial stewardship pharmacy consultations to identify high-risk pediatric patients: a retrospective study across two centers

Abstract

Background

Antimicrobial Stewardship Pharmacy Consultation (ASPC) in China has been shown to reduce patients' length of stay (LOS). However, prolonged LOS remains a challenge, resulting in unnecessary psychological and financial burden for patients.

Objective

This study aimed to develop a prediction model using ASPC parameters to identify high-risk pediatric patients with infectious diseases. These patients received ASPC interventions but still experienced prolonged LOS, which defined their high-risk status.

Methods

Predictors for the ASPC model were selected using lasso regression, a nomogram was developed using multivariate logistic regression, and internal validation was performed using tenfold cross-validation. The data set consisted of 474 electronic medical records of pediatric patients with infectious diseases from two hospitals. LOS was dichotomized at the median, and patients with LOS greater than the median were considered to have achieved the outcome.

Results

The proportion of outcome events was set at 50% by design. Five independent predictors were identified in the ASPC model: (1) the suggestions from the crucial consultation (OR: 1.74; 95% CI: 1.10 to 2.74), (2) weight (OR: 0.98; 95% CI: 0.97 to 1.00), (3) whether the patient received first aid (OR: 0.54; 95% CI: 0.3 to 1.00), (4) the aim of the crucial consultation (OR: 0.15; 95% CI: 0.03 to 0.66), and (5) whether the patient was critically ill (OR: 0.22; 95% CI: 0.12 to 0.41). The ASPC model showed good discrimination with a C-statistic of 0.772 (95% CI: 0.748 to 0.797) and good calibration performance with intercept and slope values of 0.00 (95% CI: -0.12 to 0.12) and 0.93 (95% CI: 0.82 to 1.04), respectively, under tenfold cross-validation.

Conclusions

The antimicrobial stewardship pharmacy consultation model has good discrimination and calibration, and effectively identifies patients at risk for prolonged length of stay.

Key messages

Extensive research on clinical pharmacy practice has highlighted the important role that clinical pharmacists play in patient care and their valuable contributions to personalized medical treatment. However, the quality of pharmacy consultations in China still requires improvement to gain wider recognition from clinicians. Our study examined the impact of Antimicrobial Stewardship Pharmacy Consultations (ASPCs) on the length of stay (LOS) among pediatric patients with infectious diseases and developed a risk prediction model to identify the high-risk patients who received ASPC interventions yet still experienced prolonged LOS. This model has the potential to enhance the quality of ASPC, reduce patients' psychological and financial burdens, and support the advancement of clinical pharmacy practice in China.

Introduction

The various responsibilities of the clinical pharmacist are collectively referred to as clinical pharmacy practice, which encompasses pharmacy services as well as pharmaceutical care [1]. The specific responsibilities and scope of work of clinical pharmacists vary across different countries and regions [2]. In China, a majority of tertiary hospitals encounter challenges in establishing multidisciplinary teams including clinical pharmacists for various departments, and the paucity of clinical pharmacists makes it difficult for their professional practice to coverage throughout patient's length of stay (LOS) [3, 4]. Consequently, pharmacy consultation (PC) has become one of the most dominant forms of clinical pharmacy practice, and this is anticipated to persist. PC is a patient-centered, problem-oriented, clinician-initiated program led by a clinical pharmacist. It provides brief pharmacy services to patients during their LOS, which is generally completed within 24 h. Problem orientation is at the heart of PC, and in the two hospitals included in this study, most of the problems were related to antimicrobial stewardship. These problems encompassed the identification of causative bacteria, the selection of appropriate antimicrobial drugs, medication adjustments, and the implementation of combination or downgraded drug regimens. The rational use of antimicrobial drugs to prevent resistance in some pathogenic microorganisms has been a long-standing global consensus [5]. Clinical pharmacists in China have attached great importance to this consensus and have been controlling the use of antimicrobial drugs for a considerable time [6]. A team of researchers conducted a series of five studies, which confirmed the effectiveness of antimicrobial stewardship pharmacy consultation (ASPC) in patients with infectious diseases in a series of four studies [7,8,9,10,11], thereby emphasizing the significant role that clinical pharmacists play in antimicrobial stewardship.

A multitude of studies have demonstrated that multifaceted, clinical pharmacist-led interventions result in a reduction in median or average LOS, which can range from to approximately 1 day to 6 days [12,13,14,15,16,17]. This is also true for more targeted antimicrobial stewardship, which has been found in a number of studies conducted in China, including a downward trend in mean LOS on a line graph in a Wuhan hospital [18], a relative reduction in mean LOS of approximately 5% in two Jiangsu hospitals [19], and a reduction in median LOS of 13 days [20]. Although there are still some studies that do not support the conclusion of a reduction in LOS [21], no study has found that these interventions lead to an increase in patient's LOS. In summary, there is sufficient evidence to recognize that clinical pharmacist-led antimicrobial stewardship can help reduce patient's LOS, as well as the ASPC.

Predictive modeling is one of the most widely used data analysis methods in statistics, encompassing widely applied regression analysis and machine learning algorithms. In healthcare domain, these methods have been used to develop a variety of diagnostic and prognostic models, which have subsequently evolved into various types of companion diagnostic systems and risk scoring tools. These tools serve to support healthcare professionals in decision-making processes concerning diagnosis and treatment. The prediction of LOS has historically been a subject of significant interest in healthcare domain, with its extensive scope giving rise to research with varied focuses. For example, a prediction model focused on the LOS following laparoscopic appendectomy in pediatric patients has underscored certain predictors, such as age, as well as inflammatory markers that can be utilized to predict LOS [22]. Another study focused on predicting LOS in pediatric patients using machine learning algorithms has highlighted the contribution of predictors such as disease diagnosis and morbidity [23]. This study, focusing on the intervention process of ASPC, sought to develop a prognostic model that could assist clinical pharmacists in identifying high-risk pediatric patients with infectious diseases who are likely to have prolonged LOS after ASPC. The expectation is that it may be possible to find that some of the details of ASPC have an impact on a patient's LOS.

Methods

Data collection, inclusion and exclusion criteria

All electronic medical records were obtained from two tertiary hospitals: the First and Second Affiliated Hospitals of Guangxi Medical University. The data were collected and reviewed independently by two experienced researchers.Inclusion criteria: Between 2021 and 2023, electronic medical records were collected for pediatric patients with infectious diseases from all affiliated wards and the Pediatric Intensive Care Unit (PICU) who had received pharmacy consultations at least once.

Exclusion criteria:

• (1) Patients who received the pharmacy consultation only once and it was not related to antimicrobial stewardship.

• (2) Patients whose clinician did not fully accept the recommendations of the ASPC.

• (3) Patients who received the multidisciplinary consultation, defined as a meeting for collaborative discussions on personalized medical care.

• (4) Patients whose discharge timing was affected by interference from family members.

• (5) Records of patients who had received the ASPC only once and completed on the day of discharge.

The exclusion criteria (1) eliminated the interference of other types of pharmacy consultations. Criteria (2) eliminated the interference of clinicians' subjective opinions. Criteria (3) eliminated an extremely strong confounding predictor that interfered with our study, namely, the multidisciplinary consultation. Patients who received multidisciplinary consultations received too many interventions, which may have obscured the effects of ASPC. Criteria (4) eliminated interference from the patient's family. Criteria (5) eliminated the interference of anomalous ASPC. Due to delays in the ASPC implementation process, an ASPC requested by a patient on the same day is usually completed the next day, and observation of efficacy takes more time.This process is time-consuming and a clinical pharmacist is required to complete multiple ASPC tasks. Therefore ASPC tasks initiated and completed on the day the patient is discharged from the hospital are not reasonably time-consuming in terms of the normal process. These brief ASPC tasks may contain a lot of rushed discussions and procedural jumps that have very little actual impact on the patient and may lead to biased results if included in the study and were therefore excluded.

Sample size, parameters and outcome

According to the extant literature on sample size [24], a data set of 385 to 478 records is recommended to include 6 to 8 predictors. In this study, the data set, which combines 315 records from the First Affiliated Hospital and 159 records from the Second Affiliated Hospital, meets these sample size requirements. Supplementary Table S1, S2 and S3 demonstrates the statistical differences between two hospitals and within themself.

A multitude of decisions are generated during the ASPC process, from its application to its completion, that have varying degrees of impact on the patient's LOS. Guided by the references [11, 25], we have identified three selection criteria for selecting the baseline parameters of the patient and for screening the clinical data generated by the ASPC as potential predictors: (1) Predictors that may influence decision-making for ASPCs, (2) Predictors resulting from clinician or pharmacist decisions, and (3) Predictors that may affect LOS, but are not affected by LOS. Following an internal discussion within our team, the supplementary table S4 presents a detailed breakdown of these predictors.

The Outcome was patient's LOS, which was dichotomized at the median. These LOS below or at the median were considered as low-risk, while those above the median were considered as high-risk. The median was determined separately for each hospital: 21 days for the 315 records from the First Affiliated Hospital and 14 days for the 159 records from the Second Affiliated Hospital.

Process of dealing with the data

The electronic medical records encompass both textual and numerical data, with each record collected and categorized by parameter name. The textual data was classified into multicategorical or binomial formats using established transformation criteria, primarily based on keywords and key phrases. More complex textual data were classified through discussions among our team, including two senior experts in the ASPC. To streamline the data set, some textual and multicategorical data were converted into binomial format based on their frequency and significance. This study hypothesizes that the binomial and multicategorical data represent different directions of decision making. Supplementary Table S5 provides detailed information on both the raw and transformed data for all parameters. If the record contains numerical missing data, said values will be filled in with the average value. Meanwhile, if the record contains binomial missing values, the record will be deleted.

Process of developing the model

Initially, statistical analysis was conducted on the data set to select suitable parameters based on the results. Lasso regression was then applied to further eliminate some parameters, with the remaining parameters being used to construct a multivariate logistic regression model. The model's performance was assessed using three indices and two figures. Discrimination was indicated by the C-index, visually represented by the ROC curve, while calibration was evaluated through the slope and intercept, visually represented by the calibration curve. The model underwent internal validation using tenfold cross-validation, and finally, the model will be displayed via a nomogram.

Statistic analysis and software

The Kolmogorov–Smirnov test was used to assess normality, while Levene's test was employed to evaluate the homogeneity of variances. When both assumptions were met, Analysis of Variance (ANOVA) was utilized for group comparisons, and the Welch test was used when either assumption was not satisfied. Binomial data were analyzed using the Chi-square test, with Fisher's exact test applied when a parameter had an expected count of less than five. Correlation analyses were conducted using the Spearman method, and all tests were conducted with a significance level set at 0.05.Statistical analyses were performed using SPSS Statistics v21, while model development was carried out in R v4.3.3, utilizing principal packages such as "glmnet," "caret," "rms," "pROC," "regplot," and "CalibrationCurves."

Result

Statistical analysis of the data set

Table 1 presents the statistical description and analysis of the data set. Significant differences were observed between the high-risk and low-risk patients in the following parameters (1) age (A), (3) weight (W), (4) ICU admission (ICU), (5) whether the patient received first aid (FA), (6) whether the patient was critically ill (Ci), (7) the aim of the crucial consultation (CCA), and (9) the suggestions from the crucial consultation (CCS). The results indicated that patients with younger age and lower body mass were more prone to prolonged LOS. Furthermore, parameters ICU, FA, and Ci that gauge the severity of the patient's illness exhibited a rational trend, Specifically, the likelihood of prolonged LOS increased in cases of ICU admission, at least one first aid, or critically ill. On the other hand, the ASPC parameter CCA indicated that a minimal number of patients exhibited prolonged LOS when the aim of the ASPC was to request for an initial dosing regimen. Moreover, ASPC parameter CCS demonstrated that patients were more likely to be discharged faster after the clinical pharmacist provided suggestions on adjusting the dosing regimen during the ASPC.

Table 1 Statistical description and analysis of the data set

The ASPC model and its nomogram

Table 2 presents the predictors and performance index of the ASPC model. Figures 1 and 2 show the ROC curve and calibration curve, respectively, under tenfold cross-validation. The ASPC model contains five independent predictors with good discrimination and calibration. The ASPC model demonstrated minimal decline in discrimination when subjected to tenfold cross-validation. The two ROC curves remained largely unchanged, as well as the two calibration curves. These findings suggest the model's stability in predicting risk for patients in the two hospitals within the study area, without indications of overfitting or calibration imbalance.

Table 2 The performence of the ASPC model under tenfold cross validation

Fig. 1

The ROC curves of the ASPC model. Blue line: the ROC curve of model (AUC, area under curve, 0.786 (95% CI, 0.745–0.827). Red line: the ROC curve of model under tenfold cross validation (AUC, 0.772 (95% CI, 0.748–0.797)

Fig. 2

The Calibration curve of the ASPC model. Red line: Ideal line. Black line: Flexible calibration (Loess). Intercept of black line: 0.00 (95% CI −0.12–0.12). slope of black line: 0.94 (95% CI 0.83–1.06)

Figure 3 shows a nomogram of the ASPC model, it can be used by the clinical pharmacist to assess current risk after the patient's initial ASPC is completed. Since the patient has only received one ASPC, it can be assumed that this is the critical one. If the patient has been identified as high-risk, it is suggests that the clinical pharmacist needs to pay more attention to the patient and utilize the in-hospital electronic system to notify all relevant healthcare professionals of the risk. The subsequent ASPCs for these high-risk patients will exhibit enhanced quality due to the risk warning. If the subsequent ASPC is more critical, then the risk can be reassessed using the nomogram to determine if it has decreased or increased.

Fig. 3

The ASPC nomogram identifies high-risk patients by predicting the probability of prolonged length of stay. The box plots and line graphs is indicative of the distribution of the datasets employed in modelling. There is a direct correlation between the total points and the probability ('Pr ()'). The complete correlation is as follows: 150 points correspond to 0.01 probability, and 450 points correspond to 0.92 probability. CCS, the suggestions from the crucial consultation, 0: tend to maintain current dosing regimen, 1: tend to adjust the dosing regimen; W, weight; FA, whether the patient received first aid, 1: no, 0: yes; CCA, the aim of the crucial consultation, 1: request for an initial dosing regimen, 0: request for adjusting current dosing regimen; Ci, whether the patient was critically ill, 1: no, 0: yes. * < 0.05, *** < 0.001. Tip: There is a strong correlation between FA and Ci. When one of them occurs, it should be considered that both occur

Discussion

In this study, four parameters related to antimicrobial stewardship pharmacy consultation (ASPC) were designed to be combined with five patient characteristic parameters to form a candidate parameter combination.Two categories of length of stay (LOS), long and short, divided by median, were also designed as the outcome, with short-term hospitalized patients considered as low-risk and vice versa as high-risk patients. Following a thorough statistical analysis, these parameters exhibiting significant differences were identified as candidate predictors. Moreover, some of the predictors were further eliminated through the lasso regression. Following repeated modeling, the final model incorporated five predictors, demonstrating adequate discrimination and calibration.

The rationale behind selecting pediatric patients as the research subjects is that children belong to a special population in the hospital, along with pregnant individuals and the elderly. The diagnosis, treatment, and medication of pediatric patients present a greater degree of complexity in comparison to adult patients, resulting in an elevated risk of medical errors and disputes. Therefore, the necessity for assistance from clinical pharmacists is apparent. We hypothesized that the impact of ASPC on pediatric patients will be more significant than that on general adult patients, and the expected effect of modeling may be better. Therefore, it can be inferred that the prediction model developed for pediatric patients has higher potential application value.

Among the candidate parameters, age and weight have been demonstrated to be negatively correlated with LOS, a finding that has practical significance. From the perspective of clinical pharmacists, the dosing regimen for younger children often exceeds the provisions of the instructions. In order to determine the dosage and usage, it is necessary to review the pharmacodynamic and pharmacokinetic parameters and the evidence-based literature. Concurrently, the medication may be strictly restricted by adverse reactions and side effects, and it should also be continuously monitored during usage. Therefore, younger patients tend to have more difficulty determining their medication regimen, which is related to the fact that their LOS tend to be longer. In addition, weight is more important than age in children's dosing regimens. Age is eliminated in lasso regression and weight is retained, which confirms this point.

Disease severity and diagnosis have been identified as significant factors affecting LOS. However, this study did not restrict patients to specific infectious disease diagnoses and instead employed three parameters to replace disease severity.While restricting patients to infectious disease diagnoses can enhance research quality, the retrospective data utilized in this study does not satisfy the sample size requirements of the former. Consequently, the Disease Severity Rating Scale is not applicable for data collection. We considered that some medical documents of patients reflect disease severity, such as (4) ICU addmission (ICU), (5) whether the patient received first aid (Ci), and (6) whether the patient was critically ill (FA), so we included these three as candidate parameters.In the process of model development, ICU was eliminated by lasso regression, and Ci and FA successfully entered the model. Given the strong positive correlation between the two (see supplementary table S6), the deletion of either one will result in a decline in model performance. Consequently, we posit that they can be utilized as a combined term to replace disease severity. Therefore, when employing the ASPC nomogram to assess patient risk, it is imperative to combine the occurrence of the two and calculate the score collectively. That is, when one of the two occurs, it is considered that both occur.

Supplementary tables S1, S2, and S3 present the statistical differences in candidate parameters and outcomes between the two hospitals and their internal statistical analysis. Supplementary tables S1 reveals that the patient characteristics are analogous, with the exception of discrepancies in disease severity and LOS. It is postulated that these disparities stem from systematic disparities caused by geographical location and the medical level and status of the hospital. However, as demonstrated in Supplementary tables S2 and S3, the analyses within each of the two hospitals reflected the same trends in the three disease severity parameters. Therefore, the data from these two hospitals were combined, with the primary objective being to meet the necessary sample size.

For the ASPC parameters, following the combination of the data from the two hospitals, the previously insignificant (7) the aim of the crucial consultation (CCA) and (9) the suggestions from the crucial consultation (CCS) exhibited significant differences and were successfully incorporated into the model. These two parameters are of paramount importance in ASPC, as they represent the purpose and results of ASPC, respectively. CCA demonstrates that the involvement of clinical pharmacists in formulating the initial dosing regimen for antibiotics has been demonstrated to more accurately reflect the role of ASPC in reducing patient's LOS than adjusting the current dosing regimen. This is due to the fact that infectious diseases manifest rapidly, and if the initial dosing regimen can be full discussion, it will aid in the management of the infection in the early stages and will significantly improve patient prognosis. When the purpose of ASPC application is to adjust the medication, it signifies that the patient's current dosing regimen is no longer capable of effectively managing the infection. However, due to the pharmacological and pharmacokinetic properties of antimicrobial drugs, which typically preclude immediate dosing regimen modification, potentially prolonging the patient's illness duration and necessitating prolonged LOS.

On the other side, CCS demonstrates that the clinical pharmacist proposes an adjustment to the dosing regimen, it can be deduced that the dosing regimen has the potential to be optimized to manage the infection, thereby signifying the efficacy of the ASPC. Conversely, when the recommendation is to maintain the current dosing regimen, whether it is not recommended to use antimicrobial drugs or the current dosing regimen cannot be changed immediately, it implies that the clinical pharmacist has not implemented effective intervention on the patient's current status and needs to continue to observe, thus increasing the risk of prolonged LOS. The prediction trends of these two predictors well confirmed our research hypothesis, making the ASPC model reflect a certain degree of application value.

The ASPC model developed in this study is the first prognostic model developed for clinical pharmacists for patients with paediatric infectious diseases. It can effectively identify high-risk patients who may require prolonged LOS, filling a gap in ASPC-related research. A study developed an ASPC model similar to our model [26]. In terms of predictors, the study used the number of basic diseases to replace the severity of the disease. However, this method is insufficient when compared with our study. The study incorporated two ASPC parameters: the pharmacist's professional title and the acceptance of consultation suggestions. The former is posited as a potentially valuable ASPC parameter, while the latter is deemed unsuitable for inclusion. If clinicians opt not to adopt the ASPC's recommendations, the potential impact of the ASPC is rendered null, resulting in biased outcomes. To maintain the integrity of the research findings, the exclusion of these samples is recommended.

In summary, the ASPC nomogram developed in this study has the potential to contribute to the advancement of individualized medicine, the enhancement of ASPC quality, and the promotion of the redistribution of medical resources. From a macro perspective, it is possible to reduce the waste of medical resources, decrease medical costs, and improve medical efficiency. These potential objectives are consistent with China's current medical and health policies. In subsequent studies, priority will be given to the expansion of the number of ASPC parameters and the conduction of external validation in new populations within our hospital and other hospitals. This will allow for the exploration of the stability and generalizability of the ASPC model.Concurrently, the sample size will need to be expanded to facilitate the collection of more detailed and rigorous patient characteristic parameters and restrict disease diagnosis. This will assist in the reduction of sample bias and the impact of confounding factors. Concurrently, we intend to undertake additional studies to assess the optimal duration of hospital stay and bolster the rationality of the ASPC model.Ultimately, we aim to conduct prospective studies to enhance the quality of the ASPC model.

Limitations

First, the ASPC model developed in this study is subject to certain biases and confounding factors. To mitigate these issues, we merged the data from two hospitals to meet the necessary sample size requirements. This approach also reduced the risk of overfitting. Secondly, the median of each of the two hospitals was directly used as the reasonable LOS, which may overestimate the risk; however, most of the studies referenced in the preface also used the median or mean of the LOS to describe the impact of pharmacist-led intervention, so it is also reasonable.In future studies, the determination of a reasonable length of stay and the enhancement of the credibility of the ASPC model prediction value will be explored. Finally, as this study was the first to explore the effect of ASPC parameters on LOS, the number of parameters included was limited, and their design oversimplified the information collected. Therefore, further exploration of additional parameters, along with a redesign of existing ones, is necessary to obtain more comprehensive data that can better elucidate the impact of ASPC on patient LOS.

Conclusions

The antimicrobial stewardship pharmacy consultation (ASPC) model has good discrimination and calibration, and effectively identifies patients at risk for prolonged length of stay (LOS). The ASPC model highlights the impact of the purpose and outcome of ASPC on the patient's LOS. Identifying high-risk patients through the ASPC nomogram can help strengthen intervention for patients and improve the quality of subsequent ASPC.