|Year : 2018 | Volume
| Issue : 2 | Page : 252-256
Failure to validate a prediction model for risk of complications in pediatric cancer patients with fever and neutropenia
Osama K Mobaireek1, Nesrin A Al-Harthy2, Abdulrhman M Alnasser1, Jinan R Al-Rashoud1, Mohammed H Hommady1, Winnie Philip3, Shoeb Qureshi3
1 College of Medicine, King Saud Bin Abdulaziz University for Health Sciences, National Guard Health Affairs, Riyadh, Saudi Arabia
2 Department of Paediatric Emergency, King Abdullah Specialist Children Hospital and College of Applied Medical Sciences (M&F), Riyadh, Saudi Arabia
3 Department of Research, College of Applied Medical Sciences, Riyadh, Saudi Arabia
|Date of Web Publication||20-Jun-2018|
Research Unit, College of Applied Medical Sciences, King Saud Bin Abdulaziz University for Health Sciences, National Guard Health Affairs, PO Box 3660, Riyadh 11481
Source of Support: None, Conflict of Interest: None
| Abstract|| |
Objectives: The aim is to clinically validate prediction model for risk of complications in pediatric cancer patients with febrile neutropenia (FN). Materials and Methods: A risk prediction model for pediatric patients with FN is previously published (Hakim et al. model). In this retrospective, cross-sectional study, we validated the model's prediction of proven invasive infection and culture-negative sepsis on 108 patients, aged 14 years or younger, who developed FN in outpatient settings, and were admitted to King Abdulaziz Medical City in Riyadh, Saudi Arabia between January 1, 2006 and December 31, 2015. Patients' charts were reviewed, and the predicted versus observed outcomes were recorded. For the clinical prediction model validation, odds ratios, sensitivity, specificity, positive predictive values (PPV), and negative predictive values (NPV) were calculated. Results: Patients' ages ranged from 4 months to 13.5 years (median: 5.7 years). Sixty-two percent of patients had acute lymphoblastic leukemia or lymphoma, 33% had solid tumors, and 5% had acute myeloid leukemia. Clinical complications were identified in 38% of patients including positive cultures in 7%. In our population, the model's sensitivity was 27%, specificity was 75%, PPV was 39%, and NPV was 63%. Univariate analysis showed no statistically significant correlation between outcome and gender, clinical appearance, temperature, absolute neutrophil count, or cancer classification. Conclusion: Risk of FN complications could not be reliably predicted in our population using the previously published risk prediction model (Hakim et al. model). Moreover, none of the clinical variables could reliably classify patients. Hence, clinical practice needs to treat all FN patients with caution of high-risk complications.
Keywords: Cancer, chemotherapy, sepsis
|How to cite this article:|
Mobaireek OK, Al-Harthy NA, Alnasser AM, Al-Rashoud JR, Hommady MH, Philip W, Qureshi S. Failure to validate a prediction model for risk of complications in pediatric cancer patients with fever and neutropenia. J Nat Sc Biol Med 2018;9:252-6
|How to cite this URL:|
Mobaireek OK, Al-Harthy NA, Alnasser AM, Al-Rashoud JR, Hommady MH, Philip W, Qureshi S. Failure to validate a prediction model for risk of complications in pediatric cancer patients with fever and neutropenia. J Nat Sc Biol Med [serial online] 2018 [cited 2020 May 30];9:252-6. Available from: http://www.jnsbm.org/text.asp?2018/9/2/252/234717
| Introduction|| |
Febrile neutropenia (FN) is a common complication in children who receive myelotoxic therapy, potentially indicating a life-threatening infection that may necessitate hospitalizations and affect management outcome. FN is the leading cause of emergency department presentation among children with cancer. FN is defined as an oral temperature ≥38.3°C, or ≥38°C for >1 h, with an absolute neutrophil count (ANC) ≤500 cells/mm 3, or expected to fall below 500 in 48 h.
The standard care in FN is immediate administration of broad-spectrum intravenous antibiotics. Standard protocols and guidelines are developed that attempt to ensure the prompt administration of antibiotics, with a “door-to-needle” time of 1 h maximum., The rationale behind this is preventing the progression of sepsis, which is a time-dependent emergency, where urgent management may have beneficial effects on outcome., Not all patients with FN are at equal risk of developing a life-threatening infection. For this reason, multiple predictive models assess risk on presentation to the emergency department.,,,, Such models are increasingly used with adults; however, few pediatric models are validated in independent populations. Depending on the risk assessment, the recommended course of management can change from inpatient treatment with intravenous antibiotics to outpatient treatment with oral antibiotics.,
The clinical predictors for high infection risk, proposed in previous studies, vary across populations and geographical locations, and no single set of criteria is in widespread use across independent populations. These clinical predictors included the presence of acute myeloid leukemia,, presence of acute lymphoid leukemia,,, vomiting, altered mental status, toxic clinical appearance, hypotension,, low neutrophil count,, low platelet count, and new infiltrates on chest X-ray.,
Previous local FN studies target adult populations and focus on the microbiology of FN,,, susceptibility patterns, the diseases' natural history, and comparison of FN patterns among different malignancies. There are no previous local studies on risk assessment of FN patients. The purpose of this study was to validate a predictive model, developed by Hakim et al., for assessing the risk of invasive bacterial infection and culture-negative sepsis in pediatric patients presenting with FN. We also sought to assess the variability of clinical prediction, across genders and age groups, in pediatric cancer patients with fever, and neutropenia at increased risk of proven invasive bacterial infection or clinical sepsis. Among the recently proposed models, this model has one of the largest sample sizes, is clinically usable and addresses many methodological standards, but lacks independent validation. The validation consists of a retrospective evaluation of patients who present at the Pediatric Emergency Department of King Abdulaziz Medical City, Riyadh, Saudi Arabia.
| Materials and Methods|| |
This study was approved by King Abdullah International Medical Research Center's Institutional Review Board, and data collection and analysis started after receiving the approval. Only patient previous records were reviewed. No patients were identified.
A risk prediction model for pediatric patients with FN was derived from 332 patients at Jude Children's Research Hospital (St. Jude), Memphis, TN, United States of America, and published in 2010 by Hakim et al. The model aimed at predicting two outcomes: “proven invasive bacterial infection or culture-negative sepsis” (inflammatory bowel disease [IBD]) and “clinical complications” (CC). In this retrospective, cross-sectional study, we validated the model's IBD outcome prediction on 463 patients, aged 14 years or younger, who developed FN in outpatient settings, and were admitted to King Abdulaziz Medical City in Riyadh, Saudi Arabia between January 1, 2006 and December 31, 2015. The median number of FN episodes per patient was 1 (range: 1–19). We excluded 345 patients according to exclusion criteria used by Hakim et al., which included: (a) patients who had already received a stem cell transplant, or (b) those who developed FN during hospitalization for other reasons. After exclusions, 118 patients met our inclusion criteria. Ten of those who met the inclusion criteria were excluded due to incomplete documentation to fulfill the prediction model [Figure 1]. In patients with >1 FN episode, a computerized program randomly selected one episode.
Patients' charts (including emergency department notes, progress notes, and discharge notes) were used to review each episode, as per the prediction model, which consisted of multiple variables. These included underlying cancer diagnosis, clinical appearance, temperature at presentation, and absolute neutrophil count. Each variable was assigned a score by the prediction model. Patients with a total score of 24 or more were predicted to be at “high-risk.” We use the charts to extract patient demographic data and document observed outcomes in terms of documented positive cultures or signs of clinical sepsis, as per the criteria provided by the international pediatric sepsis consensus conference.
We used descriptive statistics, including absolute and relative frequencies, means, and standard deviations. For the clinical prediction model validation, odds ratios, sensitivity, specificity, positive predictive values (PPV), and negative predictive values (NPV) were calculated. We provided 95% confidence intervals, and all statistical tests were considered significant at P < 0.05. All analyses were done using SPSS version 22 (IBM SPSS, Armonk, NY, USA).
| Results|| |
A total of 108 patients were included in this study. Patients' ages ranged from 4 months to 13.5 years (median: 5.7 years). The majority (63%) of patients were males. In terms of cancer types, 62% had acute lymphoblastic leukemia or lymphoma, 33% had solid tumors, and 5% had acute myeloid leukemia [Figure 2]. CCs were identified in 38% of patients including positive cultures in 7%. Comparing our data to Hakim's study, the sensitivity was 27% and 75%, the specificity was 75% and 77%, PPV was 39% and 36%, and NPV was 63% and 95%, respectively.
We tested the sensitivity and specificity at alternative cutoff scores (20, 21, 22, and 23). None showed clinically-useful improvement [Table 1]. Univariate analysis was performed to explore the prediction model variables, and the receiver operating characteristic curve was plotted [Figure 3]. Sex (P = 0.456), clinical appearance (P = 0.314), ANC (P = 0.784), underlying cancer category (P = 0.301), and temperature (P = 0.468) were not associated with FN. Because there were no statistically significant predictors from univariate analyses, multivariate analyses were not conducted.
|Figure 3: Receiver operating characteristic (ROC) curve, AUC (95% CI) = 0.528 (0.417-0.638)|
Click here to view
| Discussion|| |
We validated, in an independent population, a prediction model for invasive bacterial infection and culture-negative sepsis, published by Hakim et al. This model, unlike other models, focuses on the pediatrics. This study uses a similar inclusion and exclusion criteria to that of the original study. When the baseline characteristics of the populations of the two studies were compared [Table 2], clinical appearance and cancer type were similar; however, our population had more males, and we followed the local protocol of defining the pediatric population as those patients aged up to 14 years. Local validation of risk prediction models is essential before implementation since results in one population do not guarantee similar results in another population where different environmental or genetic factors may be involved. Moreover, risk prediction models of FN complications better inform patients and their families. This is especially needed when models can modify the management course. For example, patients with low-risk FN can be treated with oral (rather than intravenous) antibiotics as outpatients. This would help reduce the mortality and morbidity in the high-risk group while avoiding excessive treatment in patients with lower risk. Moreover, this is likely to save costs, optimize resource distribution, and minimize the risk of developing antibiotic resistance.
|Table 2: Comparison of baseline characteristics of patients at the time of presentation|
Click here to view
The higher proportion of observed poor outcomes in our population was overwhelmingly due to clinical signs of sepsis, rather than proven invasive bacterial infection. In fact, our rate of positive cultures was lower than what is reported in the literature. Previous studies showed that the prevalence of positive cultures among FN patients was 25%,,, but in our study, it was only 7%. It is our judgment that this is likely to be due to our hospital's strict protocol of immediate administration of intravenous antibiotics for all FN patients.
No individual predictor suggested by the prediction model (i.e., clinical appearance, temperature, ANC, and underlying cancer diagnosis) correlated with FN complications, so we were unable to identify a different subset of variables that distinguish high-risk patients in our population.
The high NPV observed in our dataset, relative to PPV (63% vs. 39%), is similar to the result seen by Hakim et al. (95% NPV and 36% PPV) and is common in many risk prediction model studies, reflecting the ratio of patients without complications. Higher NPV indicates that a prediction model is selective to patient group where prophylaxis can be safely omitted. Despite that and given that the observed NPV was not nearly as high as in the original study, in addition to the low sensitivity of this prediction model, this model would not seem to modify the management of pediatric patients, at least for the time being. As prediction models fall short of guiding medical practice, individualized clinical judgment becomes even more crucial for patient care.
This marked decline in performance is not uncommon during external validation. Over 59% of prediction models underperform during external validation.
Several reasons can explain the large difference in results between our study and Hakim's. Our population was entirely independent from Hakim's, which can raise the possibility that unknown variables were not accounted for by Hakim's model (of note, the chemotherapy regimen was not incorporated in Hakim's model). Second, the validation that was applied by Hakim et al. was based on bootstrap analysis rather than an external validation, which affirms the crucial role the later plays at identifying generalizability of prediction models.
This study had some limitations. The small patient population from within a single hospital could potentially limit generalizability. In addition, as a retrospective study, we had to rely on patients' charts, as they were, but to increase reliability, we also incorporated data from the electronic medical records, especially with regard to automated laboratory results. In addition, there was no objective definition for clinical appearance nor culture-negative clinical sepsis. In both cases, we had to rely on the documented appearances and outcomes, which are based on the subjective clinical judgments. We suggest using a better-defined risk predictor and outcome. This might be a major cause of variability in results. Risk prediction models are meant to provide clearer guidance for clinical practice. It is therefore suggested that further, comparative studies are conducted to evaluate the effect of introducing risk prediction models for assisting pediatric physicians in the Saudi health-care system promote evidence-based medicine and enhancing patient outcomes.
| Conclusion|| |
We have provided the first external validation of a previously published, pediatric-specific, risk prediction model for FN patients. The model could not reliably predict poor outcomes in our population. Moreover, we have concluded that sex, clinical appearance, temperature, ANC, and cancer classification could not reliably predict poor outcomes in our population. Further development and validation of prediction models for FN in pediatric patients with cancer is therefore needed.
We would like to thank Dr. Reem Al-Sudairy for her expert opinions on the subject. This research was supported by King Abdullah International Medical Research Center.
Financial support and sponsorship
Conflicts of interest
There are no conflicts of interest.
| References|| |
Mueller EL, Sabbatini A, Gebremariam A, Mody R, Sung L, Macy ML, et al.
Why pediatric patients with cancer visit the emergency department: United States, 2006-2010. Pediatr Blood Cancer 2015;62:490-5.
Freifeld AG, Bow EJ, Sepkowitz KA, Boeckh MJ, Ito JI, Mullen CA, et al
. Executive summary: Clinical practice guideline for the use of antimicrobial agents in neutropenic patients with cancer: 2010 Update by the Infectious Diseases Society of America. Clin Infect Dis 2011;52:427-31.
Clarke RT, Warnick J, Stretton K, Littlewood TJ. Improving the immediate management of neutropenic sepsis in the UK: Lessons from a national audit: A UK Audit of Neutropenic Sepsis Management. Br J Haematol 2011;153:773-9.
Dellinger RP, Levy MM, Rhodes A, Annane D, Gerlach H, Opal SM, et al.
Surviving sepsis campaign: International guidelines for management of severe sepsis and septic shock, 2012. Intensive Care Med 2013;39:165-228.
Gaieski DF, Mikkelsen ME, Band RA, Pines JM, Massone R, Furia FF, et al.
Impact of time to antibiotics on survival in patients with severe sepsis or septic shock in whom early goal-directed therapy was initiated in the emergency department. Crit Care Med 2010;38:1045-53.
Haeusler GM, Carlesse F, Phillips RS. An updated systematic review and meta-analysis of the predictive value of serum biomarkers in the assessment of fever during neutropenia in children with cancer. Pediatr Infect Dis J 2013;32:e390-6.
Phillips RS, Lehrnbecher T, Alexander S, Sung L. Updated systematic review and meta-analysis of the performance of risk prediction rules in children and young people with febrile neutropenia. PLoS One 2012;7:e38300.
Alexander SW, Wade KC, Hibberd PL, Parsons SK. Evaluation of risk prediction criteria for episodes of febrile neutropenia in children with cancer. J Pediatr Hematol Oncol 2002;24:38-42.
Mullen CA, Petropoulos D, Roberts WM, Rytting M, Zipf T, Chan KW, et al.
Outpatient treatment of fever and neutropenia for low risk pediatric cancer patients. Cancer 1999;86:126-34.
Hakim H, Flynn PM, Srivastava DK, Knapp KM, Li C, Okuma J, et al.
Risk prediction in pediatric cancer patients with fever and neutropenia. Pediatr Infect Dis J 2010;29:53-9.
Teuffel O, Sung L. Advances in management of low-risk febrile neutropenia. Curr Opin Pediatr 2012;24:40-5.
Paganini HR, Sarkis CM, De Martino MG, Zubizarreta PA, Casimir L, Fernandez C, et al.
Oral administration of cefixime to lower risk febrile neutropenic children with cancer. Cancer 2000;88:2848-52.
Petrilli A, Altruda Carlesse F, Alberto Pires Pereira C. Oral gatifloxacin in the outpatient treatment of children with cancer fever and neutropenia. Pediatr Blood Cancer 2007;49:682-6.
al-Fawaz IM, Kambal AM, al-Rabeeah AA, al-Rasheed SA, al-Eissa YA, Familusi JB, et al.
Septicaemia in febrile neutropenic children with cancer in Saudi Arabia. J Hosp Infect 1991;18:307-12.
Madani TA. Clinical infections and bloodstream isolates associated with fever in patients undergoing chemotherapy for acute myeloid leukemia. Infection 2000;28:367-73.
Sirkhazi M, Sarriff A, Aziz NA, Almana F, Arafat O, Shorman M, et al.
Bacterial spectrum, isolation sites and susceptibility patterns of pathogens in adult febrile neutropenic cancer patients at a specialist hospital in Saudi Arabia. World J Oncol 2014;5:196-203.
Al-Ahwal MS. Pattern of febrile neutropenia in solid tumors – A hospital based study. Pak J Med Sci 2005;21:249-52.
Al-Ahwal MS, Al-Sayws F, Johar I. Febrile neutropenia comparison between solid tumours and hematological malignancies. Pan Arab Medical Journal. 2005;2:4-7.
Delebarre M, Macher E, Mazingue F, Martinot A, Dubos F. Which decision rules meet methodological standards in children with febrile neutropenia? Results of a systematic review and analysis. Pediatr Blood Cancer 2014;61:1786-91.
Goldstein B, Giroir B, Randolph A, International Consensus Conference on Pediatric Sepsis. International pediatric sepsis consensus conference: Definitions for sepsis and organ dysfunction in pediatrics. Pediatr Crit Care Med 2005;6:2-8.
Rosenblum J, Lin J, Kim M, Levy AS. Repeating blood cultures in neutropenic children with persistent fevers when the initial blood culture is negative. Pediatr Blood Cancer 2013;60:923-7.
Nesher L, Rolston KV. The current spectrum of infection in cancer patients with chemotherapy related neutropenia. Infection 2014;42:5-13.
Kanamaru A, Tatsumi Y. Microbiological data for patients with febrile neutropenia. Clin Infect Dis 2004;39 Suppl 1:S7-10.
Siontis GC, Tzoulaki I, Castaldi PJ, Ioannidis JP. External validation of new risk prediction models is infrequent and reveals worse prognostic discrimination. J Clin Epidemiol 2015;68:25-34.
Moons KG, Kengne AP, Grobbee DE, Royston P, Vergouwe Y, Altman DG, et al.
Risk prediction models: II. External validation, model updating, and impact assessment. Heart 2012;98:691-8.
[Figure 1], [Figure 2], [Figure 3]
[Table 1], [Table 2]