Validation of a machine learning model for predicting early deterioration in the emergency department.

doi:10.1016/j.ajem.2026.05.007

What this article is about

In brief

This 2026 study validates two XGBoost ML models using structured ED data and BioClinicalBERT embeddings from nursing notes to predict early deterioration (ICU admission/death within 7 days). Model B, prioritizing high-risk patients via class weighting, showed improved recall (0.

For nursing students

Study summary

This article, published in The American Journal of Emergency Medicine on January 1st, 2026 (DOI: 10.1016/j.ajem.2026.05.007), explores a critical issue in emergency department (ED) care: the early identification of patients at risk for deterioration. The study focuses on developing and evaluating machine learning models to predict such deterioration before an initial physician assessment, aiming to create a tool that can help prioritize patient care effectively.

The research problem is framed around the limitations of traditional early warning scores in dynamic ED environments. These scores often rely solely on structured triage data (like vital signs) and may not capture the full clinical picture conveyed by nursing assessments or free-text notes, leading to poor performance. The authors propose that integrating natural language processing with machine learning could offer a more nuanced approach.

The study analyzed 17,481 consecutive adult ED visits over six months. This large dataset allowed for robust model training and evaluation. Structured variables such as demographics (age), vital signs (respiratory rate, systolic blood pressure), and eCTAS scores were combined with BioClinicalBERT-derived embeddings from free-text nursing triage notes to form a multimodal feature representation. Two XGBoost models were developed: Model A used standard class weighting for the binary classification task of predicting 'early deterioration' (defined as ICU admission or death within 7 days, which had a prevalence of 4.5% in this dataset) versus all other outcomes. Model B applied increased weighting to the early deterioration class specifically to prioritize identifying high-risk patients.

The findings are quite significant for understanding how these models perform. Model A achieved a recall (sensitivity) of 0.66, meaning it correctly identified 66% of actual deteriorating cases; its precision was 0.17, indicating that only 17% of the cases flagged by this model as high-risk were true positives; and the ROC-AUC (Area Under the Receiver Operating Characteristic Curve) was 0.75. Model B showed improvements: recall increased to 0.77, precision to 0.22, and ROC-AUC to 0.90. These metrics suggest that while both models have room for improvement, especially in terms of precision (many false positives), they are better at catching true deteriorating cases than not.

The authors highlight an important point about feature importance: although XGBoost's internal analysis attributed the majority of predictive weight to the free-text embeddings from nursing notes, SHAP (SHapley Additive exPlanations) analysis identified age, respiratory rate, and systolic blood pressure as the dominant individual contributors. This means that while incorporating text data adds value beyond just structured variables alone, these specific vital signs remain crucial predictors.

For students appraising this research, it's essential to consider several aspects. First, the study is a validation of machine learning models in an ED setting, which has direct implications for nursing practice and patient safety. The use of free-text notes is particularly relevant as nurses are often responsible for documenting these initial assessments. However, students should also critically evaluate the limitations mentioned by the authors: the primary outcome was defined retrospectively (ICU admission or death within 7 days), models were trained on data from a single institution over six months, and while they show promise, safe clinical adoption requires prospective shadow testing in real-time workflows to assess operational feasibility and impact on decision-making. The study does not claim these tools should replace clinical judgment but rather function as an adjunct layer of situational awareness.

A nurse would reason from this evidence by understanding that AI-driven risk prioritization tools could help identify patients who might otherwise be missed by traditional scores, potentially leading to earlier interventions and improved outcomes. However, nurses must remember these are adjuncts; their clinical expertise remains paramount for interpreting the model's outputs and making final care decisions. The study suggests a future where such technology supports nursing vigilance in the fast-paced ED environment.

In summary, this research demonstrates promising steps towards using machine learning to enhance early deterioration detection in emergency departments by leveraging both structured data and free-text nursing notes. While not perfect, these models offer potential for improved patient safety when integrated thoughtfully into clinical workflows.

Original publication

Source abstract and study details

Read the source abstract

Early recognition of patients at risk for deterioration in the emergency department (ED) is critical for patient safety. Traditional early warning scores rely on structured triage data and often perform poorly in the dynamic ED environment. We developed and evaluated two machine learning models integrating structured triage data with transformer-based embeddings of free-text nursing triage notes to predict early clinical deterioration prior to initial physician assessment, designed as a risk-based prioritization tool to rank patients by predicted probability of adverse outcome. We analyzed 17,481 consecutive adult ED visits over six months. Structured variables (demographics, vital signs, eCTAS scores) were combined with BioClinicalBERT-derived embeddings from free-text nursing triage notes to form a multimodal feature representation. Two XGBoost models (A, B) were trained on the same binary classification task, predicting "early deterioration" (ICU admission or death within 7 days, prevalence 4.5%) versus all other outcomes, differing only in class weighting. Model A used standard class weighting; Model B applied increased weighting to the early deterioration class to prioritize identification of high-risk patients. Model A achieved a recall of 0.66 (95% CI: 0.59-0.73), precision of 0.17 (95% CI: 0.15-0.20), and ROC-AUC of 0.75 (95% CI: 0.72-0.79). Model B improved recall to 0.77 (95% CI: 0.72-0.84), precision to 0.22 (95% CI: 0.19-0.25), and ROC-AUC to 0.90 (95% CI: 0.88-0.92). While XGBoost's internal feature importance attributed the majority of predictive weight to free-text embeddings, SHAP analysis identified age, respiratory rate, and systolic blood pressure as the dominant individual contributors, with triage note embeddings providing meaningful incremental value confirmed by structured-variable ablation. These findings suggest that AI-driven risk prioritization may function as an adjunct layer of situational awareness in the ED, complementing clinical judgement rather than replacing it. Safe clinical adoption will require prospective shadow testing in real-time workflows to quantify ranking accuracy, assess operational feasibility, and evaluate impact on decision-making before any clinician-facing implementation.

Source type: Journal Article Validation Study

Reviewed findings

Main findings

Two XGBoost models were developed using structured ED triage data (demographics, vital signs, eCTAS) combined with BioClinicalBERT embeddings from free-text nursing notes.
Model A used standard class weighting; Model B applied increased weighting to the 'early deterioration' class (ICU admission or death within 7 days).
The prevalence of early deterioration in the study cohort was 4.5%.
Model B showed improved recall (0.77 vs. 0.66 for Model A) and precision (0.22 vs. 0.17 for Model A), with a higher ROC-AUC (0.90 vs. 0.75).
While free-text embeddings contributed significantly to predictive power, SHAP analysis identified age, respiratory rate, and systolic blood pressure as dominant individual predictors.

For teaching and learning

Education implications

AI-driven risk prioritization tools could serve as an adjunct layer of situational awareness in the ED.
Such models may help identify patients at high risk for deterioration earlier than traditional scores alone.
Incorporating free-text nursing notes into predictive analytics can add value beyond structured data, highlighting their importance.
The findings suggest a potential pathway to improve patient safety through early intervention by flagging deteriorating patients sooner.
Prospective shadow testing in real-time ED workflows is crucial before any clinician-facing implementation of such models.

For educators

Teaching and appraisal notes

This article presents a validation study of two XGBoost-based machine learning models designed to predict early clinical deterioration (defined as ICU admission or death within 7 days) in the emergency department (ED), prior to initial physician assessment. The primary aim is to develop an AI-driven risk prioritization tool that integrates structured triage data with transformer-based embeddings from free-text nursing triage notes, aiming to improve upon traditional early warning scores which often perform poorly in dynamic ED environments.

The study analyzed 17,481 consecutive adult ED visits over a six-month period. A multimodal feature representation was created by combining structured variables (demographics, vital signs like age, respiratory rate, systolic blood pressure, and eCTAS scores) with BioClinicalBERT-derived embeddings from free-text nursing triage notes. Two models were trained: Model A used standard class weighting for the binary classification task, while Model B applied increased weighting to the 'early deterioration' class (prevalence 4.5%) to prioritize identification of high-risk patients.

The key performance metrics are as follows: * **Model A:** Recall = 0.66 (95% CI: 0.59-0.73), Precision = 0.17 (95% CI: 0.15-0.20), ROC-AUC = 0.75 (95% CI: 0.72-0.79). * **Model B:** Recall = 0.77 (95% CI: 0.72-0.84), Precision = 0.22 (95% CI: 0.19-0.25), ROC-AUC = 0.90 (95% CI: 0.88-0.92).

These results indicate that Model B, with its class weighting strategy to prioritize high-risk patients, demonstrates superior performance in terms of recall and precision compared to Model A. The ROC-AUC for both models suggests good discriminatory ability. The study highlights a nuanced finding regarding feature importance: while XGBoost's internal analysis indicated free-text embeddings contributed significantly to predictive power (suggesting the value of nursing notes), SHAP analysis pinpointed age, respiratory rate, and systolic blood pressure as the dominant individual contributors. This implies that these vital signs are strong predictors even when combined with text data.

The authors frame their findings cautiously, stating that such AI-driven tools should function as an adjunct layer of situational awareness in the ED, complementing rather than replacing clinical judgment. They emphasize that safe clinical adoption requires prospective shadow testing in real-time workflows to quantify ranking accuracy, assess operational feasibility, and evaluate impact on decision-making before any clinician-facing implementation.

For nursing education and practice, this study underscores the potential of AI/ML tools in enhancing patient safety by identifying at-risk patients earlier. It highlights the importance of nursing documentation (free-text notes) as a valuable data source for such predictive models. However, it also stresses that these technologies are not standalone solutions; they require careful integration into existing workflows and must be validated rigorously before clinical deployment to ensure they do not introduce new risks or biases.

The paper's strengths include its use of a large dataset from an ED setting, the innovative combination of structured data with NLP-derived features (BioClinicalBERT), and clear reporting of model performance metrics. Limitations acknowledged by the authors include retrospective outcome definition, single-institution training data, and the need for prospective validation.

This research contributes to the growing body of literature on clinical decision support systems in emergency care and highlights an area where nursing practice can intersect with advanced technology to improve patient outcomes.

Critical appraisal

Limitations

Retrospectively defined primary outcome (ICU admission or death within 7 days).
Models trained on data from a single institution over six months, limiting generalizability without external validation.
Need for prospective shadow testing to assess operational feasibility and impact on decision-making before clinical adoption.

Classroom use

Discussion Questions

How might the integration of AI-driven risk prioritization tools affect nurse-physician communication in the ED?
What are the potential ethical considerations if these models have biases or errors, particularly concerning patient subgroups not well-represented in the training data?
Can you think of other types of free-text clinical notes (e.g., progress notes) that might be valuable for similar predictive modeling tasks?
How would you design a study to prospectively validate such an AI tool in a real-world ED setting, including how to measure its impact on patient outcomes and workflow efficiency?
What specific training or education would nurses need to effectively use and interpret outputs from these types of machine learning models?
Discussion question 6: What does "Validation of a machine learning model for predicting early deterioration in the emergency department." help nursing students evaluate?

Source-based questions

Frequently asked questions

What are the names of the two XGBoost models developed in this study?

Two XGBoost models were developed: Model A and Model B. They differed only in their class weighting strategy for handling the imbalanced dataset (where early deterioration was a minority outcome).