BACKGROUNDThe Advanced Alert Monitor (AAM) is a data-driven Early Warning Score (EWS) system designed to predict ICU admission or mortality within 12 hours. It has demonstrated superior performance compared to the National Early Warning Score (NEWS) and contributed to a consortium-wide reduction in mortality, showcasing its potential for clinical use.OBJECTIVEThe study evaluates the generalizability of the AAM in a Dutch hospital setting, comparing its performance against the NEWS and a locally optimized AAM (LO-AAM). Additionally, it investigates how different outcome definitions influence system performance.METHODElectronic medical record (EMR) data from Catharina Hospital in Eindhoven, Netherlands, were used to reproduce the AAM and train the LO-AAM. Both were evaluated against the NEWS using two outcome definitions: the original study definition and an adapted definition that includes mortality regardless of care order status. Feature importance analysis was conducted to assess the impact of these outcome definitions on model performance.RESULTSThe AAM achieved an AUROC of 79.9% against NEWS's 74.2%. However, under the modified outcome definition, the NEWS outperformed the AAM. LO-AAM outperformed both AAM and NEWS in all outcomes. Feature importance analysis showed a greater emphasis on physiological features for the LO-AAM trained on the adapted outcome.CONCLUSIONThe AAM demonstrates generalizability beyond its original population, but local optimization significantly enhances its performance. Outcome definitions critically affect the performance of the NEWS, AAM, and LO-AAM. The adapted outcome includes a wider scope of mortality as an adverse event, leading to an increase in performance for all EWSs.