The study assesses the real-world performance of a FDA approved AI tool for pneumothorax (PTx) detection on ICU chest radiographs (CXRs) and its utility in triaging scans on PACS, especially during on call hours, by assessing improvement in radiologist reporting times.

This retrospective study involved 27,399 consecutive frontal CXRs collected from 08/2020 to 04/2021 [12,728 CXRs scanned using AI tool (TotalAI) and 14,761 on conventional scanners without AI (Controls)]. Final radiologist report was taken as ground truth. Pneumothorax location and size, priority status as per the clinical order (STAT vs routine), reporting time (time of first dictation - time of study appearance on PACS), were also obtained for all CXRs. Scans flagged positive by the AI (irrespective of ordering priority), were also routed into the resident on-call list (5:00 PM to 8:00 AM). Wilcoxon rank sum test and Receiver operator characteristic (ROC) analysis were used to assess differences between the groups and AI tool performance, respectively. Visualization was achieved using Box and whisker, and violin plots.

1402 CXRs flagged positive for PTx (Ptx posAI) and 11,326 were negative for PTx (Ptx neg AI) by AI. Area under ROC curve (AUC) for the AI tool was 78% (p<0.0001) with sensitivity of 60.4% and specificity of 96.6%. When selecting for CPTx, AUC and sensitivity increased to 92% (p<0.0001), and 89% with specificity of 96.6%. In studies with confirmed PTx, reporting time for Ptx posAI group was less than half as compared to Controls group, (Median - 104 vs 260 minutes, p < 0.0001). CXRs with STAT+ Ptx posAI showed improved reporting time as well, when compared with STAT+ Controls group (Median 44 vs 202 minutes, p<0.0001). Subgroup analysis during on-call hours showed significant improvement in reporting time between Ptx posAI and routine studies (Median 148 vs 345 mins; p<0.0001) in the CPTx group, with 107 true positive “routine” CPTx diagnosed

Real world deployment of given AI tool accurately diagnosepneumothoracebut more importantly improvereporting times, especially during call hourwhere “routine” studieare not read by the on-call resident unlesthey are flagged aSTAT by provider or AI (Ptx posAI).

This study validated the real-world performance of an AI tool and showed how its implementation can directly improve patient care by allowing the radiologist to provide timely and actionable input in clinical management. As a result of AI-augmented workflow, routine studies with CPTx were accurately flagged by AI and received urgent clinicalinterventions.