J Med Internet Res. 2026 Jul 8;28:e29701. doi: 10.2196/29701.
ABSTRACT
BACKGROUND: Large language models (LLMs) are increasingly being explored for clinical decision support. However, whether inference-only LLM outputs can be interpreted as reliable quantitative risk estimates in structured clinical prediction remains unclear.
OBJECTIVE: This study aimed to evaluate the predictive performance and decision-making behavior of inference-only LLMs in a structured clinical prediction task and compare their outputs with those of an outcome-trained machine learning model.
METHODS: We conducted a controlled benchmarking study using identical structured clinical inputs from patients admitted to intensive care units with spontaneous intracerebral hemorrhage. An outcome-trained extreme gradient boosting model was compared with predictions generated by a general-purpose LLM using four prompting strategies: zero-shot, few-shot, chain-of-thought, and combined few-shot plus chain-of-thought prompting. Performance was evaluated using discrimination metrics, threshold-dependent classification behavior, and concordance between Shapley Additive Explanations-derived feature importance rankings and LLM-derived feature prioritization. The independent testing cohort included 435 patients, of whom 86 (19.7%) experienced in-hospital mortality.
RESULTS: The outcome-trained machine learning model demonstrated superior discriminative performance compared with all LLM-based approaches. LLM predictions achieved moderate discrimination but exhibited substantial variability in threshold-dependent classification behavior across prompting strategies. At a fixed probability threshold of 0.5, LLM approaches consistently demonstrated high sensitivity and lower specificity, whereas operating characteristics varied considerably when thresholds were optimized using the Youden index. The optimal thresholds for LLM-based approaches ranged from 0.74 to 0.88, compared with 0.1555 for the extreme gradient boosting model. Concordance between Shapley Additive Explanations-derived attribution and LLM-derived feature prioritization was modest, suggesting only partial alignment between empirically learned predictor structure and language-based reasoning patterns.
CONCLUSIONS: In this structured clinical prediction setting, inference-only LLM outputs demonstrated prompt-sensitive decision behavior despite moderate discriminative performance. These findings suggest that LLM-generated probability outputs should be interpreted cautiously when used for quantitative clinical risk estimation. A complementary framework integrating outcome-trained predictive models with LLM-assisted reasoning may provide a more reliable direction for future clinical decision support systems.
PMID:42422967 | DOI:10.2196/29701