Effect of Model Reasoning Level on Guideline Selection, Triage Question Selection, and Disposition Accuracy in LLM-Based Telehealth Triage

Authors

Matthew Thompson, Product Manager, AI Initiatives, LaGrange Medical Software / Schmitt-Thompson Clinical Content

Laurie O’Bryan RN, Nurse Editor, Schmitt-Thompson Clinical Content

David A. Thompson MD, Senior Medical Editor, Schmitt-Thompson Clinical Content; Northwestern University Feinberg School of Medicine

Background: Current large language models (LLMs) have a reasoning setting that controls how much the model works through a problem before answering. Higher settings take more time and use more tokens, a measure of model output and cost. Whether they improve clinical accuracy inside a guideline-constrained triage workflow is an open question. An earlier internal comparison of low and high reasoning on 527 adult scenarios found no disposition accuracy benefit and 40% slower response time. We examined whether higher reasoning levels lead to meaningful improvements in guideline selection, triage question selection, and disposition accuracy. Using a larger scenario set, we compared four reasoning levels while holding all other settings constant.

Methods: The evaluation used automated batch testing, with each scenario submitted as a single message. The study used the adult Schmitt-Thompson Clinical Content (STCC) after-hours Gen AI guideline database (314 guidelines, September 2026) and a library of 690 clinical scenarios written for 37 target guidelines. The scenarios were developed and validated through a structured 14-step process, including blinded review by five expert triage nurse leaders. Each scenario had a recommended guideline, a positive Triage Assessment Question (TAQ), and a disposition. Where more than one guideline fit the scenario, up to two additional guidelines were listed as acceptable. The chatbot (proof-of-concept build 0.13.1) retrieved candidate guidelines, selected one, and assigned the disposition of the first positive TAQ within that guideline, working from the highest urgency level down. It processed each scenario once at each of four GPT-5.2 reasoning levels: none, low, medium, and high. All other configurable settings were identical. Guideline selection was scored correct when the selected guideline matched any acceptable guideline for that scenario; TAQ and disposition were scored as exact matches. Runs that returned no disposition were scored as incorrect. Under-triage was defined as a selected disposition of lower acuity than the recommended disposition on the STCC disposition scale. Because every level ran the same 690 scenarios, the levels were compared with paired tests: Cochran’s Q across the four levels, McNemar exact tests between pairs, and Wilcoxon signed-rank tests for latency.

Results: Guideline selection accuracy was 91.4%, 93.0%, 92.6%, and 92.3% at none, low, medium, and high reasoning (p = 0.16). TAQ selection accuracy was 73.6%, 73.3%, 71.6%, and 72.2% (p = 0.17). Disposition accuracy was 93.2%, 92.9%, 91.2%, and 92.5% (p = 0.061). Accuracy did not rise with reasoning level. The only pairwise difference reaching p < 0.05 was medium below low on disposition accuracy (1.7 points, p = 0.029), likely due to chance given 18 comparisons. Under-triage occurred in 27 scenarios (3.9%) with no reasoning versus 19 (2.8%), 20 (2.9%), and 21 (3.0%) at low, medium, and high (none versus low, p = 0.077); the no-reasoning level also produced fewer over-triage results (15 versus 20 to 24) and fewer runs with no disposition (5 versus 10 to 17). Disposition was wrong at all four levels in 26 scenarios (3.8%); 18 of these had the correct guideline, which points to TAQ matching as the source of error. Response time increased with each level: mean 37.7, 42.6, 45.1, and 47.5 seconds per scenario (none to high +9.8 seconds, 26%; all pairwise p < 0.001). Output tokens doubled from 2,540 (none) to 5,048 (high).

Conclusions: In a triage system that retrieves telehealth triage guidelines through retrieval-augmented generation (RAG), raising the GPT-5.2 reasoning level did not measurably improve guideline, TAQ, or disposition accuracy, but it slowed responses and increased cost. The one statistically significant difference is more likely chance than a true effect. Low reasoning remains a reasonable default setting. Turning reasoning off would save about five seconds per scenario but produced eight more under-triage cases than low reasoning in this run, so that trade-off needs replication. Most persistent errors arose in TAQ matching, which added reasoning did not fix. These findings are limited to one model, written adult scenarios, and 37 of 314 guidelines.

Citation

Thompson M, O’Bryan L, Thompson DA. Effect of Model Reasoning Level on Guideline Selection, Triage Question Selection, and Disposition Accuracy in LLM-Based Telehealth Triage. Schmitt-Thompson Clinical Content. November 16, 20266.