A site chat assistant was moved to a newer small AI model. Soon after, some replies were a single word like “Hi,” some visitors saw a generic “had trouble responding” error, and one reply stopped halfway through an internal formatting tag. The same questions had worked fine on the old model.
After an AI Model Upgrade, Chatbot Replies Came Back as One Word or Cut Off Mid-Sentence
⚡ Quick Fix (TL;DR)
The Culprit: The newer model has "thinking" switched on by default, and thinking tokens count against the maximum reply length. With a tight limit, the model could spend most or all of its budget thinking and return little or no visible text. Because the request itself succeeded, nothing looked like an error. The first suspect, the web search tool, turned out to be wrong; a temporary log of every non-normal response showed an ordinary success status with a "hit the token limit" stop reason and thinking blocks in the reply.
The Fix: Turned thinking off for ordinary replies (this assistant is a fast FAQ bot), raised the length limit a little because the new model counts about 30 percent more tokens for the same text, and stripped any unfinished internal tag at the end of a reply as a safety net. The takeaway: after any model change, re-read the defaults before the first test, and log the stop reason and response type before guessing at a cause.
Tagged in :