A chat assistant was given a web search tool, but every test question that needed current information got a polite “I can’t look that up from here.” Nothing in the logs showed a search attempt, and the assistant’s own answers sounded reasonable, which made it easy to blame the model.
For a Chatbot, A Tool Offered to the AI Never Got Used, and the Cause Was Two Different Things
⚡ Quick Fix (TL;DR)
The Culprit: Two separate causes were stacked. First, the tool is only offered when the site's own reference lookups find nothing, and those lookups check the last few messages, not just the newest one. A term mentioned earlier in the same test chat, or a test question that itself contained a reference term like "WordPress" or "Shopify," quietly switched search off for that turn. Second, once the tool really was offered, the model still skipped it while its thinking was switched off.
The Fix: Added a temporary per-turn log showing whether the tool was offered, what blocked it, and whether it was used. That immediately separated "never offered" from "offered and ignored." Then tested one question per fresh chat window, with no reference terms in it, and turned thinking on only for turns where the tool is offered. The takeaway: log the gate decision before blaming the model, and test in a clean session so earlier messages can't change the result.
Tagged in :