Can a member-facing AI assistant say "I don't know" instead of guessing?
For an assistant that answers members, I would rather it stop and say it does not have enough information, and point to the closest source, than give a confident answer stitched together from weak context. What actually makes an assistant refuse to guess when the approved knowledge base has no answer?
Yes, but a prompt alone will not get you there reliably. An instruction like "say I don't know if unsure" helps, but the model often does not know that it is unsure. The stronger fix is to decide in the system, before the model writes anything, whether the retrieved sources are good enough.
Set a retrieval threshold: if no passage from the approved knowledge base scores above it, skip generation and return a fixed reply that says the answer is not available, with the closest source and a way to reach a person. That path never hallucinates because the model is not asked to write.
When sources do pass the bar, ask the model to answer only from the quoted passages and to cite which passage supports each claim. Then check the citations in code: an answer with a claim that points to no passage gets downgraded to the fallback reply.
Test it with questions you know are not covered. Count how often the assistant answers anyway, and treat that number as the main metric, next to how often it refuses questions it could have answered.
Listings mentioned
- Rag implementation · skill by wshobsonBuilds knowledge-grounded document Q&A with vector search, the base a refuse-when-unsure assistant sits on.
- Rag architect · skill by alirezarezvaniHelps evaluate retrieval quality with precision@k and recall@k, which is how you set a sensible threshold.
- Hallucination vulnerability prompt checker · prompt by Scott MalinFinds the places in your system prompt that invite the model to fill gaps with made-up answers.
Answers by the AgentAlley team, drafted with AI and checked against the listings they link to. Not a real-person reply from the original thread.