ORIGINAL TECHNICAL PRACTICE
Make abstention useful and testable
When should a question-answering assistant say it cannot answer, and how would you evaluate that behavior?
Written for independent practice. The model answer presents one approach; clarify constraints and explain alternatives.
A REASONED APPROACH
Model answer.
Abstention is appropriate when required evidence is missing, contradictory, outside scope or inaccessible to the current user. A useful refusal explains the gap and, when possible, offers a safe next step.
Work through it
- Define task-specific evidence requirements and acceptable uncertainty. Include answerable, partially answerable and deliberately unsupported examples in evaluation.
- Measure incorrect confident answers as well as unnecessary refusals. Inspect whether citations support each material claim rather than treating the presence of any citation as sufficient.
- Give a clarifying question, a bounded partial answer or a source-location suggestion when that genuinely helps. Keep permission failures distinct from a claim that the information does not exist.
Tradeoffs to discuss
- Stricter thresholds reduce unsupported answers but can frustrate users with answerable questions. Calibrate the policy to the task’s consequences and available evidence.
Common pitfalls
- A fluent uncertainty statement can still contain fabricated details. A model’s self-reported confidence should not be treated as a calibrated probability without validation.