Demos · What this means for research
Attaching the source
Ask the model to back up its answer and, without the paper, it can’t. Give it the paper and the same request becomes something you can check in seconds. Guide: Context →
What we asked
In Kahneman & Tversky (1979), what percentage of subjects chose option C in Problem 2? … Quote the exact sentence from the paper that reports this number.
Asked with nothing attached and with the paper attached, 20 runs each. The paper says 83.
What came back
Nothing attached
15 of 20
refused to quote the paper
5 of 20
quoted a sentence that is not in the paper, for example:
“In Problem 2, 80% of the subjects chose C, whereas 20% preferred D.”
The paper attached
20 of 20
answered 83, the right number
19 of 20
quoted the real sentence:
“The data show that 82 per cent of the subjects chose B in Problem 1, and 83 per cent of the subjects chose C in Problem 2.”
gpt-4o via the API · temperature 1.0 · September 2026
What to notice
- Without the paper, the invented quotes look like quotes. Right problem, right format, a plausible sentence. The number in them is the wrong one.
- With the paper, the model stops predicting the answer from memory and continues from text that contains it. The right number becomes the likely one.
- The quote is what makes the answer checkable. You can search the paper for that sentence in seconds. Context is what makes verification cheap.
Limits
One question, one paper. We used a text copy of the paper, which you’d have to supply yourself.