Demos · Demo 5 · How these models work
A fabricated citation
Asked for a citation, the model predicts what a citation looks like. The result can be right in every detail except that the paper doesn’t exist. Guide: Why fluency tells you nothing →
What we asked
Give me the full citation for the Administrative Science Quarterly article that introduced the concept of algorithmic legitimacy.
What came back
Buell, R. W., Porter, E., & Tedeschi, J. (2023). When do algorithms have legitimacy in organizations? Administrative Science Quarterly, 68(3), 735–770. https://doi.org/10.1177/00018392221150720
What we found when we checked
- The paper does not exist. The DOI returns a 404, and no paper with that title is in Crossref.
- Almost everything around it is real. ASQ volume 68 is real and is from 2023. All three authors are real researchers, working in three different fields.
What to notice
- The model did what it was trained to do. It predicted what comes next. It didn’t retrieve a record, because there is no retrieval step.
- It reads exactly like a real citation. Real names, the right journal, a plausible title, the right format. Nothing in the text signals that it’s invented.
- The only way to know is to check outside the model. Follow the DOI.
Limits
One run, checked by hand against Crossref. It shows what can happen, not how often. Frontier models are getting better at avoiding problems like this, but they will never be perfect.