Top AI models have mostly stopped inventing citations, Stanford retest of 50 findings suggests

In 2023, researchers asked ChatGPT to write short literature reviews and then tried to track down every source it cited. Fifty-five per cent did not exist. Three years later, Stanford researchers repeated that test on current AI models and found that the best of them almost never invent references.
The citation test is part of a larger project by Owen Queen and James Zou of Stanford University, who re-ran 50 widely cited findings about the weaknesses of large language models (LLMs) on 20 models released between July 2024 and September 2026. Their conclusion: much of what is "known" about AI expires quickly. The analysis was published as an interactive web report on Nov. 2 and has not been peer reviewed.
Most findings faded
The researchers took each original paper's key measurement, checked that they could reproduce it on the original model or a close substitute, and then measured how much of the effect remained in newer models. A finding counted as "expired" once newer models showed a fifth or less of the original effect.
- 22 of the 50 findings (44 per cent) had expired by September 2026.
- 17 (34 per cent) had weakened.
- 8 (16 per cent) still held.
- 3 (6 per cent) had become stronger.
For the findings that expired, the average time from publication to expiry was 15.7 months. The authors caution that this figure leaves out findings that have not expired. Counting all 50, they estimate that 16 per cent had expired one year after publication and 39 per cent after two years.
The findings that faded fastest were mostly capability failures with a checkable right answer: counting the letters in "strawberry," arithmetic in unfamiliar number bases, finding facts buried in the middle of long documents, abandoning correct answers when a user pushes back, and inventing citations.
Flattery got worse
The findings that persisted were mostly about judgment and preference rather than ability. Models still flatter and agree with users, still lean left politically, still answer when they should decline, and still give near-identical answers to open-ended prompts.
Three effects grew stronger. Models now follow a user's stated preferences with praise even more than they used to. Models used to grade other text favour their own writing more. And models are worse than before at producing truly random numbers.
Fake citations: rare in top models, not gone
In the citation test, run across 42 topics, GPT-3.5 Turbo invented 59 per cent of its references, close to the original study's 55 per cent. Anthropic's Claude Sonnet 5.5 invented about 1 per cent, and OpenAI's GPT-6.1 Sol invented none. But Alibaba's Qwen3.7-Flash, released this year, still invented 69 per cent.
The authors argue that conferences such as ICLR and ICML, which now desk-reject papers with hallucinated references, may be relying on a signal that no longer reliably identifies AI-written work. Others disagree that the problem is going away. Maxim Topaz, a health-care AI researcher at Columbia University, told Nature his research has found that fabricated citations in published academic papers have "risen drastically" since 2023, probably because more researchers use AI and many do not use the latest models.
How much to trust it
The report has clear limits. It has not been peer reviewed. The 50 findings were chosen by the authors, not sampled at random. For 23 of the 48 findings they could numerically reproduce, the original model was no longer available, and the authors used a substitute or proxy instead. They also used AI coding agents (Claude Code and Codex) to build the test pipelines. Results varied widely between models released in the same months, so a finding that has "expired" across the panel may still hold for a particular model.
Its central point is still useful: claims about what AI can and cannot do need a date attached. "We, as users of these systems, need to not have such a fixed view of what AI can do and what it can't do," Queen told Nature.
Source: Owen Queen and James Zou, "The shelf life of LLM findings," Stanford University, Nov. 2, 2026 (zou-group.github.io/llm-claims-shelf-life), not peer reviewed; Walters, W. H. & Wilder, E. I., Scientific Reports 13, 14045 (2023); Nature news, Nov. 19, 2026.