AI source and citation checking means auditing every reference an AI draft produces and sorting each failure into one of three classes: a citation pointing at nothing, a citation pointing at a real paper with wrong details, or a claim carrying no citation at all. One score cannot separate them, and the repair for each is different.
Why This Matters
Walters and Wilder, publishing in Scientific Reports (13:14045), had ChatGPT-3.5 and ChatGPT-4 draft short literature reviews across 42 multidisciplinary topics. The output ran to 84 papers carrying 636 bibliographic citations. Then they checked every reference against the actual scholarly record.
The results split by model, and that split is the whole point. Among GPT-3.5's citations, 55 percent were fabricated — they did not correspond to any real scholarly work. GPT-4's fabricated rate was 18 percent. Same task, same research design, dramatically different failure rate.
A publisher who reads "AI cites sources badly" and stops there will apply one fix to two different problems. The fabricated reference needs to be deleted. The real-but-mangled reference needs to be repaired. Doing either to the wrong class wastes the effort and leaves the draft just as unusable.
What Are the Three Failure Classes in a Citation Audit?
The three classes are: references that point at nothing, references that point at real work with corrupted details, and claims that carry no reference at all. Each requires a distinct action — delete, repair, or attach — and a single quality score cannot tell you which one you are looking at.
Start with class one. The check is simple: does a DOI resolve, does the journal exist, does the author list appear anywhere outside the AI's own output? Walters and Wilder found 55 percent of GPT-3.5 citations and 18 percent of GPT-4 citations failed this test outright. The mechanism is a language model predicting plausible-looking citation strings — author names, years, journal titles — because that is what a citation looks like, not because a record backs it. The consequence for a publisher is the worst of the three: a fabricated reference in a published piece is a retraction-grade error, and it is the one readers check first because it is the easiest to check.
Class two is quieter and more dangerous. Among citations that did point to real papers, 43 percent of the GPT-3.5 citations and 24 percent of the GPT-4 citations contained substantive citation errors. The paper exists. The volume, year, page range, or author order is wrong. Because the reference looks real, it survives a casual skim and fails only when someone tries to retrieve the source from the citation as written.
Class three is a claim with no reference attached at all. You find these by reading the draft as a list of assertions and asking, for each one, what would have to be true for this to be checkable.
The boundary matters here. Finding a citation error means the claim behind it is unverified — not that the claim is false. A mangled page range does not make the underlying finding wrong; it means you cannot confirm it from what you have. Treat the audit as a to-do list, not a verdict.
| Class | What the check looks for | Action | Consequence if missed |
|---|---|---|---|
| Fabricated reference | DOI resolves; journal and authors exist independently | Delete the reference and the claim it supports | Retraction-grade error; readers spot it first |
| Corrupted reference | Volume, year, pages, author order match the record | Repair the details, then re-verify the claim | Source cannot be retrieved as cited |
| Missing reference | Every checkable claim has an attached source | Attach a source or cut the claim | Claim reads as asserted, not evidenced |
How Do You Check an AI Citation Without Reading Every Paper?
You check in two passes: a fast mechanical pass that catches class one and class two, then a slow claim-level pass that catches class three. The mechanical pass is cheap because a DOI lookup either resolves or it does not.
Run every reference through a DOI resolver first. A citation whose DOI returns nothing is class one until proven otherwise — do not spend time hunting for the paper by title, because the title is part of what the model generated. For anything that resolves, compare four fields against the actual record: year, volume, page range, and author order. Those are the fields Walters and Wilder flagged as substantively wrong in 43 percent of GPT-3.5's real citations and 24 percent of GPT-4's.
The claim-level pass is slower and cannot be automated. Read the draft sentence by sentence and mark every assertion a reader could dispute. Then check whether a reference sits behind it. The ForgeRank published dataset scored 29 drafts and recorded 2,839 claims; the scorer judged 51 of those claims to need a source and found 38 with none attached.
That ratio — 38 of 51 — is the number to internalize. Missing citations cluster on the claims that feel most obvious, which is exactly why the writer never stopped to source them.
Why Can't a Quality Score Replace This Audit?
A quality score cannot replace the audit because it collapses three failure classes into one number, and the ForgeRank published dataset shows source attachment is the dimension that moves the headline score on its own. Fixing the other dimensions will not lift it.
In that dataset, ten drafts scored a flat 4.0 across all four dimensions. Ten others held 4.0 on three dimensions with source compliance at 7.0, and those came out at 4.8. The only variable separating the two groups was whether sources were attached.
Read that carefully. A draft can be clean on structure, clarity, and originality and still land at 4.0 because nothing is sourced. The score tells you a problem exists in the sourcing dimension. It does not tell you whether you are holding a fabricated reference, a corrupted one, or a bare claim — and those three need delete, repair, and attach respectively.
Scores are triage. The audit is the treatment.
What Most People Get Wrong About AI Citation Checking
The widespread mistake is treating "AI hallucinates citations" as one problem with one fix, usually "verify the sources." Verification is the right instinct applied at the wrong resolution.
The two model rates in Walters and Wilder — 55 percent for GPT-3.5, 18 percent for GPT-4 — get quoted as a single scary statistic. They are not one statistic. They are two measurements of two different models on the same 42-topic task, and the gap between them is the finding. A team that reads only the higher number builds a process calibrated to a model it may not be using.
The misconception persists because a blended rate is more quotable than a split one. A single percentage fits in a headline; a model-by-model breakdown requires a sentence. So the nuance gets dropped, and teams end up either over-checking drafts from a lower-fabrication model or under-checking drafts from a higher one.
The second error is subtler: assuming a reference that resolves is a reference that is correct. GPT-4's 18 percent fabrication rate means 82 percent of its citations pointed at real work — but 24 percent of those real citations carried substantive errors. A resolved DOI is the start of the check, not the end of it.
Google Search Central framed its March 2024 core update around reducing low-quality, unoriginal content in search results by 40 percent. Unsourced and mis-sourced claims sit squarely in that category, which makes the audit a distribution question, not just an accuracy one.
Key Takeaways
- AI citation failures split into three classes — fabricated references, corrupted references, and unsourced claims — and each needs a different action: delete, repair, or attach.
- Walters and Wilder (Scientific Reports 13:14045) found 55 percent of GPT-3.5 citations and 18 percent of GPT-4 citations were fabricated across 84 papers and 636 references.
- Among citations pointing at real papers, 43 percent of GPT-3.5's and 24 percent of GPT-4's contained substantive errors, so a resolving DOI does not mean a correct citation.
- In the ForgeRank published dataset, source compliance was the only dimension that moved a headline score on its own — ten drafts at 4.0 on three dimensions scored 4.8 once sourcing hit 7.0.
- A citation error means the claim is unverified, not false. Audit output is a to-do list, not a verdict on accuracy.
Frequently Asked Questions
What is AI source and citation checking?
It is the process of auditing every reference in an AI-generated draft against the real scholarly or primary record. You verify that each citation points at something that exists, that its details match the actual source, and that claims needing support have a reference attached.
How many AI citations are fabricated?
Walters and Wilder, in Scientific Reports (13:14045), found 55 percent of ChatGPT-3.5 citations and 18 percent of ChatGPT-4 citations were fabricated across 84 papers and 636 total citations. The rate depends heavily on which model produced the draft.
Does a working DOI mean the citation is correct?
No. Among citations that pointed at real papers, Walters and Wilder found 43 percent of GPT-3.5's and 24 percent of GPT-4's contained substantive errors — wrong volume, year, pages, or author order. A resolving DOI confirms the paper exists, not that the citation describes it accurately.
Can a quality score catch citation problems?
Not at the class level. The ForgeRank published dataset shows source attachment drives the headline score independently, but a score cannot tell you whether you hold a fabricated reference, a corrupted one, or a missing one. Those need different fixes.
Does a citation error mean the claim is false?
No. A corrupted or fabricated reference means the claim behind it is unverified — you cannot confirm it from what you have. The underlying finding may still be correct; the audit simply tells you it is not yet evidenced.
Conclusion
Sort every reference in your AI draft into one of three buckets before you touch the prose: fabricated, corrupted, or missing. Delete the first, repair the second, attach a source to the third. A blended score will not do this work for you, and a resolved DOI will not either.
Open your most recent AI-assisted draft tonight and run the DOI pass on every reference in it. Count how many return nothing. That count is your class-one list, and it is the one you fix first.