AI content detectors score how predictable each token is, given everything before it. A language model assigns probabilities to word choices; detectors run the same math and flag text whose words sit at the high-probability end. The output is an authorship guess, not a measurement of whether the writing is any good.
Why This Matters
Two students submit the same essay. One grew up speaking English at home, the other learned it at fourteen. The detector flags the second one. Weixin Liang and co-authors, in arXiv 2304.02819 (published in Patterns), found that roughly 61 percent of TOEFL essays written by humans were flagged as AI-generated. That is a false-positive rate on real human writing, and it lands hardest on people whose English is fluent but statistically conventional.
The mechanism explains the bias. Detectors reward low perplexity — word sequences a model would have predicted. A writer who reaches for the standard academic connector instead of the idiosyncratic one produces exactly that profile. So does a writer who has been taught formal register by a textbook.
What you get back is a number between 0 and 1 labeled "AI probability." What you asked for was whether a person wrote it. Those are different questions, and the gap between them is where most of the trouble lives.
How Does a Detector Actually Score Text?
Most detectors lean on statistical signals rather than on any one test, and implementations differ between products. The two most commonly cited are perplexity, which measures how surprised a language model is by each word you chose, and burstiness, which measures how much sentence length and structure vary across the passage. Low perplexity plus flat burstiness pushes a score toward "AI". Some tools add token-predictability measures, sentence-level classifiers, or reference-corpus retrieval on top, so a description of one detector's internals is not a specification for the category.
The math is borrowed from the models themselves. A detector loads a reference language model, feeds your text through it, and records the probability the model assigned to each actual token. Predictable tokens score high. Then it aggregates. OpenAI shut down its own AI classifier in July 2023, having reported a detection rate and a human misclassification rate too weak to justify the confidence the tool implied: a documented admission that the signal did not support shipping. The rates it reported are recorded in our detection benchmark.
The consequence for you: a clean human draft with uniform sentence lengths and conventional vocabulary can score as machine-written, and there is no appeal process. The boundary is genre. Poetry, dialogue, and heavily edited marketing copy all break the assumption that natural human writing has high variance, because those forms deliberately suppress it.
Are AI Detectors Accurate? Every Published Rate Carries a Condition
Both major vendors publish their own error rates, and each published rate arrives with a condition that most summaries drop. Turnitin defines its document-level figure only for documents already at least a fifth machine-written, and it publishes a separate sentence-level figure that counts a different object. GPTZero's headline detection rate comes from RAID, a benchmark GPTZero itself describes as containing only fully human or fully AI texts. Read that way, the condition is the substance: every published rate describes a population a given writer is probably not in.
The consequence for reading any of these numbers: a benchmark assembled from clean endpoints cannot estimate an error rate for the ambiguous middle, and the ambiguous middle is where accusations actually start. Our detection benchmark lists each published figure with its unit, the condition that makes it true, and a link to the vendor page it came from, alongside the false positive classes we found in our own scorer.
Why Two Detectors Disagree About the Same Paragraph
The same paragraph can collect different verdicts from different tools, and the reason is mechanical. Each detector loads its own reference language model, applies its own threshold, and normalizes its own score. Two tools that both measure token predictability will still diverge wherever their reference models assign different probabilities to the same token, and neither exposes enough of its internals for an outsider to predict where.
Disagreement widens on text that has been edited. GPTZero states that under adversarial attacks such as swapping synonyms, misspelling words, and paraphrasing, "every method suffers a decrease in TPR@FPR=1%." Length does the same damage: Turnitin's own evaluation found a higher false positive rate on very short submissions, and the company responded by refusing to score anything under 300 words. Short text carries too little signal, so the score stops being stable.
The consequence for a writer: one detector's number is not a measurement you can rank against another detector's number. The boundary appears the moment you need a decision. Two tools disagreeing tells you the text sits near a threshold, not which tool is right, and no vendor publishes the threshold. If you need something you can act on, the productive question is not who wrote the draft but what in it is weak — and that is the question a rubric answers and a detector cannot.
What Do the Four Quality Dimensions Measure?
Four dimensions — structure, evidence, sourcing discipline, and phrasing — measure properties of the text itself, which is a different object from authorship likelihood. The ForgeRank published dataset (https://forgerankai.com/static/data/ai-content-quality-29-drafts.json) scores 29 drafts across those dimensions on a 0–10 scale.
Take phrasing. It measures whether the prose leans on templated constructions. The mechanism matters because templated phrasing is what detectors key on, so the two scores correlate without measuring the same thing. The consequence for a writer: you can score 7.0 or higher on quality while still drawing a heavy-AI-phrasing verdict — eight of the nine drafts at 7.0 or above did exactly that, and the only draft whose phrasing came back clean scored 4.0 overall. The boundary: phrasing says nothing about whether a human or a model produced the sentence, only about how the sentence is built.
Detector vs. Content Quality: What's the Difference?
A detector estimates who wrote the text. A content quality score estimates what the text does. One is a forensic guess about origin; the other is a rubric applied to the artifact in front of you.
The published dataset makes the split concrete. Across its 29 drafts, the scorer recorded 2,839 claims. Of those, 51 were judged to need a source, and 38 had none attached. That is a property of the drafts — an evidence-discipline finding — and no authorship detector would surface it, because an unsourced claim and a sourced claim can carry identical perplexity. The quality dimensions also move independently: all ten 4.0 drafts scored 4.0 on all four dimensions, while the single 8.5 draft scored 8.0/9.0/9.0/8.0, meaning each step up the distribution came from one dimension moving rather than all four rising together.
The consequence: a detector can tell you a draft looks machine-written and leave the missing citations untouched. The boundary is scope. Quality scoring requires a rubric and a human or model applying it; detection requires only a reference language model and a probability threshold.
Detector vs. Content Checker: Which One Do You Need?
A content checker verifies specific claims — a statistic, a quote, a citation — against a source. A detector estimates authorship. If your draft cites a figure, the checker is the tool that catches a wrong or absent attribution.
The dataset shows where that matters. Of 51 claims flagged as needing a source, 38 arrived with nothing attached. A checker would have caught the gap before publication. A detector would have reported a percentage and moved on. Google's March 2024 core update stated its aim plainly: to "reduce low-quality, unoriginal content in search results by 40 percent," per Google Search Central. That target is about content properties, and it is the reason quality scoring has a business case that detection does not.
The consequence for a publishing workflow: run the checker on every factual claim, and treat the detector score as advisory. The boundary appears when the text is opinion or first-person narrative — there is nothing external to check, so the checker has no work to do and the detector's guess is all you have.
Free AI Detectors and Free Checkers: What Each One Gives You
Search for a free AI detector and you will find tools that return a probability, usually behind a word or character cap. ZeroGPT, GPTZero, and Copyleaks each offer a free way in. Turnitin does not sell to writers at all, because it is licensed to institutions, which is why students usually meet its score through a submission system rather than a website. What no free tier can give you is an error rate for your own text: the vendor figures above are population averages under stated conditions, and your document is a single sample from a distribution none of them publishes.
A free quality checker answers a different question, and the difference matters most after a detector has already flagged you. A probability tells you what the tool concluded about authorship; it gives you nothing to edit. A rubric applied to the same draft reports properties of the text itself, so the output is a list of things you can still change before the work ships.
The consequence for picking a tool: decide which question you are asking. If you need to know whether a specific detector will flag a submission, no free tool can answer that in advance, because the threshold is unpublished and the reference model changes without notice. If you need to know what is weak in the draft, the four-dimension read at forgerankai.com is free, needs no signup, and returns findings rather than a verdict about you.
What Most People Get Wrong About AI Detectors
The common belief is that a high AI score proves a machine wrote the text. The evidence supports a narrower claim: the score reflects how closely the text matches a language model's probability distribution, and human writers who match that distribution get flagged. The TOEFL result — roughly 61 percent of human-written essays flagged, per Liang et al. in arXiv 2304.02819 — is the correction.
The belief persists because the output is a confident number. A 0.97 reads like a verdict, and a verdict invites no further questions. The quality distribution tells a different story about what measurement can do. A rubric that reports "38 of 51 claims needing a source had none" gives you something to act on. A detector that reports "97 percent AI" gives you a number you cannot verify, cannot appeal, and cannot fix.
What the dataset shows is that scoring text on its own properties — structure, evidence, sourcing, phrasing — produces findings you can edit against. Authorship detection produces a guess about a person. When the guess is wrong, the person pays.
Key Takeaways
- Detectors score token predictability, not authorship, and the output is a probability rather than a finding.
- Roughly 61 percent of TOEFL essays written by humans were flagged as AI-generated, according to Liang et al. in arXiv 2304.02819.
- Eight of nine drafts scoring 7.0 or above in the ForgeRank published dataset still received a heavy-AI-phrasing verdict; the one clean-phrasing draft scored 4.0.
- Of 2,839 claims recorded across 29 drafts, 51 needed a source and 38 had none attached — a content-level finding no detector surfaces.
- Google's March 2024 core update aimed to "reduce low-quality, unoriginal content in search results by 40 percent," per Google Search Central.
- Every published detector error rate carries a condition; our detection benchmark lists each one with its unit, its condition and its source.
Frequently Asked Questions
Can a human-written essay be flagged as AI?
Yes. Liang et al. found roughly 61 percent of TOEFL essays written by humans were flagged as AI-generated, published in arXiv 2304.02819. Formal register and conventional vocabulary push perplexity down, which is the same profile detectors associate with machine output.
Why did OpenAI shut down its AI classifier?
OpenAI retired its classifier in July 2023 because the signal was too weak to support the confidence the tool implied. The detection and mislabelling rates it reported are recorded in our detection benchmark.
Does a high AI score mean the writing is low quality?
No. The ForgeRank published dataset scored 29 drafts on structure, evidence, sourcing, and phrasing, and eight of the nine drafts at 7.0 or above still drew a heavy-AI-phrasing verdict. Authorship likelihood and text quality are separate measurements.
What is the difference between a detector and a content checker?
A detector estimates who wrote the text using token probability. A content checker verifies whether a specific claim, quote, or statistic matches a source. In the ForgeRank published dataset, 38 of 51 claims needing a source had none attached — a gap only a checker catches.
Can I lower my AI detection score?
You can raise perplexity by varying sentence length and choosing less predictable word sequences, but the ForgeRank published dataset shows phrasing and quality diverge: the one draft with clean phrasing scored 4.0 overall. Optimizing for the detector can cost you the quality score.
Is Turnitin's AI detector accurate?
Turnitin publishes a document-level rate and a sentence-level rate, and the headline one is conditional: it describes documents that already contain at least a fifth AI writing, not a fully human essay. Turnitin also limits scoring to submissions of 300 words or more, after its own evaluation found a higher false positive rate on shorter text. Our detection benchmark records each figure with the condition it depends on.
Is GPTZero accurate?
GPTZero's headline detection rate comes from RAID, an independent third-party benchmark, and GPTZero states that RAID "considers only fully human or fully AI texts." The figure therefore describes clean endpoints rather than drafts that mix human and machine writing. Our detection benchmark carries the figure, its unit and that condition together.
What is the false positive rate of AI detectors?
It depends on the unit and the population sampled, which is why the published rates cannot be lined up against each other. One vendor reports a document-level rate and a separate sentence-level rate, and those count different objects. Against those vendor figures, Liang et al. in arXiv 2304.02819 found roughly 61% of human-written TOEFL essays were flagged, a result Turnitin has publicly disputed in its own research on English Language Learner writers. Our detection benchmark lists each number with its unit and its condition.
Do AI detectors agree with each other?
Not reliably. Each detector uses its own reference language model and threshold, so they diverge wherever those models assign different probabilities to the same token. GPTZero states that under adversarial attacks such as synonyms, misspellings, and paraphrasing, every method's detection rate at a 1% false positive rate falls. When two tools disagree, the text sits near a threshold, and the disagreement says nothing about authorship.
The Bottom Line
Detection and quality measurement answer different questions, and only one of them gives you something to fix. A detector reports a probability about a person; a rubric reports properties of a page you can still edit. When a score comes back high, open the draft and count the claims that need a source but have none — that number tells you what to change tonight.