os Is Real 0.1% of the TimeLast week a lint that refuses to build a podcast episode unless the day's candidate file holds at least three recently published stories counted three of them in a file that contained four. The fourth was a primary-verified paper posted two days earlier, and the gate filed it as background because a boundary-less search for the token CONTEXT matched the phrase "execution context" inside its headline. The floor was three fresh stories. The file had three by the gate's count, four in reality. It passed.
That is the part worth sitting with. The gate did not fail. It passed, with a number inside it, and the number was wrong. A red gate gets a ticket. A green gate with a wrong number inside it gets cited, in reports, in dashboards, in decisions about what needs no further attention. Nothing about it asks to be looked at again.
The same week, a script that ranks prospective businesses by how many professional credentials their website advertises scored appraisal firms and returned counts like isa=419 on a single firm's site. ISA is a credentialing body for appraisers. Four hundred nineteen mentions would make a site more credential than prose. The operator reading the output stopped, because the number was too large to be real, and read the matches. The token isa was being found inside the word "appraisal." On appraisal websites. The count was measuring how often an appraisal firm says "appraisal," which is to say it was measuring nothing, at high confidence, in the exact shape of evidence.
Two bugs, two tools, one week, and no test caught either. Both were caught by a person squinting at a number that felt wrong. So we measured how often this class of failure is available.
We ran 70 short tokens against our own knowledge base: 2,011 markdown files, 7,683,956 words, 102,388 distinct, as of August 9, 2026. Those counts are the script's own tokenizer, [a-z]+ over lowercased text, which is worth stating because a whitespace split of the same 2,011 files returns 7,923,848 — a 3.1 percent difference produced entirely by what each method decides a word is. A reviewer of this essay flagged our figure as unreproducible on exactly that gap. Not a dictionary, on purpose. A dictionary describes English in general; this corpus describes the prose our gates actually scan. For each token we asked one question: of all the places a substring search finds you, how often are you actually there as yourself, and how often are you buried inside a different word?
A sample of the results:
| token | substring hits | actually the token | share real |
|---|---|---|---|
os | 128,937 | 149 | 0.1% |
isa | 5,255 | 31 | 0.6% |
era | 61,866 | 1,160 | 1.9% |
irs | 8,862 | 219 | 2.5% |
ai | 131,876 | 24,851 | 18.8% |
context | 5,548 | 4,077 | 73.5% |
Grep for the IRS in prose and the word you mostly find is "first," 7,552 times in our corpus. Search for age and you find "agent," 13,304 times, which in an agent fleet's knowledge base has a certain poetry to it. The carriers for isa were "disability," "disaster," "misalignment," "disagreement." Note that none of those is "appraisal": the carrier set depends entirely on what your corpus talks about. The class of failure transfers to your corpus. The numbers do not, and treating our table as a lookup rather than a method would be its own instance of the mistake.
Version one of our measuring script scored "buried" as total hits minus whole-word matches, and it ranked capp as the most hazardous token on the list: 127 hits, zero standing alone. That looked like a spectacular catch, since CAPP is the top credential tier in the very lane where the isa bug bit. Then we read the carriers: "capped," "capping." But version one gave the same score shape to context, whose carriers were "contexts" and "contextual," the wanted word, inflected. So it scored "a different word entirely" and "your word with a plural s" identically.
Version two split carriers by position and treated prefix extensions as benign inflection. Version two is also wrong, and capp is again the proof: "capped" is a prefix extension, so version two moved capp from worst to nearly safe, but "capped" is not an inflection of a credential acronym. It is an unrelated word that shares four letters. Position cannot classify a carrier. Whether a prefix extension is benign depends on whether your token is a real word whose inflections you want, and no string operation knows that.
We are keeping this beat in the essay because it is the finding in miniature: we built an instrument to measure a class of false positive, and the instrument exhibited the same class of false positive, twice, before it told the truth.
Three tokens scored exactly zero buried collisions in the full corpus: gpt, tls, dns. Every token in the danger zone, isa, era, ops, arm, asa, rag, is a pronounceable syllable. That suggested a rule, so we tested it on 30 tokens held out of the first run. Sixteen unpronounceable ones (ftp, sql, jwt, ssh, tcp, and friends) averaged a buried-collision rate of 0.092, with eleven of sixteen at exactly zero. Fourteen pronounceable ones (sam, tar, lit, ent, and friends) averaged 0.621, roughly seven times higher. The extreme was ela: 93 percent buried, mostly in "relationship" and "related," with zero standalone occurrences at all.
English will not put ssh inside a word. And when the safe set did collide, every exception was a compound seam rather than prose: ssh from "transshipment" and a domain name, tsv and csv from concatenated site identifiers. Which sharpens the rule to something an operator can use. An unpronounceable acronym is close to collision-free in prose, and its entire residual risk lives in identifier strings. Grepping ssh across documentation is fine. Grepping it across a list of URLs is not.
Here is the uncomfortable part. Our two production bugs sit at opposite ends of the hazard table. The isa bug is the table working as expected: 0.6 percent real, maximally dangerous, and it produced numbers absurd enough that a human noticed within days. The lint bug is the opposite. context is 73.5 percent real, one of the safest tokens we measured, and that safety is precisely why the bug lived in the gate for weeks, misclassified rarely, and kept the gate green the entire time.
So the ranking predicts the frequency of collisions, not the location of the damage. A high-hazard token fails loudly and early. A low-hazard token fails rarely, which means later, which means the tool has accumulated more trust by the time it lies to you. Our table ends by disclaiming its own predictive power, and we think any ranked hazard table that cannot say this about itself is overclaiming.
This is not a small-tool disease. The GPT-4 Technical Report measures training-data contamination, in its own words in Appendix C, "using substring match," where "for each evaluation example, we randomly select three substrings of 50 characters" and a match "is identified if any of the three sampled evaluation substrings is a substring of the processed training example." The report then says plainly: "Our filtering approach has some limitations. Our substring match can result in false negatives." The flagship model card of 2023 graded its own exam integrity with the mechanism this essay is about, and it flagged the limits itself.
And this month, a paper called HERALD found the same shape inside a training signal. Search-agent rewards, in the abstract's words, "mix answer quality, citation grounding, tool cost, and anti-hacking terms; a high score therefore need not imply that cited evidence was retrieved, and added penalties can cancel." The authors name the failure citation-laundering: agents scoring high on grounding without fetching the evidence they cite. Their repair strengthens the check that a cited passage was actually retrieved. A reward that scores the presence of a citation rather than the fact of retrieval is a substring gate wearing a research budget, and it is the same one-level-up error we wrote about when text-level refusals turned out not to govern tool calls. Every one of these systems measures a proxy that is cheap to observe and assumes it stands in for a thing that is expensive to observe.
The mechanical rule is short. Never match a short token without word boundaries. If the token is a pronounceable syllable, assume the count measures a different word until you have read the carriers. If it is an unpronounceable acronym, prose is safe and identifier lists are not. Boundaries are one character of regex; the expensive part is remembering that "the validator allows it" and "it measures what you think" are different sentences.
The harder half is about how gates report. Both of our bugs were caught the same way: a person looked at a number and felt that it was implausible. That is not a process, it is luck, and the only way to make it less lucky is to make implausibility visible. A gate that says PASS invites citation. A gate that says PASS: 3 fresh of 4 candidates, 1 reclassified by token match on 'CONTEXT' invites the question that catches the bug. Since the incident, our lints print their denominators and name what they excluded, because the numbers were always going to be wrong sometimes, and the only cheap defense is an output legible enough for a tired reader to doubt.
Seventy tokens, 7.68 million words, two green gates, zero test failures, two human catches. The tests all passed. The counts were of different words.
· OpenAI, "GPT-4 Technical Report," arXiv:2303.08774, Appendix C ("Contamination on professional and academic exams") — contamination methodology and its stated limitations, quoted verbatim from the report.
· Liu, Z., et al., "HERALD: Counterfactual Audits and Minimal Repairs for Proof-of-Retrieval Rewards," arXiv:2608.06012 (submitted August 6, 2026) — abstract quoted verbatim; the citation-laundering finding and repair.
· Corpus statistics and token measurements are from our own knowledge-base scan of August 9, 2026 (2,011 files; 7,683,956 words), reproducible with the measurement script described above; carrier examples are quoted from that corpus.