> [!question] Lingering Questions > - What is the actual false-positive rate of any commercial detector on non-native English writing? The entire ethical argument depends on a number nobody has published. > - If detectors are trivially defeatable by benchmarking against ~40 text metrics, what is the remaining legitimate use case beyond intimidation? > - Does the sentence-level-reordering-only editing constraint actually preserve detector scores, or merely reduce the effect? Untested in the source. > [!tip] Article Ideas > - "Detectors measure predictability, everyone reads them as measuring honesty" — the category error as a public-safe explainer, with the Yoda minimal-pair demonstration as the hook. > [!link] Cross Reference > - [[3. Resources/MOCs/AI Agent Safety MOC]] — primary home; a control that misfires systematically against the least-advantaged is a safety and equity finding at once > - [[3. Resources/Permanent Notes/A Defense Experts Fail Is Not a Defense for Users]] — closely related but distinct: that note is about experts reliably failing a review task; this source is about a control that experts can defeat entirely while the naive still get caught by it > [!quote] Vocabulary > - **Surprisal / next-word probability** — the statistical measure nearly all current AI detectors actually compute; long runs of highly predictable words score as "machine-generated" regardless of who wrote them > - **Negative parallelism, triadic rhythm** — named stylistic "tells" (e.g. "it's not this, it's this"; rule-of-three phrasing) that recur across ChatGPT and Claude output and are used informally as detection heuristics independent of statistical scoring ## Bibliographic capsule Katie Robbert, Christopher S. Penn, John Wall, Trust Insights *So What?* live show, "How AI Detectors Work." christopherspenn.com, companion video YouTube 2026-07-31, 46m05s. Source: [[3. Resources/Readwise/Full Document Contents/Articles/So What How AI Detectors Work]] ## Author's thesis (as I read it) **What triggered the episode?** A reader accused the Trust Insights newsletter of being AI slop written by liars, around the same time Substack adopted the detector Pangram. Rather than defend abstractly, the team built their own detector on a laptop and demonstrated exactly what it measures. **What do they show?** That detectors overwhelmingly measure next-word predictability (surprisal), not authorship — and that AI *editing* specifically, by rearranging a human's own words into a more statistically probable order, is what actually triggers detection. A colleague's dictated-then-Claude-tidied posts kept flagging; her own unedited rough draft scored as more clearly human than her polished one. ## Key arguments (≤3) 1. **The category error**: "They do not detect for quality... They only detect the likelihood of word order within a corpus of text." Predictable human writing (technical writers, plain description, second-language English) flags identically to machine writing. 2. **The misfire concentrates on the least advantaged**: non-native English speakers, neurodivergent writers, and anyone trained on the formulaic five-paragraph essay — traced to training corpora built from whatever data is easiest to obtain, which skews wealthy and majority. 3. **Detection targets editing, not authorship**: the Yoda minimal pair ("For 800 years have I trained Jedi" vs. "I have trained Jedi for 800 years" — identical words, only the reordered version flags) proves AI-assisted *editing* is what triggers detectors, independent of who originated the content. ## Claim atoms ### Detectors measure predictability and are read as measuring honesty > [!quote] ID: rw_aidetect_01 > They do not detect for quality... They only detect the likelihood of word order within a corpus of text. **Where:** mechanism segment. **Why it matters:** names the category error at the root of the entire practice, and is the direct defense for anyone falsely accused. ### AI editing rearranges words into probable order — that is what gets detected > [!quote] ID: rw_aidetect_02 > when an AI tool edits, it rearranges the words back into a statistically probable pattern **Where:** the Becca case and the Yoda minimal pair. **Why it matters:** reframes the entire category from authorship detection to editing detection, and yields a concrete, immediately usable mitigation (constrain AI editing to sentence-level reordering only). ### The misfire is not random — it concentrates on the least advantaged writers **Where:** who-gets-penalized and training-corpus segments. **Why it matters:** non-native speakers, neurodivergent writers, and the formulaically taught, traced to a mechanism (corpora built from the easiest-to-obtain data, which skews wealthy and majority) rather than merely asserted as bias. ## Permanent candidates - **Editing toward the probable is what detection detects, not authorship** — distinct enough from the vault's existing note on shared-tool homogenization (different mechanism: this is about word-order statistics specifically, not general output convergence) to warrant its own note. **Created as permanent note.** - A control the sophisticated evade and the naive fail is worse than no control — closely related to [[3. Resources/Permanent Notes/A Defense Experts Fail Is Not a Defense for Users]] (possibly the inverse framing of the same family); connect rather than create a near-duplicate. ## Processing checklist - [x] Read the full deep-read notes field (Refactor Appendix present, ported directly) - [x] Create Literature note in Konik format - [x] Create Permanent note for the highest-resonance uncaptured claim - [ ] Find any published false-positive rate for commercial detectors on non-native English — the whole equity argument needs this number - [ ] Adopt the sentence-level-only AI editing constraint in own publishing prompts ## Links - [[3. Resources/MOCs/AI Agent Safety MOC]] — primary home; a control that misfires systematically is a safety/equity finding - [[3. Resources/MOCs/AI & the Future of Knowledge Work MOC]] — the equity and false-positive material bears directly on institutional AI-detection policy - [[3. Resources/MOCs/Personal Knowledge Management MOC]] — applies-in: the MOC lists this note under Synthesis additions.