# Disclosing A Methods Limits Is What Makes Findings Usable ## Claim Publishing a measurement's accuracy and failure modes alongside its headline number is what converts an interesting finding into something a reader can act on. <!-- atomicity: 1 idea ✓ --> ## Scope & conditions - Holds when the audience is numerate enough to use the caveat and the disclosure is complete. - Breaks when disclosure is selective — reporting the favorable statistic (recall) while withholding the unfavorable one (precision) is selective disclosure wearing the costume of transparency, and is worse than no disclosure because it borrows credibility it hasn't earned. ## Evidence Christopher Penn built a deterministic classifier for a job-search demo specifically so he could check its work, and disclosed 65% overall accuracy against a hand-labeled holdout and 100% recall on the category his headline finding depended on. But he never reports precision anywhere — and 100% recall is trivially achievable by over-classifying, so the 34%-of-comments-are-referral-farming headline can't actually be evaluated without it. The disclosure habit is genuinely good; the specific disclosure in this case is incomplete in exactly the direction that flatters the finding. ## Links - [[3. Resources/Permanent Notes/A Contribution Pipeline Needs a Defined Next Step After Training]] — because contrasts: that note's evidence is careful about stating a pilot target rather than an already-proven result — the same discipline of not overclaiming past what the data supports - [[3. Resources/MOCs/AI & the Future of Knowledge Work MOC]] — because applies-in: benchmark and evaluation writeups generally, not just this one demo - [[3. Resources/MOCs/Evidence & Claims Literacy MOC]] — applies-in: Fits claims literacy on how disclosed method limits let readers judge a finding. ## Tests / falsifiers - Request or compute the missing precision figure for Penn's classifier; if precision is low, the 34% headline is substantially inflated by false positives and the finding needs revision, not just an asterisk. - Audit a sample of other "methods disclosed" claims (in this vault or elsewhere) for the same selective pattern — reporting whichever metric flatters the headline while omitting its complement. ## Maintenance - Last tended: 2026-08-16 - Next review: 2026-11-16 ## Source - [[3. Resources/Literature Notes/How to Demonstrate Your AI Chops in Job Hunting (Almost Timely News)]] — the classifier methodology and its incomplete disclosure.