# Automation Removes the Social Checks a Process Silently Depended On ## Claim When a task is automated, the informal human behaviours that quietly held the surrounding process together disappear with it, and the process fails in ways nobody predicted because nobody had written those behaviours down. <!-- atomicity: 1 idea ✓ --> ## Evidence **Source**: [He's Right.](3.%20Resources/Readwise/Articles/He's%20Right..md) — Theo (t3.gg), on a visibly broken UI that shipped through review > [!quote] > now we live in a world where the code review going up does not imply any effort was put into checking the thing ahead The worked case is precise. A model wrote interface text by reading the source and never rendering it, so it never discovered the container was too narrow. The bug shipped through a code review that found nothing wrong — because the code *was* fine. What actually failed was an unwritten assumption. Code review was never a full verification mechanism; it worked because it sat on top of a **social** fact: a human building that feature would have looked at it in a browser at least once. Theo names the mechanism as courtesy rather than rigour — *"why would you waste the time of your co-workers with code that you never even saw the output of?"* Agents have no such reflex, and nobody noticed the reflex was load-bearing until it was gone. The general shape: the automated step was documented, the norm surrounding it was not, and the process was silently depending on the undocumented half. ## Scope & conditions - Holds wherever a process rests on unstated norms rather than enforced gates. Code review depending on author testing is the instance here; the pattern is not specific to software. - The fix is to make the implicit check **explicit and mandatory** — require evidence of execution, not disclosure of AI authorship. The second is trivially satisfiable and mostly beside the point. - Fails as a critique of automation itself. The norm was always fragile; automation revealed the fragility rather than causing it. A new hire who never ran their code would have produced the same failure, more slowly. - Generalises to any handoff where *"of course they'd have checked"* was doing real work — QA, editorial, approvals, moderation. ## Links - [[AI & the Future of Knowledge Work MOC]] — what disappeared from the review process when authoring got automated - [[AI Agent Safety MOC]] — the unwritten pre-review check whose loss makes independent verification mandatory - [[3. Resources/Permanent Notes/The Maker Should Not Grade Its Own Homework]] — the structural remedy once the social check is gone: a separate checker, because the producer will not catch this - [[3. Resources/Permanent Notes/An Agent That Reports on Its Own Work Verifies the Change Not the State]] — the specific failure mode when you ask the automated party to supply the missing check itself - [[3. Resources/Permanent Notes/Intent Written on the Outside Compounds While Re-Derived Intent Is Debt]] — the same lesson about undocumented assumptions, applied to intent rather than to process norms - [[3. Resources/Permanent Notes/Weak Links Cap Aggregate Gains Even Under Widespread Automation]] — why removing this check doesn't speed anything up: review was the binding constraint, not authoring - [[3. Resources/Permanent Notes/Compressing Access Time Hides Friction Rather Than Removing It]] — because these are one claim at two altitudes: a removed delay takes with it something load-bearing that nobody had written down.