# A Nondeterministic Check Breaks the Loop's Stop Condition ## Claim An unattended loop's verification step has to be deterministic and idempotent — a flaky check that goes green then red on the same unchanged state is worse than no check, because it corrupts the stop condition: the loop fixes what isn't broken or halts on what is. <!-- atomicity: 1 idea ✓ --> ## Evidence **Source**: [Loop Engineering: A Technical Roadmap for an Autonomous Loop (h100envy)](https://read.readwise.io/read/01kw37fxkz7ndac7x0hyr1nqyg). > [!quote] > "A flaky test (green then red on the same code) is worse than no test, because it breaks the stop condition: the loop will fix what is not broken, or stop on what is broken. Before building the loop, run the check ten times on one state. If the result is not stable, fix the check first, then the loop." A loop reads its checker's verdict to decide whether to keep going or stop. If that verdict isn't a stable function of the state, the loop's control flow is driven by noise: it can thrash on a passing artifact or declare a broken one done. The prescribed fix is a determinism pre-flight — run the check ~10 times on one frozen state and stabilize the check *before* wiring it into the loop. Determinism is a property of the checker that has to hold before the checker's independence even matters. ## Scope & conditions - Holds for any unattended maker/checker loop whose stop condition consumes the check's verdict. - Relaxes for attended loops where a human is the real stop condition and can override a flaky signal — but a flaky check still wastes the human's attention. ## Links - [[The Maker Should Not Grade Its Own Homework]] — extends — independence makes a checker *trustworthy*; determinism makes its verdict *usable* as a stop condition. Both are required: an independent but flaky checker still breaks the loop - [[Encoding What Good Looks Like Is What Makes Agent Verification Trustworthy]] — extends — the external standard must also be stable; a drifting or flaky "what good looks like" is the same failure one level up - [[Bounded Action Spaces Not Better Models Make Agents Safe to Run]] — applies-in — a deterministic check is part of the bounded, observable environment that makes unattended runs safe - [[AI Agent Safety MOC]] — applies-in — verification reliability is a control-layer property, not just a testing nicety - [[3. Resources/Permanent Notes/Eval Quality Is the Rate-Limiter on Safe Agent Autonomy]] — evidence-for: a noisy eval corrupts the stop signal and therefore caps how safely an autonomous loop can improve or terminate. ## Implications - For Courtney's loop-engineering OS: every checker in the registry needs a determinism pre-flight — run it 10x on a frozen artifact before trusting it as a gate. A flaky checker is a silent loop-killer that looks like a working one. Worth adding to the `loop-template` "when not to loop" gate. - Open question: where checks are inherently noisy (LLM-judge verifiers, network-dependent QA), what's the right stabilization — majority vote over N runs, a tolerance band, or a deterministic proxy?