Human in the loop [01kw33y9t82s93d0y2jcxpqq13]

Human in the /loop 🔗 **Source:** [Human in the /loop](https://twitter.com/ericzakariasson/status/2070493377267646797/?rw_tt_thread=True) > [!note]+ Document Notes > tags: [loop-engineering, ai-agents, agentic-workflows, human-in-the-loop, definition-of-done, async-development] > author: eric zakariasson > source: Twitter/X > reference: https://twitter.com/ericzakariasson/status/2070493377267646797/?rw_tt_thread=True > date created: 2026-06-26 09:00 > > # 🏷️ Title: Human in the /loop >  ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ### 🔗 References - [eric zakariasson — "Human in the /loop" (original thread)](https://twitter.com/ericzakariasson/status/2070493377267646797/?rw_tt_thread=True) - Andrej Karpathy's "autoresearch" framing for ML training loops (referenced inline, no link in source) - Playwright (browser automation, referenced as a QA tool) - MCP plus a `/notify` path routed to Slack (referenced as the notification channel) ## 📝 Summary - A practitioner describes how he runs several agent coding tasks in parallel while doing other work, showing up only when a run finishes or needs a decision. - It matters because it reframes "agentic coding" from babysitting one prompt into operating a small fleet of self-scoring loops — the human becomes a reviewer and escalation target, not a driver. - Key stakeholders: developers using coding agents, tooling vendors who ship loop and notify primitives, and anyone building maker/checker automation. - Risks and tradeoffs: loops only work when "done" is genuinely measurable; a weak definition of done lets a loop hill-climb toward the wrong target, and running three to five at once raises review-queue and context-switch load. - Bottom line: write the check first. The whole method rests on a definition of done the agent can score by itself, a loop that stops on clear conditions, and pings so the human leans in at the right moment. ## 📖 Main Content - The core argument: the value in current agent coding is parallelism with low attention cost. You start a long-running task, walk away, and return when it finishes or gets stuck. - The enabling move is a "definition of done the agent can score" — an objective check the agent can run without you. Examples given: a target eval score for model work, a QA pass (load the page or run Playwright, screenshot, confirm behavior) for web/UI, a green test suite for backend/refactor, a number like p95 or a benchmark for speed/flakiness, and a zero-failures count for data/content cleanup. - He frames writing the loop as "writing how you'd check the work yourself" — the check is the spec. He starts more explicit than feels necessary, then loosens once he sees what the model can infer. - The loop is: change something, measure, keep or revert, repeat. The step doesn't have to be tiny; it just has to be measurable against the target. He cares most about stop conditions: metric hits target, no improvement after a few tries, out of ideas, or blocked/unsure (stop and ask). - Instead of babysitting, the agent gets a notify path (MCP plus `/notify`), usually to Slack because that's where everything else lives. He treats it as a generic notification channel, not full Slack access. His reply becomes the next thing the loop runs on. - It runs in the cloud so loops persist for hours without his laptop open; his own client acts as orchestrator and fans work out to cloud agents. - Then he starts the next loop — usually three, sometimes five long loops, plus shorter agents on PRs or one-off investigations. If three are waiting on him, he stops launching and goes to review. - Uncertainty/contested: "loosen the prompt once you see what it can infer" is judgment-heavy and unmeasured; there's no data on failure rates or how often a loop hill-climbs to a bad target. The prompt template image ("/loop drives the iterations and /notify keeps me posted") couldn't be imported, so the exact prompt shape isn't in the text. ### 🗒️ Notes - The unit of work is a *verifiable* task, not a prompt. If you can't state the check, you can't loop it. - Five concrete done-definitions map cleanly to maker/checker: eval score, QA/Playwright pass, green test suite, perf number under a line, validation count to zero. - Stop conditions are first-class. "Blocked or unsure → stop and ask" is the human-in-the-loop hook; the human's answer is the loop's next input. - Notification is deliberately thin: a generic channel (Slack/iMessage), not agent access to the whole tool. Status updates and "I need a decision" arrive as normal messages. - Concurrency is throttled by the human's review capacity, not by the machine: three waiting → stop starting, go review. - Cloud execution decouples loop lifetime from the laptop; the local client is just the orchestrator. - Explicit nod to Karpathy's autoresearch for ML training loops as the closest prior pattern. ## 📚 References - [eric zakariasson — "Human in the /loop"](https://twitter.com/ericzakariasson/status/2070493377267646797/?rw_tt_thread=True) - [Playwright](https://playwright.dev/) — used for the web/UI QA check - Model Context Protocol (MCP) — the transport behind the `/notify` path ## 🗣️ Paraphrase - The best part of coding with agents now is being able to leave a few runs going and keep working; you show up only when something finishes or needs a call. The setup is a definition of done the agent can score, a loop that runs until it should stop, and pings so you know when to lean in. - Before a long task, lock a check the agent can verify: a target score, a QA/Playwright pass, a green test suite, a perf number, or a validation count to zero. Writing the loop is mostly writing how you'd check the work yourself — start explicit, then loosen. - Wrap the check in a loop (change, measure, keep or revert, repeat) and care most about the stop conditions. Give the agent a thin notify path to Slack so it pings you for status or a decision; your reply is what it runs next. Run it in the cloud, then start another loop — about three at a time, and when several wait on you, stop starting and go review. ## 📰 Newsletter - Not applicable to this source — the thread is about general agent-coding loops with no WordPress hook. Light bridge for the .org crowd: the same "definition of done the agent can score" discipline maps onto OSS automation — a green PHPUnit suite, a passing Playwright e2e run, or a Lighthouse/perf number under a set line are all stop-conditions a contributor's agent could loop against, with a thin Slack/Matrix ping when it needs a human decision. ## ✅ Action Items #### DevRel - Developer advocacy: draft a short demo showing a "definition-of-done" loop (green test suite as the target) for a coding-agent talk; measure audience adoption via a follow-up poll. - Education: build an office-hours module on writing the check first — turn each done-definition into a hands-on exercise. - Developer experience: ship a quickstart template pairing a `/loop`-style runner with a `/notify` webhook so devs copy a working stop-condition setup. - Developer marketing: content series on "agent fleets, not single prompts" with attribution to which done-definition drove the most engagement. - Community management: open a thread asking practitioners which stop conditions they actually use. - Internal/external/cross-project: align on a shared notify-channel convention so agent pings don't fragment across tools. #### OSPO - Policy & Governance: set a contribution policy that agent-submitted PRs must carry a machine-checkable definition of done. - License Compliance: ensure looped agents run a license scan as one of their pass/fail checks. - Security Management: require a security check (SAST pass) as a stop condition before an agent loop marks "done." - Software Inventory: treat each loop's check output as an artifact with ownership and freshness SLOs. - Developer Support: offer consultation hours on safe cloud-agent orchestration patterns. - Community Engagement: fund a reference implementation of the maker/checker loop for upstream projects. - Strategic Alignment: report on how many automated contributions pass their own checks unattended. #### Community - Maintainers: publish a release-hygiene checklist that doubles as an agent stop-condition set. - Contributors: add a "first issue" rotation where the issue ships with its definition of done pre-written. - Sponsors: tie milestone payouts to checks the work passes on its own, not subjective review. - Ecosystem: run a cross-project drill where agents loop on a shared benchmark and ping a common channel. #### WordPress - Core: define machine-checkable release criteria that an agent could score against. - Core Performance: set p95/LCP numbers as explicit "under-the-line" loop targets with a perf dashboard. - Core AI: build evaluation sets so AI features have a "target is a score" done-definition. - **Core Program Team:** add a ritual where cross-team dependencies declare their own pass/fail check. - Mobile: wire telemetry checks into agent loops for offline UX changes. - Playground: host reproducible loop demos (green-test-suite target) as education modules. - Community: include "write the check first" in WordCamp speaker mentorship kits. - Training: add a learning-path lesson on definition-of-done and stop conditions. - Marketing: narrative on human-in-the-loop contribution, not full automation. - Support: turn escalation paths into the agent's "blocked or unsure → stop and ask" route. - Design: audit the UX of agent status/decision notifications. - Documentation: examples-first guide to looping an agent against a test suite. - Photos: validation-count loop (zero failing items) for taxonomy hygiene. - TV: caption-quality pass as a stop condition for submitted content. - Polyglots: translation-coverage count as a loop target. - Accessibility: WCAG check as a required pass before an agent loop reports done. - Meta: expose API reliability numbers an agent could measure against. - CLI: scaffold a `wp` smoke-test command usable as a loop check. - Hosting: cross-host checklist that agents can run as their pass/fail gate. - Tide: false-positive review loop as a worked example of stop conditions. - Openverse: ingestion validation count to zero as a loop target. - Themes: lint/test matrix as the theme-review agent's definition of done. - Plugins: provenance and security checks as required loop pass conditions. #### The WPCC - Programs: fund a fellowship building maker/checker loop tooling for maintainers. - Projects: documentation sprint turning the five done-definitions into reusable templates. - Funding experiments: milestone payouts gated on self-passing checks. ### FAIR - Technical independence: run provenance and signature checks as agent stop conditions, locally verifiable. - Policies and covenants: require funding transparency artifacts to pass their own validation loop. - Productized governance artifacts: ship scorecards whose criteria are machine-checkable. ## 🧪 Hypotheses - Teams that write the verification check before starting an agent task complete more tasks unattended than teams that prompt first and check later. - A measurable definition of done reduces the rate of "wrong-target" hill-climbing compared with subjective done criteria. - Throttling concurrency to the human's review capacity (stop starting when three wait) yields higher throughput than fixed maximum concurrency. - A thin, generic notify channel produces faster human response than embedding the agent in a full chat tool. - Cloud-resident loops complete more long-running tasks than laptop-resident loops over a multi-hour window. - Starting explicit and loosening prompts over time improves output quality more than staying maximally explicit throughout. ## 📊 Methodology - Run a controlled study: two cohorts of developers on matched tasks, one writing the check first, one prompting first; measure unattended completion rate and rework. - Instrument loops to log target, each step's measurement, and the stop reason; analyze hill-climb-to-wrong-target frequency against done-definition type. - A/B concurrency policies (capacity-throttled vs fixed) and measure tasks completed per reviewer-hour. - Compare median human response latency across notify channels (thin webhook vs full chat integration). - Track completion of long (>1 hour) loops in cloud vs local runners over the same task set. ### ❓ Questions - What fraction of real coding tasks actually admit a definition of done an agent can score without a human? - How do you detect a loop that's improving the metric but degrading something unmeasured? - What's the right ceiling on parallel loops before review quality drops? - How do you keep a thin notify channel from becoming a noisy interrupt stream? - When does loosening the prompt start to cost more in errors than it saves in effort? ### 🔮 Future Research - A taxonomy of done-definitions by task type, with reliability data on each. - Guardrails that detect goodharting (metric up, true quality down) inside a loop. - Patterns for batching agent pings so the human gets a single review window, not constant interrupts. - Orchestration models for human-capacity-aware scheduling of agent fleets. ### 💡 Implications - Practice: the skill shifts from prompt-craft to check-craft — defining and instrumenting verifiable targets. - Policy: contribution and CI policies can require a machine-checkable done-definition for agent-submitted work. - Security: stop conditions are a natural place to gate on scans and attestations before "done." - Education: curricula should teach writing the check first and choosing stop conditions. - Funding models: milestone payouts can key on self-passing checks rather than subjective review. ### 🧠 Learning Styles Assessment - Reading level estimate: Flesch-Kincaid Grade ~7–8 (short sentences, plain words, conversational). Flesch Reading Ease roughly 65–75. #### 👀 Visual learners - Present: an embedded prompt-template image (which failed to import). - Improve: add an architecture diagram of orchestrator → cloud agents → notify channel, and a before/after of one loop iteration. #### 🔊 Aural learners - Present: none. - Improve: a 2–3 minute audio walkthrough of the loop setup. #### 🗣️ Verbal learners - Present: clear definitions, worked examples per task type, the analogy to Karpathy's autoresearch. - Improve: a short glossary (definition of done, stop condition, orchestrator). #### 🧩 Kinesthetic learners - Present: concrete recipes (run Playwright, screenshot, keep-or-revert). - Improve: a reproducible sandbox with a runnable `/loop` + `/notify` template. #### 🧠 Logical learners - Present: explicit stop-condition list, keep/revert decision rule. - Improve: a decision tree for "loop vs don't loop," KPIs for loop health. #### 🧑‍🤝‍🧑 Social learners - Present: closing call for others to share how to make looping easier. - Improve: a discussion prompt or shared repo of done-definitions. #### 🧘‍♀️ Solitary learners - Present: self-contained, self-paced explainer. - Improve: reflection prompts on which of your tasks are actually checkable. #### 🌿 Naturalistic learners - Present: none. - Improve: frame loops as an ecosystem of agents with feedback cycles; note compute/energy cost of long cloud loops. ### 📊 Accessibility Assessment #### Screen reader semantics - Present: plain text thread, readable in order. - Improve: the embedded prompt image needs descriptive alt text; "content could not be imported" leaves a gap for screen-reader users. #### Keyboard navigation - Present: native platform (X) handling; not author-controlled. - Improve: n/a at the source level. #### Contrast and readability - Present: platform default contrast. - Improve: ensure any republished version meets 4.5:1. #### Text resizing and reflow - Present: platform reflow. - Improve: n/a at source. #### Media and captions - Present: no audio/video. - Improve: provide a text version of the prompt-template image so no information is image-only. #### Forms and interactions - Present: none. - Improve: n/a. #### Standards reference - Gap maps to WCAG 2.2 AA 1.1.1 Non-text Content (the un-imported prompt image carries information with no text alternative). ### 🌍 DEIB Assessment #### Cultural breadth - Present: tool-agnostic framing ("Slack, iMessage, or whatever"). - Improve: examples beyond a single Western-default tool stack. #### Inclusive language - Present: gender-neutral, jargon mostly defined. - Improve: define "p95," "MCP," "autoresearch" inline for newcomers. #### Neurodiversity support - Present: chunked sections, consistent structure, short paragraphs. - Improve: a summary box up top and icon+text section labels. #### Representation - Present: single first-person practitioner voice. - Improve: cite varied practitioners' loop setups. #### Psychological safety - Present: low-stakes, invitational tone ("I'd love to hear how we can make it easier"). - Improve: none needed. ### ✅ Logical Validity #### Strong reasoning - Inductive (whole thread): generalizes from five concrete task types to a single method — strong because each example is independently checkable and the pattern (target → loop → stop) holds across all five. Clarify by adding failure cases where no good check exists. - Causal (loop section): "writing the loop is mostly writing how you'd check the work yourself" — a sound causal claim that verifiability enables autonomy. Strengthen with data on how often verifiable tasks complete unattended. - Analogical (model-work example): mapping to "Karpathy's autoresearch for ML training loops" — apt because both are score-driven iteration. Clarify the limits of the analogy for non-numeric targets. #### Fallacies - Hasty generalization (risk, line: "Most of this doesn't stay on my laptop... It runs in the cloud"): presents one personal workflow as the default without sample or counterexamples. Weakness: readers may assume cloud orchestration is necessary. Fix: scope the claim to "in my setup" and note when local runs suffice. - Suppressed/uncontested premise (line: "Writing the loop is mostly writing how you'd check the work yourself"): assumes the work is checkable at all; tasks with no objective check are silently excluded. Fix: state the precondition explicitly and give an example of a task that doesn't qualify. - Anecdotal evidence (overall): the whole method rests on first-person experience with no metrics. Fix: pair claims with even rough success rates or before/after numbers. #### Tone of voice - Practical, modest, and invitational. Examples: "Examples I actually use," "I start more explicit than I think I need, then loosen it," and the closing "I'd love to hear how we can make it easier for you!" The tone teaches by demonstration rather than asserting authority. ### 🛡️ DARVO Analysis - Not applicable to this source — the thread is a non-adversarial how-to with no accusation, denial, or victim/offender dynamic. ## 🎯 Conclusion - Bottom line: agent coding scales when each task carries a definition of done the agent can score, a loop that stops on clear conditions, and a thin ping so the human leans in only when needed. The check is the spec. - Next steps for readers: pick one current task, write its check first (test suite, eval score, perf number, or validation count), wrap it in a change-measure-keep/revert loop with explicit stop conditions, route a `/notify` ping to wherever you already are, and only start a second loop once the first runs clean. For Courtney's loop-engineering OS this is direct reinforcement: the "definition of done the agent can score" is the maker's target and the stop conditions are the checker's gate — same maker/checker discipline, framed for a fleet of parallel coding agents. ## 🔄 Refactor Appendix — Human in the /loop > Density: 2–5 Literature atoms (H3), 0–3 Permanent candidates (H4). > Defaults: 3 Literature, 1 Permanent. ### 📚 Literature Atoms (H3) #### [H1] Lock a definition of done the agent can score > [!quote] ID:rw_1 > "Before kicking off a longer running task, I lock a definition of done." **Context & use**: - Thread, "Find something the agent can verify" section. - This is the load-bearing move — without a self-scorable check there's no autonomous loop. Mirrors Courtney's maker target. **Tags**: #xref #aphorisms **Highlight**: [view highlight](readwise://)  #### [H2] Writing the loop is writing your own check > [!quote] ID:rw_2 > "Writing the loop is mostly writing how you'd check the work yourself." **Context & use**: - Thread, "Wrap it in a loop" lead-in. - Reframes prompt-craft as check-craft; the check is the spec. **Tags**: #xref #aphorisms **Highlight**: [view highlight](readwise://)  #### [H3] Stop conditions are the loop's real design > [!quote] ID:rw_3 > "I care most about the stop conditions, which might be" **Context & use**: - Thread, "Wrap it in a loop" — precedes the list (metric hits target, no improvement, out of ideas, blocked/unsure). - "Blocked or unsure → stop and ask" is the human-in-the-loop escalation; maps to the checker's gate. **Tags**: #xref **Highlight**: [view highlight](readwise://)  #### [H4] Throttle concurrency to your review capacity > [!quote] ID:rw_4 > "If three are waiting on me I stop starting stuff and go review." **Context & use**: - Thread, "Then start the next one" section. - Human review bandwidth, not the machine, caps the fleet size. **Tags**: #xref **Highlight**: [view highlight](readwise://)  #### [H5] The reply is the loop's next input > [!quote] ID:rw_5 > "When I answer, that reply is the next thing the loop runs on." **Context & use**: - Thread, "Get pinged instead of babysitting" section. - Closes the human-in-the-loop cycle: the agent pings, the human decides, the decision feeds the next iteration. **Tags**: #xref **Highlight**: [view highlight](readwise://)  ### 🧠 Permanent Candidates (H4) #### [P1] A loop is only as good as its check **Claim** (one sentence, Courtney's voice): An agent task can run unattended exactly as far as its definition of done is machine-checkable. **Evidence pointer**: [[#H1]] and [[#H2]] **Scope & conditions**: Holds when the target is objective (score, green tests, perf number, validation count); breaks when "done" needs human judgment that can't be scored. **Links**: - [[Maker-checker discipline]] — because extends - [[Goodhart's law]] — because contrasts on metric-vs-true-quality - [[Loop engineering OS]] — because applies-in Courtney's fleet  ### ✏️ Fleeting Queue - [F1] The prompt-template image didn't import — find the original to capture the exact `/loop` + `/notify` shape. — thread "Prompting" section #fleeting #inbox - [F2] Map each of the five done-definitions to a concrete check in Courtney's loop-registry. — #fleeting #inbox ### → Things 3 (max 2 actions) - [ ] Add a "definition of done the agent can score" field to the loop-template skill — #things - [ ] Cross-link this note to the loop-engineering OS hub in the vault — #things  ## Links - [[Human Review Capacity Sets the Agent WIP Limit]] — distilled claim: stop starting agent work when the human review queue reaches capacity. - [[A Nondeterministic Check Breaks the Loops Stop Condition]] — applies-in: an unattended loop is bounded by the check it can score.