Human in the /Loop  🔗 **Source:** [Human in the /Loop](https://twitter.com/ericzakariasson/status/2070493377267646797/?rw_tt_thread=True) > [!note]+ Document Notes > tags: [loop-engineering, ai-agents, agentic-workflows, human-in-the-loop, definition-of-done, async-development] > author: eric zakariasson > source: Twitter/X > reference: https://twitter.com/ericzakariasson/status/2070493377267646797/?rw_tt_thread=True > date created: 2026-06-26 09:00 > > # 🏷️ Title: Human in the /loop > <!-- deep-read:sections start --> ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ### 🔗 References - [eric zakariasson — "Human in the /loop" (original thread)](https://twitter.com/ericzakariasson/status/2070493377267646797/?rw_tt_thread=True) - Andrej Karpathy's "autoresearch" framing for ML training loops (referenced inline, no link in source) - Playwright (browser automation, referenced as a QA tool) - MCP plus a `/notify` path routed to Slack (referenced as the notification channel) ## 📝 Summary - A practitioner describes how he runs several agent coding tasks in parallel while doing other work, showing up only when a run finishes or needs a decision. - It matters because it reframes "agentic coding" from babysitting one prompt into operating a small fleet of self-scoring loops — the human becomes a reviewer and escalation target, not a driver. - Key stakeholders: developers using coding agents, tooling vendors who ship loop and notify primitives, and anyone building maker/checker automation. - Risks and tradeoffs: loops only work when "done" is genuinely measurable; a weak definition of done lets a loop hill-climb toward the wrong target, and running three to five at once raises review-queue and context-switch load. - Bottom line: write the check first. The whole method rests on a definition of done the agent can score by itself, a loop that stops on clear conditions, and pings so the human leans in at the right moment. ## 📖 Main Content - The core argument: the value in current agent coding is parallelism with low attention cost. You start a long-running task, walk away, and return when it finishes or gets stuck. - The enabling move is a "definition of done the agent can score" — an objective check the agent can run without you. Examples given: a target eval score for model work, a QA pass (load the page or run Playwright, screenshot, confirm behavior) for web/UI, a green test suite for backend/refactor, a number like p95 or a benchmark for speed/flakiness, and a zero-failures count for data/content cleanup. - He frames writing the loop as "writing how you'd check the work yourself" — the check is the spec. He starts more explicit than feels necessary, then loosens once he sees what the model can infer. - The loop is: change something, measure, keep or revert, repeat. The step doesn't have to be tiny; it just has to be measurable against the target. He cares most about stop conditions: metric hits target, no improvement after a few tries, out of ideas, or blocked/unsure (stop and ask). - Instead of babysitting, the agent gets a notify path (MCP plus `/notify`), usually to Slack because that's where everything else lives. He treats it as a generic notification channel, not full Slack access. His reply becomes the next thing the loop runs on. - It runs in the cloud so loops persist for hours without his laptop open; his own client acts as orchestrator and fans work out to cloud agents. - Then he starts the next loop — usually three, sometimes five long loops, plus shorter agents on PRs or one-off investigations. If three are waiting on him, he stops launching and goes to review. - Uncertainty/contested: "loosen the prompt once you see what it can infer" is judgment-heavy and unmeasured; there's no data on failure rates or how often a loop hill-climbs to a bad target. The prompt template image ("/loop drives the iterations and /notify keeps me posted") couldn't be imported, so the exact prompt shape isn't in the text. ### 🗒️ Notes - The unit of work is a *verifiable* task, not a prompt. If you can't state the check, you can't loop it. - Five concrete done-definitions map cleanly to maker/checker: eval score, QA/Playwright pass, green test suite, perf number under a line, validation count to zero. - Stop conditions are first-class. "Blocked or unsure → stop and ask" is the human-in-the-loop hook; the human's answer is the loop's next input. - Notification is deliberately thin: a generic channel (Slack/iMessage), not agent access to the whole tool. Status updates and "I need a decision" arrive as normal messages. - Concurrency is throttled by the human's review capacity, not by the machine: three waiting → stop starting, go review. - Cloud execution decouples loop lifetime from the laptop; the local client is just the orchestrator. - Explicit nod to Karpathy's autoresearch for ML training loops as the closest prior pattern. ## 📚 References - [eric zakariasson — "Human in the /loop"](https://twitter.com/ericzakariasson/status/2070493377267646797/?rw_tt_thread=True) - [Playwright](https://playwright.dev/) — used for the web/UI QA check - Model Context Protocol (MCP) — the transport behind the `/notify` path ## 🗣️ Paraphrase - The best part of coding with agents now is being able to leave a few runs going and keep working; you show up only when something finishes or needs a call. The setup is a definition of done the agent can score, a loop that runs until it should stop, and pings so you know when to lean in. - Before a long task, lock a check the agent can verify: a target score, a QA/Playwright pass, a green test suite, a perf number, or a validation count to zero. Writing the loop is mostly writing how you'd check the work yourself — start explicit, then loosen. - Wrap the check in a loop (change, measure, keep or revert, repeat) and care most about the stop conditions. Give the agent a thin notify path to Slack so it pings you for status or a decision; your reply is what it runs next. Run it in the cloud, then start another loop — about three at a time, and when several wait on you, stop starting and go review. ## 📰 Newsletter - Not applicable to this source — the thread is about general agent-coding loops with no WordPress hook. Light bridge for the .org crowd: the same "definition of done the agent can score" discipline maps onto OSS automation — a green PHPUnit suite, a passing Playwright e2e run, or a Lighthouse/perf number under a set line are all stop-conditions a contributor's agent could loop against, with a thin Slack/Matrix ping when it needs a human decision. ## ✅ Action Items #### DevRel - Developer advocacy: draft a short demo showing a "definition-of-done" loop (green test suite as the target) for a coding-agent talk; measure audience adoption via a follow-up poll. - Education: build an office-hours module on writing the check first — turn each done-definition into a hands-on exercise. - Developer experience: ship a quickstart template pairing a `/loop`-style runner with a `/notify` webhook so devs copy a working stop-condition setup. - Developer marketing: content series on "agent fleets, not single prompts" with attribution to which done-definition drove the most engagement. - Community management: open a thread asking practitioners which stop conditions they actually use. - Internal/external/cross-project: align on a shared notify-channel convention so agent pings don't fragment across tools. #### OSPO - Policy & Governance: set a contribution policy that agent-submitted PRs must carry a machine-checkable definition of done. - License Compliance: ensure looped agents run a license scan as one of their pass/fail checks. - Security Management: require a security check (SAST pass) as a stop condition before an agent loop marks "done." - Software Inventory: treat each loop's check output as an artifact with ownership and freshness SLOs. - Developer Support: offer consultation hours on safe cloud-agent orchestration patterns. - Community Engagement: fund a reference implementation of the maker/checker loop for upstream projects. - Strategic Alignment: report on how many automated contributions pass their own checks unattended. #### Community - Maintainers: publish a release-hygiene checklist that doubles as an agent stop-condition set. - Contributors: add a "first issue" rotation where the issue ships with its definition of done pre-written. - Sponsors: tie milestone payouts to checks the work passes on its own, not subjective review. - Ecosystem: run a cross-project drill where agents loop on a shared benchmark and ping a common channel. #### WordPress - Core: define machine-checkable release criteria that an agent could score against. - Core Performance: set p95/LCP numbers as explicit "under-the-line" loop targets with a perf dashboard. - Core AI: build evaluation sets so AI features have a "target is a score" done-definition. - **Core Program Team:** add a ritual where cross-team dependencies declare their own pass/fail check. - Mobile: wire telemetry checks into agent loops for offline UX changes. - Playground: host reproducible loop demos (green-test-suite target) as education modules. - Community: include "write the check first" in WordCamp speaker mentorship kits. - Training: add a learning-path lesson on definition-of-done and stop conditions. - Marketing: narrative on human-in-the-loop contribution, not full automation. - Support: turn escalation paths into the agent's "blocked or unsure → stop and ask" route. - Design: audit the UX of agent status/decision notifications. - Documentation: examples-first guide to looping an agent against a test suite. - Photos: validation-count loop (zero failing items) for taxonomy hygiene. - TV: caption-quality pass as a stop condition for submitted content. - Polyglots: translation-coverage count as a loop target. - Accessibility: WCAG check as a required pass before an agent loop reports done. - Meta: expose API reliability numbers an agent could measure against. - CLI: scaffold a `wp` smoke-test command usable as a loop check. - Hosting: cross-host checklist that agents can run as their pass/fail gate. - Tide: false-positive review loop as a worked example of stop conditions. - Openverse: ingestion validation count to zero as a loop target. - Themes: lint/test matrix as the theme-review agent's definition of done. - Plugins: provenance and security checks as required loop pass conditions. #### The WPCC - Programs: fund a fellowship building maker/checker loop tooling for maintainers. - Projects: documentation sprint turning the five done-definitions into reusable templates. - Funding experiments: milestone payouts gated on self-passing checks. ### FAIR - Technical independence: run provenance and signature checks as agent stop conditions, locally verifiable. - Policies and covenants: require funding transparency artifacts to pass their own validation loop. - Productized governance artifacts: ship scorecards whose criteria are machine-checkable. ## 🧪 Hypotheses - Teams that write the verification check before starting an agent task complete more tasks unattended than teams that prompt first and check later. - A measurable definition of done reduces the rate of "wrong-target" hill-climbing compared with subjective done criteria. - Throttling concurrency to the human's review capacity (stop starting when three wait) yields higher throughput than fixed maximum concurrency. - A thin, generic notify channel produces faster human response than embedding the agent in a full chat tool. - Cloud-resident loops complete more long-running tasks than laptop-resident loops over a multi-hour window. - Starting explicit and loosening prompts over time improves output quality more than staying maximally explicit throughout. ## 📊 Methodology - Run a controlled study: two cohorts of developers on matched tasks, one writing the check first, one prompting first; measure unattended completion rate and rework. - Instrument loops to log target, each step's measurement, and the stop reason; analyze hill-climb-to-wrong-target frequency against done-definition type. - A/B concurrency policies (capacity-throttled vs fixed) and measure tasks completed per reviewer-hour. - Compare median human response latency across notify channels (thin webhook vs full chat integration). - Track completion of long (>1 hour) loops in cloud vs local runners over the same task set. ### ❓ Questions - What fraction of real coding tasks actually admit a definition of done an agent can score without a human? - How do you detect a loop that's improving the metric but degrading something unmeasured? - What's the right ceiling on parallel loops before review quality drops? - How do you keep a thin notify channel from becoming a noisy interrupt stream? - When does loosening the prompt start to cost more in errors than it saves in effort? ### 🔮 Future Research - A taxonomy of done-definitions by task type, with reliability data on each. - Guardrails that detect goodharting (metric up, true quality down) inside a loop. - Patterns for batching agent pings so the human gets a single review window, not constant interrupts. - Orchestration models for human-capacity-aware scheduling of agent fleets. ### 💡 Implications - Practice: the skill shifts from prompt-craft to check-craft — defining and instrumenting verifiable targets. - Policy: contribution and CI policies can require a machine-checkable done-definition for agent-submitted work. - Security: stop conditions are a natural place to gate on scans and attestations before "done." - Education: curricula should teach writing the check first and choosing stop conditions. - Funding models: milestone payouts can key on self-passing checks rather than subjective review. ### 🧠 Learning Styles Assessment - Reading level estimate: Flesch-Kincaid Grade ~7–8 (short sentences, plain words, conversational). Flesch Reading Ease roughly 65–75. #### 👀 Visual learners - Present: an embedded prompt-template image (which failed to import). - Improve: add an architecture diagram of orchestrator → cloud agents → notify channel, and a before/after of one loop iteration. #### 🔊 Aural learners - Present: none. - Improve: a 2–3 minute audio walkthrough of the loop setup. #### 🗣️ Verbal learners - Present: clear definitions, worked examples per task type, the analogy to Karpathy's autoresearch. - Improve: a short glossary (definition of done, stop condition, orchestrator). #### 🧩 Kinesthetic learners - Present: concrete recipes (run Playwright, screenshot, keep-or-revert). - Improve: a reproducible sandbox with a runnable `/loop` + `/notify` template. #### 🧠 Logical learners - Present: explicit stop-condition list, keep/revert decision rule. - Improve: a decision tree for "loop vs don't loop," KPIs for loop health. #### 🧑‍🤝‍🧑 Social learners - Present: closing call for others to share how to make looping easier. - Improve: a discussion prompt or shared repo of done-definitions. #### 🧘‍♀️ Solitary learners - Present: self-contained, self-paced explainer. - Improve: reflection prompts on which of your tasks are actually checkable. #### 🌿 Naturalistic learners - Present: none. - Improve: frame loops as an ecosystem of agents with feedback cycles; note compute/energy cost of long cloud loops. ### 📊 Accessibility Assessment #### Screen reader semantics - Present: plain text thread, readable in order. - Improve: the embedded prompt image needs descriptive alt text; "content could not be imported" leaves a gap for screen-reader users. #### Keyboard navigation - Present: native platform (X) handling; not author-controlled. - Improve: n/a at the source level. #### Contrast and readability - Present: platform default contrast. - Improve: ensure any republished version meets 4.5:1. #### Text resizing and reflow - Present: platform reflow. - Improve: n/a at source. #### Media and captions - Present: no audio/video. - Improve: provide a text version of the prompt-template image so no information is image-only. #### Forms and interactions - Present: none. - Improve: n/a. #### Standards reference - Gap maps to WCAG 2.2 AA 1.1.1 Non-text Content (the un-imported prompt image carries information with no text alternative). ### 🌍 DEIB Assessment #### Cultural breadth - Present: tool-agnostic framing ("Slack, iMessage, or whatever"). - Improve: examples beyond a single Western-default tool stack. #### Inclusive language - Present: gender-neutral, jargon mostly defined. - Improve: define "p95," "MCP," "autoresearch" inline for newcomers. #### Neurodiversity support - Present: chunked sections, consistent structure, short paragraphs. - Improve: a summary box up top and icon+text section labels. #### Representation - Present: single first-person practitioner voice. - Improve: cite varied practitioners' loop setups. #### Psychological safety - Present: low-stakes, invitational tone ("I'd love to hear how we can make it easier"). - Improve: none needed. ### ✅ Logical Validity #### Strong reasoning - Inductive (whole thread): generalizes from five concrete task types to a single method — strong because each example is independently checkable and the pattern (target → loop → stop) holds across all five. Clarify by adding failure cases where no good check exists. - Causal (loop section): "writing the loop is mostly writing how you'd check the work yourself" — a sound causal claim that verifiability enables autonomy. Strengthen with data on how often verifiable tasks complete unattended. - Analogical (model-work example): mapping to "Karpathy's autoresearch for ML training loops" — apt because both are score-driven iteration. Clarify the limits of the analogy for non-numeric targets. #### Fallacies - Hasty generalization (risk, line: "Most of this doesn't stay on my laptop... It runs in the cloud"): presents one personal workflow as the default without sample or counterexamples. Weakness: readers may assume cloud orchestration is necessary. Fix: scope the claim to "in my setup" and note when local runs suffice. - Suppressed/uncontested premise (line: "Writing the loop is mostly writing how you'd check the work yourself"): assumes the work is checkable at all; tasks with no objective check are silently excluded. Fix: state the precondition explicitly and give an example of a task that doesn't qualify. - Anecdotal evidence (overall): the whole method rests on first-person experience with no metrics. Fix: pair claims with even rough success rates or before/after numbers. #### Tone of voice - Practical, modest, and invitational. Examples: "Examples I actually use," "I start more explicit than I think I need, then loosen it," and the closing "I'd love to hear how we can make it easier for you!" The tone teaches by demonstration rather than asserting authority. ### 🛡️ DARVO Analysis - Not applicable to this source — the thread is a non-adversarial how-to with no accusation, denial, or victim/offender dynamic. ## 🎯 Conclusion - Bottom line: agent coding scales when each task carries a definition of done the agent can score, a loop that stops on clear conditions, and a thin ping so the human leans in only when needed. The check is the spec. - Next steps for readers: pick one current task, write its check first (test suite, eval score, perf number, or validation count), wrap it in a change-measure-keep/revert loop with explicit stop conditions, route a `/notify` ping to wherever you already are, and only start a second loop once the first runs clean. For Courtney's loop-engineering OS this is direct reinforcement: the "definition of done the agent can score" is the maker's target and the stop conditions are the checker's gate — same maker/checker discipline, framed for a fleet of parallel coding agents. ## 🔄 Refactor Appendix — Human in the /loop > Density: 2–5 Literature atoms (H3), 0–3 Permanent candidates (H4). > Defaults: 3 Literature, 1 Permanent. ### 📚 Literature Atoms (H3) #### [H1] Lock a definition of done the agent can score > [!quote] ID:rw_1 > "Before kicking off a longer running task, I lock a definition of done." **Context & use**: - Thread, "Find something the agent can verify" section. - This is the load-bearing move — without a self-scorable check there's no autonomous loop. Mirrors Courtney's maker target. **Tags**: #xref #aphorisms **Highlight**: [view highlight](readwise://) <!-- atomicity: 1 idea ✓ --> #### [H2] Writing the loop is writing your own check > [!quote] ID:rw_2 > "Writing the loop is mostly writing how you'd check the work yourself." **Context & use**: - Thread, "Wrap it in a loop" lead-in. - Reframes prompt-craft as check-craft; the check is the spec. **Tags**: #xref #aphorisms **Highlight**: [view highlight](readwise://) <!-- atomicity: 1 idea ✓ --> #### [H3] Stop conditions are the loop's real design > [!quote] ID:rw_3 > "I care most about the stop conditions, which might be" **Context & use**: - Thread, "Wrap it in a loop" — precedes the list (metric hits target, no improvement, out of ideas, blocked/unsure). - "Blocked or unsure → stop and ask" is the human-in-the-loop escalation; maps to the checker's gate. **Tags**: #xref **Highlight**: [view highlight](readwise://) <!-- atomicity: 1 idea ✓ --> #### [H4] Throttle concurrency to your review capacity > [!quote] ID:rw_4 > "If three are waiting on me I stop starting stuff and go review." **Context & use**: - Thread, "Then start the next one" section. - Human review bandwidth, not the machine, caps the fleet size. **Tags**: #xref **Highlight**: [view highlight](readwise://) <!-- atomicity: 1 idea ✓ --> #### [H5] The reply is the loop's next input > [!quote] ID:rw_5 > "When I answer, that reply is the next thing the loop runs on." **Context & use**: - Thread, "Get pinged instead of babysitting" section. - Closes the human-in-the-loop cycle: the agent pings, the human decides, the decision feeds the next iteration. **Tags**: #xref **Highlight**: [view highlight](readwise://) <!-- atomicity: 1 idea ✓ --> ### 🧠 Permanent Candidates (H4) #### [P1] A loop is only as good as its check **Claim** (one sentence, Courtney's voice): An agent task can run unattended exactly as far as its definition of done is machine-checkable. **Evidence pointer**: [[#H1]] and [[#H2]] **Scope & conditions**: Holds when the target is objective (score, green tests, perf number, validation count); breaks when "done" needs human judgment that can't be scored. **Links**: - [[Maker-checker discipline]] — because extends - [[Goodhart's law]] — because contrasts on metric-vs-true-quality - [[Loop engineering OS]] — because applies-in Courtney's fleet <!-- atomicity: 1 idea ✓ --> ### ✏️ Fleeting Queue - [F1] The prompt-template image didn't import — find the original to capture the exact `/loop` + `/notify` shape. — thread "Prompting" section #fleeting #inbox - [F2] Map each of the five done-definitions to a concrete check in Courtney's loop-registry. — #fleeting #inbox ### → Things 3 (max 2 actions) - [ ] Add a "definition of done the agent can score" field to the loop-template skill — #things - [ ] Cross-link this note to the loop-engineering OS hub in the vault — #things <!-- deep-read:sections end --> 📄 **Full text:** [[Full Document Contents/Tweets/Human in the Loop.md|Human in the /Loop by eric zakariasson]] ## Highlights - Before kicking off a longer running task, I lock a definition of done. Examples I actually use: ([View Highlight](https://read.readwise.io/read/01kw3drwkw84aznp82v8nyztpg)) - Note: Lock a scoreable definition of done before a long run. - Tags: atom/literature/h1 - Writing the loop is mostly writing how you'd check the work yourself. Some tasks need every step on the page. Others I give a goal and a rough direction and let the model fill in the middle. I start more explicit than I think I need, then loosen it once I see what it can infer. ([View Highlight](https://read.readwise.io/read/01kw3drz84m7a7h8tgwtfgtbkq)) - Note: Writing the loop = writing how you'd check the work yourself; start explicit, then loosen. - Tags: atom/literature/h1 - Definition of done in hand, I tell the agent to loop on it. Change something, measure, keep or revert, go again. Doesn't have to be one tiny edit each time. The step just has to be measurable against the target. I care most about the stop conditions, which might be ([View Highlight](https://read.readwise.io/read/01kw3ds0qtbw5xnvdfqs0n49fx)) - Note: Each step must be measurable against the target; stop conditions matter most. - Tags: atom/literature/h2 - So the agent gets a notify path (MCP plus /notify) and reaches me there. Usually Slack, because that's where everything else already is. Same setup could be iMessage or whatever. I treat it as a generic notification channel, not full Slack access for the agent. Status updates and "I need a decision" show up like normal messages. When I answer, that reply is the next thing the loop runs on. ([View Highlight](https://read.readwise.io/read/01kw3ds4xr312vjtyep24msgrt)) - Note: Human-in-the-loop via a notify channel (Slack/iMessage); your reply becomes the loop's next input. - Tags: atom/zettel/z1 - Once a loop is running, I start another. Usually three or so, sometimes five. And that's only the long loops. I often have other agents up at the same time on shorter work: a PR, a one-off investigation, something that isn't a multi-hour hill climb. If things are quiet I fire off another. If three are waiting on me I stop starting stuff and go review. ([View Highlight](https://read.readwise.io/read/01kw3ds338gd8x2x7gqszyy9n3)) - Note: Parallel loops with a WIP cap: ~3-5 long loops; when 3 wait on you, stop starting and review. - Tags: atom/literature/h2