Human in the /Loop  🔗 **Source:** [Human in the /Loop](https://twitter.com/ericzakariasson/status/2070493377267646797/?rw_tt_thread=True) > [!note]+ Document Notes > --- > tags: [loop-engineering, ai-agents, agentic-workflows, human-in-the-loop, definition-of-done, async-development] > author: eric zakariasson > source: Twitter/X > reference: https://twitter.com/ericzakariasson/status/2070493377267646797/?rw_tt_thread=True > date created: 2026-06-26 09:00 > --- > # 🏷️ Title: Human in the /loop > ## 🧭 Table of Contents > - [📝 Summary](#-summary) > - [📖 Main Content](#-main-content) > - [🗒️ Notes](#️-notes) > - [📚 References](#-references) > - [🗣️ Paraphrase](#️-paraphrase) > - [📰 Newsletter](#-newsletter) > - [✅ Action Items](#-action-items) > - [🧪 Hypotheses](#-hypotheses) > - [📊 Methodology](#-methodology) > - [❓ Questions](#-questions) > - [🔮 Future Research](#-future-research) > - [💡 Implications](#-implications) > - [🧠 Learning Styles Assessment](#-learning-styles-assessment) > - [📊 Accessibility Assessment](#-accessibility-assessment) > - [🌍 DEIB Assessment](#-deib-assessment) > - [✅ Logical Validity](#-logical-validity) > - [🛡️ DARVO Analysis](#-darvo-analysis) > - [🎯 Conclusion](#-conclusion) > ### 🔗 References > - [eric zakariasson — "Human in the /loop" (original thread)](https://twitter.com/ericzakariasson/status/2070493377267646797/?rw_tt_thread=True) > - Andrej Karpathy's "autoresearch" framing for ML training loops (referenced inline, no link in source) > - Playwright (browser automation, referenced as a QA tool) > - MCP plus a `/notify` path routed to Slack (referenced as the notification channel) > ## 📝 Summary > - A practitioner describes how he runs several agent coding tasks in parallel while doing other work, showing up only when a run finishes or needs a decision. > - It matters because it reframes "agentic coding" from babysitting one prompt into operating a small fleet of self-scoring loops — the human becomes a reviewer and escalation target, not a driver. > - Key stakeholders: developers using coding agents, tooling vendors who ship loop and notify primitives, and anyone building maker/checker automation. > - Risks and tradeoffs: loops only work when "done" is genuinely measurable; a weak definition of done lets a loop hill-climb toward the wrong target, and running three to five at once raises review-queue and context-switch load. > - Bottom line: write the check first. The whole method rests on a definition of done the agent can score by itself, a loop that stops on clear conditions, and pings so the human leans in at the right moment. > ## 📖 Main Content > - The core argument: the value in current agent coding is parallelism with low attention cost. You start a long-running task, walk away, and return when it finishes or gets stuck. > - The enabling move is a "definition of done the agent can score" — an objective check the agent can run without you. Examples given: a target eval score for model work, a QA pass (load the page or run Playwright, screenshot, confirm behavior) for web/UI, a green test suite for backend/refactor, a number like p95 or a benchmark for speed/flakiness, and a zero-failures count for data/content cleanup. > - He frames writing the loop as "writing how you'd check the work yourself" — the check is the spec. He starts more explicit than feels necessary, then loosens once he sees what the model can infer. > - The loop is: change something, measure, keep or revert, repeat. The step doesn't have to be tiny; it just has to be measurable against the target. He cares most about stop conditions: metric hits target, no improvement after a few tries, out of ideas, or blocked/unsure (stop and ask). > - Instead of babysitting, the agent gets a notify path (MCP plus `/notify`), usually to Slack because that's where everything else lives. He treats it as a generic notification channel, not full Slack access. His reply becomes the next thing the loop runs on. > - It runs in the cloud so loops persist for hours without his laptop open; his own client acts as orchestrator and fans work out to cloud agents. > - Then he starts the next loop — usually three, sometimes five long loops, plus shorter agents on PRs or one-off investigations. If three are waiting on him, he stops launching and goes to review. > - Uncertainty/contested: "loosen the prompt once you see what it can infer" is judgment-heavy and unmeasured; there's no data on failure rates or how often a loop hill-climbs to a bad target. The prompt template image ("/loop drives the iterations and /notify keeps me posted") couldn't be imported, so the exact prompt shape isn't in the text. > ### 🗒️ Notes > - The unit of work is a *verifiable* task, not a prompt. If you can't state the check, you can't loop it. > - Five concrete done-definitions map cleanly to maker/checker: eval score, QA/Playwright pass, green test suite, perf number under a line, validation count to zero. > - Stop conditions are first-class. "Blocked or unsure → stop and ask" is the human-in-the-loop hook; the human's answer is the loop's next input. > - Notification is deliberately thin: a generic channel (Slack/iMessage), not agent access to the whole tool. Status updates and "I need a decision" arrive as normal messages. > - Concurrency is throttled by the human's review capacity, not by the machine: three waiting → stop starting, go review. > - Cloud execution decouples loop lifetime from the laptop; the local client is just the orchestrator. > - Explicit nod to Karpathy's autoresearch for ML training loops as the closest prior pattern. > ## 📚 References > - [eric zakariasson — "Human in the /loop"](https://twitter.com/ericzakariasson/status/2070493377267646797/?rw_tt_thread=True) > - [Playwright](https://playwright.dev/) — used for the web/UI QA check > - Model Context Protocol (MCP) — the transport behind the `/notify` path > ## 🗣️ Paraphrase > - The best part of coding with agents now is being able to leave a few runs going and keep working; you show up only when something finishes or needs a call. The setup is a definition of done the agent can score, a loop that runs until it should stop, and pings so you know when to lean in. > - Before a long task, lock a check the agent can verify: a target score, a QA/Playwright pass, a green test suite, a perf number, or a validation count to zero. Writing the loop is mostly writing how you'd check the work yourself — start explicit, then loosen. > - Wrap the check in a loop (change, measure, keep or revert, repeat) and care most about the stop conditions. Give the agent a thin notify path to Slack so it pings you for status or a decision; your reply is what it runs next. Run it in the cloud, then start another loop — about three at a time, and when several wait on you, stop starting and go review. > ## 📰 Newsletter > - Not applicable to this source — the thread is about general agent-coding loops with no WordPress hook. Light bridge for the .org crowd: the same "definition of done the agent can score" discipline maps onto OSS automation — a green PHPUnit suite, a passing Playwright e2e run, or a Lighthouse/perf number under a set line are all stop-conditions a contributor's agent could loop against, with a thin Slack/Matrix ping when it needs a human decision. > ## ✅ Action Items > #### DevRel > - Developer advocacy: draft a short demo showing a "definition-of-done" loop (green test suite as the target) for a coding-agent talk; measure audience adoption via a follow-up poll. > - Education: build an office-hours module on writing the check first — turn each done-definition into a hands-on exercise. > - Developer experience: ship a quickstart template pairing a `/loop`-style runner with a `/notify` webhook so devs copy a working stop-condition setup. > - Developer marketing: content series on "agent fleets, not single prompts" with attribution to which done-definition drove the most engagement. > - Community management: open a thread asking practitioners which stop conditions they actually use. > - Internal/external/cross-project: align on a shared notify-channel convention so agent pings don't fragment across tools. > #### OSPO > - Policy & Governance: set a contribution policy that agent-submitted PRs must carry a machine-checkable definition of done. > - License Compliance: ensure looped agents run a license scan as one of their pass/fail checks. > - Security Management: require a security check (SAST pass) as a stop condition before  What I like most about coding with agents right now is the room to leave a few runs going and still get on with other work. When something finishes or needs a call, I show up. This post is a short explainer of the setup I use, a definition of done the agent can score, a loop that keeps going until it should stop, pings so I know when to lean in.  #### Find something the agent can verify Before kicking off a longer running task, I lock a definition of done. Examples I actually use: * **Model or eval work.** Target is a score. Change the approach, run the eval, keep the change only if the number moved the right way. Closest to [Karpathy's autoresearch](https://github.com/karpathy/autoresearch) for ML training loops. * **Web app or UI.** Target is a QA pass. Load the page or run Playwright, screenshot it, make sure it still does the thing. * **Backend or refactor.** Target is the test suite. Failing tests first, then green, and it has to stay green. * **Speed or flakiness.** Target is a number (p95, a benchmark). Change and measure until you're under the line you set. * **Data or content cleanup.** Target is a count. Loop until zero rows fail validation, or every item passes the check. Writing the loop is mostly writing how you'd check the work yourself. Some tasks need every step on the page. Others I give a goal and a rough direction and let the model fill in the middle. I start more explicit than I think I need, then loosen it once I see what it can infer. #### Wrap it in a loop Definition of done in hand, I tell the agent to loop on it. Change something, measure, keep or revert, go again. Doesn't have to be one tiny edit each time. The step just has to be measurable against the target. I care most about the stop conditions, which might be * Metric hits the target * No improvement after a few tries * Out of ideas * Blocked or unsure (stop and ask) #### Get pinged instead of babysitting So the agent gets a notify path (MCP plus /notify) and reaches me there. Usually Slack, because that's where everything else already is. Same setup could be iMessage or whatever. I treat it as a generic notification channel, not full Slack access for the agent. Status updates and "I need a decision" show up like normal messages. When I answer, that reply is the next thing the loop runs on.  #### Run it in the cloud Most of this doesn't stay on my laptop. It runs in the cloud so a loop can keep going for hours without my machine being open. I use my own client as the orchestrator and fan work out to cloud agents from there.  #### Then start the next one Once a loop is running, I start another. Usually three or so, sometimes five. And that's only the long loops. I often have other agents up at the same time on shorter work: a PR, a one-off investigation, something that isn't a multi-hour hill climb. If things are quiet I fire off another. If three are waiting on me I stop starting stuff and go review.  #### Prompting Rough template of how i prompt. /loop drives the iterations and /notify keeps me posted Some content could not be imported from the original document. [View content ↗](https://twitter.com/ericzakariasson/status/2070493377267646797/?rw_tt_thread=True) If you're running loops, I'd love to hear how we can make it easier for you!, Human in the /Loop, eric zakariasson, 01kw33y9t82s93d0y2jcxpqq13.