Building Your Own Software Factory — David Crawshaw  🔗 **Source:** [Building Your Own Software Factory — David Crawshaw](https://www.youtube.com/watch?v=bOihWL41m8I) > [!note]+ Document Notes > tags: [software-factory, ai-agents, personalized-software, open-source, developer-tools] > author: David Crawshaw > source: Dev Rel (YouTube) > reference: https://www.youtube.com/watch?v=bOihWL41m8I > date created: 2026-10-15 17:47 > > > [!info] Generated analysis — not the author's words. Sections below are machine-written > > from the captured source. Only text inside `> [!quote]` blocks is verbatim from the author. > > Do not quote or attribute any other line to the named author. > > # 🏷️ Title: Building your own software factory — David Crawshaw > <!-- deep-read:sections start --> ## 🧭 Table of contents - [📝 Summary](#-summary) - [📖 Main content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning styles assessment](#-learning-styles-assessment) - [📊 Accessibility assessment](#-accessibility-assessment) - [🌍 DEIB assessment](#-deib-assessment) - [✅ Logical validity](#-logical-validity) - [🛡️ DARVO analysis](#️-darvo-analysis) - [🎯 Conclusion](#-conclusion) - [🔄 Refactor appendix](#-refactor-appendix--building-your-own-software-factory--david-crawshaw) ## 📝 Summary - **What happened.** The Dev Rel channel published a 42:16 recording (uploaded 2026-10-14) of David Crawshaw, co-founder of Tailscale and CEO of exe.dev, presenting at DevRelCon 2026 in NYC. Crawshaw builds a "software factory" live-in-slides: he starts from an open-source web agent (called "Shelly" in the auto-transcript), then adds prompts and timers until the agent spawns one VM per task, studies CI flakes each night, triages GitHub issues each hour, and rebases his local changes onto upstream every night. A 17-minute Q&A follows. - **The claim.** A software factory is a bug tracker, an agent, a code review system, and CI wired to version control and production. The talk's real thesis, which Crawshaw says he found while writing it, is different: because an agent can both modify a program and keep that modification current against upstream, software written for yourself no longer decays. His conclusion is that dev tools must be open source so users can modify them, and that most configuration systems become unnecessary. - **The mechanism.** Two prompts carry the argument. One tells the agent to change a program. The other sets a nightly cron job (he says "systemd timer") that fetches upstream and rebases the local commits. He reports running the second for about a week (00:17:16). - **Why it matters.** If personal forks stay cheap to maintain, plugin systems, config files and extension APIs lose their purpose, closed-source tools become a liability, and product value moves toward services (payroll correctness, git storage) with thin user interfaces on top. - **Key stakeholders.** exe.dev (employer, product placement, maker of the agent in the demo), tool vendors who ship config-heavy or closed-source products, open-source maintainers who may receive fewer upstream patches, and developer-tool teams deciding whether their APIs suit agents. - **Risks and tradeoffs.** Every number is self-reported from one person's workflow: about $5 a day, a useful commit about 50% of the time, one commit per week. Agents with broad credentials deleted a VM in a colleague's factory. Crawshaw says vibe-coded software is "not fine as software you ship to customers," so the thesis is scoped to personal software. The Q&A answer on open-source health is candid but unsupported by data. - **Bottom line.** The architecture (agent plus timers plus a scoped credential) is concrete and reproducible. The prediction that personalization will replace configuration is plausible, rests on anecdote, and comes from the speaker of a company that sells the VMs these agents run in. ## 📖 Main content **Definition, offered as provisional.** Crawshaw asks the room who builds with a software factory (about five hands) and says nobody knows what the term means. His working definition at 00:01:35: a bug tracker, an agent, a code review system and CI, hooked to version control and connected to production. He calls it "all of our dev tools melted in a pot with an agent in the loop." He says this is not the final definition. **The build.** He starts from Shelly, the web agent that ships in the default exe.dev VM image, for two reasons: he uses it daily and it is open source. He suggests the Pi agent as an alternative starting point because it is deliberately small. The factory is assembled with prompts, shown as slides because live demos mean waiting on an agent: - Ask the agent to clone its own source, build it and replace the installed copy, so it can edit itself. - Give it a tag-scoped SSH key that can create and delete VMs but only those carrying the tag. The scope exists because an agent in a colleague's factory finished a task, decided it was done, and deleted its own VM. exe.dev then spent about 20 minutes of log analysis treating it as an infrastructure outage. - Rework the agent so each conversation (renamed "task") gets its own VM, with the git repository injected, and the VM deleted when the task is archived. - Keep an escape hatch: requests about the factory itself run as a local conversation, so the agent can't brick the tool it is editing. - Strip UI features, silence VM notifications (he got about 50 phone pushes in a row), and sync the repo before each task. **The timers that make it a factory.** A nightly job studies 48 hours of CI history, picks the top two flaky tests and starts a task for each. An hourly job scans the issue tracker, starts tasks only for issues "written in such a way that it could be solved with a code change," and feeds updates into existing tasks. His tuning rule: every wasted piece of agent work is a bug in the prompt, because it interrupted you. **The new part.** A nightly job fetches upstream Shelly and rebases all local commits on top. The agent keeps getting new models and UI fixes without the owner paging the code back into memory. Crawshaw ties this to a Tailscale-era observation: most engineers he interviewed had no software they wrote for themselves, because maintenance cost exceeded return. That cost is what changed. **Implications he draws.** Configuration becomes source code, but only when the tool is open source. An agent can change a font or add a "run obscure test suite" button without a settings system and without the old objection that you must carry a patch across machines and versions. So he now sees no alternative to open-source dev tools, and says the reason he gave for open-sourcing Shelly (everything in your VM should be yours) is something he is "going to retcon" with this one. **Q&A.** Questions cover upstreaming (he won't upstream personal changes and says they have no value to others), the health of open source (a mess before agents, and the long tail of three-star projects is what will shrink), product strategy (products shift toward correctness-bearing APIs with minimal UIs), design docs (rare, used above roughly three distributed processes, and prototypes replace them), API design for agents (a token-count A/B test favored a human-style SSH-command API over REST) and language choice (he declines to recommend; two defensible strategies are to meet the tool where it is or to pick architectural trade-offs like SQLite versus Postgres). **Contested or unverifiable.** The $5 a day, 50% hit rate, one-week rebase run and 10-person company size are single-source. The REST versus SSH-command token test is described without task list, model, or counts, though Crawshaw says a blog post is planned. The claim that agents maintain forks reliably over time rests on one week of operation and on his statement that he has seen counterexamples where agents led him wrong (00:32:56). **Transcript quality.** The auto-captions mangle several proper nouns: "XEVM," "XDEV," "EXE," "XI" and "exev" all appear for exe.dev; "chron" is cron; "PI agent" is likely the Pi coding agent; "diffs.com" belongs to the Pierre Computing Company; "IP FOU" is probably the Linux FOU tunneling subsystem; "SQite" is SQLite. "Shelly" is kept as transcribed. Verify against exe.dev's own docs before citing the agent's name. ## 🗒️ Notes #### ⏱️ [00:00:08](https://www.youtube.com/watch?v=bOihWL41m8I&t=8) Framing and disclaimers - Last talk of the day at DevRelCon. Crawshaw says his talks are new ideas each time and "not polished." - Says the software factory is the stated topic but, while writing, "found something much more interesting to say." - Audience poll: about five people use a software factory. #### ⏱️ [00:01:35](https://www.youtube.com/watch?v=bOihWL41m8I&t=95) What a software factory is - Term is "nebulous"; it names the shift agents caused to the whole development life cycle. - Working definition: bug tracker, agent, code review, CI, version control, a path to production. - Stated aim: re-evaluate 30 years of programming common sense piece by piece. > [!quote] [00:02:17] > what if we took all of our dev tools and melted them in a pot into a big mix with an agent in the loop #### ⏱️ [00:03:42](https://www.youtube.com/watch?v=bOihWL41m8I&t=222) Shelly and exe.dev - exe.dev sells "full stack durable sandboxes," which he glosses as VMs with the repeated setup steps done ahead of time. - Default image includes a web agent so you can program from a phone. This was the fourth agent the team built; he says agents are a weekend project because the models do the work. - Starts from Shelly because he uses it and because it is open source. Suggests Pi as a small alternative. - Skipped the live demo because "no one wants to sit waiting for an agent." #### ⏱️ [00:06:31](https://www.youtube.com/watch?v=bOihWL41m8I&t=391) Root access, scoped keys and the self-deleting VM - First prompt: clone the source, build it, replace the default install. - Agent has root by default. He argues tools should not be withheld, and that the safeguard is keeping nothing dangerous in the VM. > [!quote] [00:06:31] > if you're not giving your agent root and you're writing computer programs, you're making a mistake. - Tag-scoped SSH key: can add and remove VMs carrying the tag, can't reach itself or other VMs. The feature exists because a demo factory deleted its own VM after announcing it was done (00:07:58). #### ⏱️ [00:08:41](https://www.youtube.com/watch?v=bOihWL41m8I&t=521) One VM per conversation - Each conversation becomes its own VM; git integration injects the repo. He doesn't tell the agent how and it picks SCP or a clonable remote. - Mid-edit he realized the agent would stop being able to edit itself, and sent a second prompt that routes factory-related requests to a local conversation. It landed in time. #### ⏱️ [00:10:06](https://www.youtube.com/watch?v=bOihWL41m8I&t=606) Prompting as discovery - Prompting is "a process of discovery": try it, see what happens, try again. - Cleanup prompts: stop VM notifications (50 pushes in a row), remove roughly 20 UI features, delete VMs on archive, rename conversations to tasks. - Hours disappear when you "keep going" on a tool built for yourself and don't read the code. #### ⏱️ [00:12:14](https://www.youtube.com/watch?v=bOihWL41m8I&t=734) Prototype baseline - A test task asks the factory to evaluate an open-source project for another language. The agent creates a VM, injects the repo, runs a nested agent and extracts the conversation. - He calls this ordinary exploratory programming, "not anything radical and new." #### ⏱️ [00:12:59](https://www.youtube.com/watch?v=bOihWL41m8I&t=779) Nightly CI flake timer - Prompt: each night study 48 hours of CI history, find flaky tests, take the top two, start a task for each. - Morning result: two commits at the top of the factory with the flake count (example: seven failures in 48 hours) and a proposed fix. Delete what is bad. > [!quote] [00:13:42] > all the work has been done for you and is waiting for you - Argument for scheduling: typing the prompt yourself costs a 45-minute agent run you will abandon for another task, so the review never happens. #### ⏱️ [00:15:08](https://www.youtube.com/watch?v=bOihWL41m8I&t=908) Cost and hit rate - Runs for a few weeks. He reviews it about twice a week, finds a push-worthy commit about 50% of the time, so about one deflaked-test commit per week. - Cost: about $5 a day on a frontier model. He considers it worth it because he spends his days on the phone with customers. #### ⏱️ [00:15:50](https://www.youtube.com/watch?v=bOihWL41m8I&t=950) Hourly issue triage and tuning - Hourly job reads the issue tracker (GitHub or Jira), starts tasks only for issues solvable by a code change, feeds changes into existing tasks. - The gate ("could be solved with a code change") makes the model decline most issues, which saves time. - Tuning rule at 00:16:33: each wasted task is a bug because it pinged you; adjust the prompt until fewer arrive. #### ⏱️ [00:17:16](https://www.youtube.com/watch?v=bOihWL41m8I&t=1036) Nightly rebase on upstream - This is the "something new in my mind." A cron job fetches upstream Shelly and rebases all local commits. He has run it for a week. > [!quote] [00:17:58] > agents don't aren't just really good at helping you make a mess with a prototype. They're good at maintaining the mess for you. - Result: the local agent gets new models and UI fixes without drifting. #### ⏱️ [00:18:40](https://www.youtube.com/watch?v=bOihWL41m8I&t=1120) Why engineers don't keep software they write - Tailscale-era interviews (about five years earlier): most engineers had no self-written software in daily use. Crawshaw didn't either. - Reason: maintenance return didn't justify the cost, for example after a macOS release changed APIs (he has lived through this four or five times). > [!quote] [00:19:21] > there there is no work in maintaining this stuff #### ⏱️ [00:20:03](https://www.youtube.com/watch?v=bOihWL41m8I&t=1203) Configuration becomes source code - Vim and Emacs ship config files with thousands of settings, extension systems and plugin APIs to enable customization. - With source plus an agent, you ask for a font change. The old objection (carry the patch, handle version skew) is absorbed by the nightly rebase. > [!quote] [00:20:46] > an enormous amount of what was traditionally shipped configuration uh is now something you can just one-off hack up yourself #### ⏱️ [00:21:31](https://www.youtube.com/watch?v=bOihWL41m8I&t=1291) Closed source becomes a problem - With closed source you rely on the vendor's config or extension system. "That was fine before we had agents 18 months ago." - He retcons the open-sourcing of Shelly: it was originally a "you should own what runs in your VM" principle, and is now a necessity. - Example local change: a "run obscure test suite" button that calls a custom test server. > [!quote] [00:22:13] > the conclusion of those two things is I need my dev tools to be open source #### ⏱️ [00:22:57](https://www.youtube.com/watch?v=bOihWL41m8I&t=1377) Age of personalized software, scope limit - Suggested blog title: "the age of personalized software." - Scope: vibe-coded software is fine for programs you run locally just for you and "not fine as software you ship to customers." #### ⏱️ [00:23:39](https://www.youtube.com/watch?v=bOihWL41m8I&t=1419) Q&A: does open source stop growing? - Questioner asks whether nobody upstreams after everyone personalizes. Crawshaw says he won't upstream personal changes. - His history: open-source contributions since the late 1990s, a BSD device driver, years on the Go team at Google. - Position: open source "was a mess before agents" because large companies and individuals want different things, and some code is a marketing goal. > [!quote] [00:25:06] > it's not going to get worse uh because it's already terrible #### ⏱️ [00:25:47](https://www.youtube.com/watch?v=bOihWL41m8I&t=1547) Building blocks survive - Open-source libraries are the bright future. Example: the Pierre Computing Company's diffs library (written "diffs.com" in the transcript), used in Shelly and elsewhere inside exe.dev. - He would use their library but not a finished diff-viewer app, because an app is "never going to be quite the right shape." - Local personalization removes noise from "the very long tale of three stars on GitHub open source projects" and doesn't touch healthy projects. His Linux kernel changes at exe.dev stay private on host and guest because of the GPL boundary. #### ⏱️ [00:28:40](https://www.youtube.com/watch?v=bOihWL41m8I&t=1720) What products become - Cost changes imply product-market changes. DoorDash shipping a CLI points in a direction, though he doesn't claim it was strategic. - His example: payroll and HR software for a 10-person company has unusable interfaces. He wants an API that makes payroll-tax correctness across 50 US states clear, plus a minimal UI he writes himself. - Services that carry correctness or scale remain (a git storage system can be vibe-coded but has scaling problems a vendor already solved). #### ⏱️ [00:30:50](https://www.youtube.com/watch?v=bOihWL41m8I&t=1850) Why there is no design step - Asked why the factory has no design stage: he rarely designs. - Distributed systems are the limit. At about three processes that are hard to test, he writes architecture docs first. Current example: static IPs across exe.dev's global load balancer and VM hosting, unimplemented after weeks. - An agent helping with the doc surfaced Linux kernel networking subsystems he didn't know (the transcript's "IP FOU"). #### ⏱️ [00:32:56](https://www.youtube.com/watch?v=bOihWL41m8I&t=1976) Build one to throw away, finally cheap - He always said "build one and throw it away" but usually hacked the prototype into the product because typing was expensive. Now a few paragraphs regenerate the build. - Counterexamples exist where agents led him the wrong way over two years. > [!quote] [00:33:40] > writing the design doc is harder than building the prototype #### ⏱️ [00:33:40](https://www.youtube.com/watch?v=bOihWL41m8I&t=2020) Adapting existing platforms for agents - Questioner notes exe.dev's interfaces serve both humans and agents; most platforms in the room predate agents. - Crawshaw: adapting differs from building new, and clean-room rebuilds are far cheaper now, so prototype the target system from a clean slate even when you plan to adapt the old one. He doesn't know how the cost balance has shifted. - Be API-first, but the shape of future APIs is unclear to him. #### ⏱️ [00:36:39](https://www.youtube.com/watch?v=bOihWL41m8I&t=2199) The REST versus SSH-command API test - exe.dev's HTTP API is one POST endpoint that takes the same string you'd type into the SSH interface. REST was dropped because the team "couldn't make a good claim for it." - Test: prototype both, feed each set of docs and about six tasks into clean agent contexts, count tokens, check the programs worked. > [!quote] [00:38:06] > the one that was more for humans and wasn't so restlike uh required fewer tokens for the agent to write code to do - Takeaway: agent-friendliness can be A/B tested by token count plus correctness. A blog post is promised. #### ⏱️ [00:38:48](https://www.youtube.com/watch?v=bOihWL41m8I&t=2328) Languages and databases for personalized software - He declines to recommend languages: engineers can't be objective, and he has seen no objective data on what models are good at, which also changes as training shifts. - Three options: use what you know (you can read the code), meet the tool where it is (modify a Python program in Python), or pick what models handle best (he distrusts claims here). - His one concrete tip: choose architecture trade-offs such as SQLite (file-based, embedded) versus a Postgres or MySQL service, and say nothing on Postgres versus MySQL. - Ends at 00:41:43 with applause. ## 📚 References The transcript itself contains no links; the following come from the video description or are primary sources for named entities. - [Building your own software factory — David Crawshaw (Dev Rel)](https://www.youtube.com/watch?v=bOihWL41m8I): the source video, 42:16, uploaded 2026-10-14. - [DevRelCon 2026, New York City](https://nyc.devrelcon.dev/): the conference where the talk was presented, linked in the video description. - exe.dev: Crawshaw's company and the host of the demo agent. The description names it; no product URL appears in the description, so confirm from the company's site before citing. - Tailscale: co-founded by Crawshaw (the description says so). Named at 00:17:58 for the interview anecdote. - Shelly: the open-source web agent used in the demo, spelled as the auto-captions render it. Find the official repository before citing a spelling or license. - Pi agent: the small open-source coding agent Crawshaw recommends as an alternative starting point (00:05:06). No link in the transcript. - Pierre Computing Company diffs library: named at 00:25:47 as the diff renderer used in Shelly. The transcript says "diffs.com"; verify the domain. - GPL: the licence reason Crawshaw gives for the exe.dev kernel changes he would otherwise have to open-source (00:27:57). - Second-system syndrome: the classic software-engineering problem he cites at 00:35:11 (Fred Brooks, *The Mythical Man-Month*, is the origin; Crawshaw doesn't name it). ## 🗣️ Paraphrase Crawshaw begins by admitting nobody agrees on what a software factory is, then proposes a working version: take the tools you already use for tracking bugs, reviewing code and running CI, connect them to version control and production, and put an agent in the middle. He builds a small one from Shelly, an open-source agent that runs inside the VMs his company sells. Most of the steps are single prompts. Make the agent rebuild itself. Give it a key that can create and delete only the VMs it creates. Give every task its own VM. Each time he saw something he didn't like, he typed another prompt. The useful part is the timers. A nightly job studies recent CI failures and drafts fixes for the two flakiest tests, so the review sits ready by morning instead of depending on him remembering to ask. An hourly job reads the issue tracker and starts work only on issues that a code change can fix. He reports about $5 a day in model cost and a worthwhile commit about half the times he looks. Then he reaches the point he says he didn't plan to make. A third timer rebases his local changes onto upstream each night, so the personal version of the agent doesn't rot. Engineers have rarely kept software they wrote for themselves because maintenance outweighed the benefit. If an agent absorbs the maintenance, a personal fork is cheap, and a config file with ten thousand settings is a poor substitute for the source code. That only works for open-source tools, so he now wants his dev tools open. He limits the claim to code you run for yourself, not code you ship to customers. In the Q&A he says open source was already a mess, that its long tail will shrink while building blocks like a diff library thrive, that products will shift toward services with trustworthy correctness and thin interfaces, that prototypes have replaced most design docs, and that a token-counting test found an agent used fewer tokens on a human-friendly SSH-command API than on REST. He declines to pick a favorite language. ## 📰 Newsletter **Your maintenance bill just changed.** At DevRelCon NYC, Tailscale co-founder David Crawshaw showed how he keeps a personal fork of an open-source coding agent current: one prompt to modify it, one nightly cron job to rebase it on upstream. His claim is that agents turned the old objection to personal forks, "I'll have to maintain this forever," into a scheduled job. He reports about $5 a day in model spend, a useful commit about 50% of the time he checks, and a fork that has stayed current for a week. Those numbers are one person's workflow, not a study. Where this lands for WordPress: nearly every WordPress site is a pile of configuration, from filters and `theme.json` settings to plugin option screens, built to avoid forking code. If personal forks stay cheap, the pressure on plugin settings pages and extension points changes. The same logic raises a question for the Plugin Directory and the block ecosystem about what a "customization surface" is for. Crawshaw's other reusable idea is operational: a nightly job that drafts fixes for the top two flaky tests, so review waits for you in the morning. He also shares a test worth stealing. exe.dev built a REST API and a one-endpoint command API, fed each one's docs to an agent with the same tasks in clean contexts, and counted tokens. The human-readable command API used fewer. Run the same check on your own docs. Links: [the talk](https://www.youtube.com/watch?v=bOihWL41m8I), [DevRelCon NYC](https://nyc.devrelcon.dev/). ## ✅ Action items #### DevRel - Developer advocacy: build a talk or demo around one scheduled-agent timer (the nightly CI-flake job) and publish the daily cost and the review hit rate, not a claim about productivity. - Education: run a workshop on tag-scoped credentials for agents, using the self-deleting VM story at 00:07:58 as the opening failure case. - Developer experience: test one docs set for agent-friendliness using Crawshaw's method (same task list, clean context, count tokens, confirm the program works) and publish the result. - Developer marketing: don't reuse "$5 a day" or "50% useful" in outside material. Cite them as one speaker's reported workflow. - Community management: add a note to any link-sharing about this talk that the speaker runs a company selling the VMs the agent runs in. - Internal/external/cross-project: ask exe.dev for the promised token-count blog post and the Shelly repository before building anything on the demo. #### OSPO - Policy and governance: write down whether contributors may run private agent-maintained forks of project tooling, and what is expected upstream when they find a bug. - License compliance: check the GPL boundary Crawshaw describes at 00:27:57 (private kernel patches kept on host and guest). Add a note on when private patches cross into distribution. - Security management: model the agent-with-root pattern. Root plus a broad credential produced a deleted VM; add scoped-credential requirements for any automation that can create or delete infrastructure. - Software inventory: if a nightly rebase job runs in your org, list it as a service with an owner, a cost line and an off switch. - Developer support: define who reviews the output of scheduled agent jobs and by when, so output doesn't pile up. - Community engagement: track whether personalization reduces upstream patch volume for your projects; baseline the current rate first. - Strategic alignment: add "cost per accepted agent commit" as a metric. Crawshaw's implicit figure is about $5 a day divided by about one commit per week. #### Community - Maintainers: expect fewer drive-by config-flag requests if users can patch locally. Decide whether to document "fork and rebase" as a supported path. - Contributors: add a "personal fork, don't upstream" label so contributors know a local change is acceptable and not a failed PR. - Sponsors: report agent-run maintenance spend (tokens) as a line in sponsor impact reports. - Ecosystem: run a cross-project drill where a bot rebases a local patch set on a new upstream release and reports conflicts. #### WordPress - Core: no direct action. Nearest relevance: how many Core settings exist only to avoid forking code. - Core Performance: the nightly CI-history study maps onto the performance test suite. Pilot flake triage on one performance job. - Core AI: write guardrails for scheduled agent tasks (scoped credentials, tag-limited deletion, human review before merge), using the 00:07:58 incident as a test case. - **Core Program Team:** record a decision on whether agent-drafted flake fixes enter normal review queues, and name the owner. - Mobile: Crawshaw programs from a phone through a web agent (00:04:25). Evaluate what a phone-first agent workflow means for the mobile app backlog. - Playground: Playground already gives disposable environments. A one-environment-per-task pattern, with cleanup on archive, is a direct parallel to the VM-per-task design. - Community: no direct action from this source; nearest relevance: meetup talks on scheduled agents. - Training: draft a lesson on "give each agent task its own sandbox and delete it when archived." - Marketing: no direct action from this source; nearest relevance: the "age of personalized software" framing as a post angle, credited and marked as one speaker's view. - Support: the hourly issue-triage prompt ("could be solved with a code change") is a template for screening forum threads. Pilot on a read-only basis. - Design: no direct action from this source; nearest relevance: personal UI customization without settings screens. - Documentation: run the token-count test on one DevHub reference page pair to see which style agents consume cheaper. - Photos: no direct action from this source. - TV: no direct action from this source. - Polyglots: no direct action from this source; nearest relevance: agents rebasing local translation patches onto upstream strings. - Accessibility: no direct action from this source; nearest relevance: personalizing UI for accessibility needs without waiting on a settings option. - Meta: audit the Make and Trac issue queues for which issues "could be solved with a code change," to size the hourly-triage pattern. - CLI: the exe.dev API is one endpoint that accepts the same string as the SSH interface. Evaluate whether WP-CLI command strings could back a similar single-endpoint API. - Hosting: the self-deleting VM incident is a cross-host checklist item: agent credentials scoped by tag or label, never account-wide. - Tide: scan results are a good nightly-study target. Test a job that groups recurring false positives and drafts rule changes. - Openverse: no direct action from this source; nearest relevance: nightly review of ingestion errors. - Themes: review-queue throughput could use a nightly pass that flags themes with repeat lint failures. - Plugins: same pattern, with a human gate on anything security-related. #### The WPOCC - Programs: no direct action from this source; nearest relevance: if The WP Open Community Collective funds maintainer tooling, include agent token spend as an eligible cost. - Projects: pilot a documentation sprint that measures docs by agent token cost per task, alongside reader testing. - Funding experiments: fund a small trial where a maintainer runs a nightly flake-triage agent and reports spend, accepted commits and false starts. ### FAIR - Technical independence: personal forks maintained by agents depend on open source and on a hosted model. Document that dependency explicitly in any FAIR-adjacent tooling. - Policies and covenants: no direct action from this source; nearest relevance: whether private forks fall under a package's update and provenance rules. - Productized governance artifacts: add an "agent-maintained patch set" row to operator scorecards covering who runs the rebase job and how conflicts are reviewed. ## 🧪 Hypotheses 1. A nightly agent job that drafts fixes for the two flakiest CI tests yields a push-worthy commit at least 40% of reviews in a project with a mature test suite, close to the 50% Crawshaw reports. 2. An agent-run nightly rebase of a local patch set onto upstream completes without human intervention on at least 80% of upstream releases when the patch set stays under about 20 commits. 3. Agent token cost to complete a fixed task list is lower with human-style command APIs than with REST in at least half of tested API pairs, reproducing the exe.dev result. 4. Requiring a code-change test ("could be solved with a code change") in an hourly triage prompt cuts started tasks by more than half compared with starting a task for every new issue, without missing more than 10% of issues a human would have picked up. 5. Tag- or label-scoped credentials eliminate agent self-deletion and cross-resource deletion incidents in a controlled test with injected "I'm done, clean up" behavior. 6. Open-source projects with large settings surfaces see a measurable drop in new feature-flag requests once users adopt agent-maintained local forks. ## 📊 Methodology - **H1.** Pick 5 repositories with at least 6 months of CI history. Run the nightly flake prompt for 30 days. Count commits a maintainer would merge, record cost per day and review time, and calculate the hit rate with a confidence interval. - **H2.** Take 10 open-source tools with active release cadences. Apply 5, 10 and 20 local commits and let an agent rebase across the last 10 upstream releases. Record success without intervention, conflict types and failures that pass tests but change behavior. - **H3.** Write each API pair's docs, give agents the same 6 tasks in clean contexts, count tokens, confirm correctness by running the produced programs, and repeat across at least 3 models to separate model effects from API effects. - **H4.** Run both prompts over a 90-day issue archive from one project. Compare tasks started, issues solved, and issues missed against a human-labeled "solvable by code change" set. - **H5.** Build a sandbox harness with an agent prompted to finish by cleaning up. Compare account-wide keys against tag-scoped keys over 100 runs and count deletions outside the allowed scope. - **H6.** Use the issue and PR history of 10 settings-heavy projects. Compare feature-flag request rates before and after published fork-maintenance guidance. Note that adoption of agent-run forks is not directly observable, so this needs a survey arm. ## ❓ Questions - What does the $5 a day buy in tokens, which model runs the nightly job, and how many agent runs does it start? - How was the 50% hit rate counted, over how many reviews, and what counts as useful to the person doing the pushing? - What happens when the nightly rebase hits a conflict, and who is notified? - If the upstream project changes direction, how does the personal fork decay, and how long before the agent can't reconcile? - Which of the personal changes in this talk (a test-suite button, removed UI features) would anyone else want, and how would that be known without upstreaming? - What is the token result of the REST versus SSH-command test, across which tasks, models and doc styles? - How should an agent with root be sandboxed when the VM holds credentials, not just code? - If configuration becomes source code, what happens to security updates, since a personal fork must still absorb upstream security fixes? ## 🔮 Future research - **Fork decay curves.** No dataset exists on how long agent-maintained forks track upstream before they need human intervention, split by patch size and project change rate. - **Upstream contribution effects.** Whether easier personalization reduces, redirects or has no effect on upstream patches is an empirical question the Q&A answered from experience. - **Agent-ergonomic API design.** Token cost per task is a cheap metric, but nobody has standardized tasks, doc formats or model panels for comparing APIs this way. - **Scheduled agent economics.** Cost of running a timer daily against value of the sometimes-useful output (hit rate times value of a merged change) deserves a published model. - **Credential scoping for agents.** Tag-scoped keys solved one failure. A survey of agent infrastructure incidents would show which others recur. - **Language and database choice for agent-written software.** Crawshaw has seen no objective data on which stacks models handle best (00:40:58). A benchmark would be a public good. ## 💡 Implications - **Practice.** Put the prompt in a timer. Work you would do if you remembered becomes work that is already done when you look, at the price of tokens spent on days you don't look. Tune by treating wasted runs as bugs. - **Policy.** Teams need a written rule for agent credentials: scope them to what the agent creates. The deleted-VM incident was avoidable with a per-tag key. - **Security.** A nightly rebase of local patches onto upstream also pulls in upstream code automatically. That is a supply-chain path, and it needs review gates when the agent merges without a human. - **Education.** Teach "modify, then maintain" together. The talk's lesson is that the second step, the nightly rebase, is what makes the first step worth doing. - **Funding.** If vendors expect fewer upstream patches, maintainer funding models that assume contribution volume need review, and token spend becomes a visible line item in project budgets. - **Product strategy.** Services that carry correctness or scaling (payroll tax rules, git storage) hold their value. Interface layers lose it. ## 🧠 Learning styles assessment - **Reading level estimate.** Flesch-Kincaid Grade Level of about 8 and Flesch Reading Ease of about 70, estimated by sampling transcript paragraphs (not computed in full). Spoken, informal, many false starts and parenthetical asides. Domain terms (rebase, cron, systemd, GPL, cleanroom) assume a developer audience. #### 👀 Visual learners - Present: slides of agent prompts rendered in blue, intermediate screenshots of the UI changing (00:10:48), and a before/after of the factory growing. - Improve: no architecture diagram of the final system (timers, tasks, VMs, upstream). That diagram would carry the whole talk in one frame. Add one with arrows for the nightly rebase. #### 🔊 Aural learners - Present: full narration, a conversational delivery, and audience Q&A that repeats key points in new words. - Improve: audience questions are only partly audible in the capture (the 00:23:39 and 00:28:40 questions are garbled). Add a speaker repeat of each question and a 3-minute audio summary. #### 🗣️ Verbal learners - Present: analogies ("melted in a pot," "cooking show"), concrete anecdotes (the VM deleting itself), and a stated definition at 00:01:35. - Improve: define "software factory," "systemd timer" (he glosses it as the modern fancy word for a cron job) and "rebase" once for non-specialists. #### 🧩 Kinesthetic learners - Present: the whole talk is a replayable recipe of prompts, and he suggests running the same sequence on the Pi agent in a VM (00:05:06). - Improve: no repository link or prompt file is in the transcript. Publish the prompts as a copy-paste list in the description. #### 🧠 Logical learners - Present: clear premise chain. Agents modify code and agents maintain code, so personal forks are cheap, so open source is required. He also states the limits (not for shipped software). - Improve: the conclusion rests on one week of operation. Show a table of runs, failures and costs so a reader can check the chain. #### 🧑‍🤝‍🧑 Social learners - Present: a 17-minute live Q&A with five questions, including two sharp challenges on open source and on product strategy. - Improve: no community channel is named in the transcript. Add a discussion thread for the promised token-count post. #### 🧘‍♀️ Solitary learners - Present: self-paced replication path. One person can build the factory in a weekend with a VM and an agent. - Improve: add reflection prompts such as "which software did you stop using because of maintenance cost?" which mirrors the question Crawshaw asked at Tailscale. #### 🌿 Naturalistic learners - Present: the factory metaphor and the lifecycle of VMs created and destroyed per task. - Improve: ties to sustainability go unmentioned. A daily model run at $5 and one-VM-per-task have an energy footprint worth stating. ## 📊 Accessibility assessment #### Screen reader semantics - Present: YouTube provides landmarks, and an auto-generated transcript exists. - Improve: slides carry the prompts and screenshots that the narration only partly reads aloud (00:10:06 onward). Without descriptions, 1.1.1 (Non-text Content) is unmet for slide content. #### Keyboard navigation - Present: the YouTube player is keyboard operable with visible focus, so 2.1.1 and 2.4.7 pass through the platform. - Improve: no chapters were set (the metadata chapter list is empty), so keyboard users can't jump between demo, Q&A and sections. Add chapters at the timestamps in Notes to meet 2.4.5 (Multiple Ways) in spirit. #### Contrast and readability - Present: unverifiable from a transcript. - Improve: prompts in blue on slides may fall short of 1.4.3 (Contrast Minimum) on a projected screen and in a compressed video. Check slide contrast at 4.5:1. #### Text resizing and reflow - Present: the player scales and captions resize. - Improve: text burned into slides doesn't resize or reflow. 1.4.4 (Resize Text) and 1.4.10 (Reflow) fail for that content. Provide the prompts as text in the description. #### Media and captions - Present: captions exist (auto-generated). - Improve: the captions garble the company and product names (exe.dev appears as "XEVM," "XDEV," "EXE" and "XI"), "cron" as "chron," and audience questions are partly inaudible. That's a 1.2.2 (Captions, Prerecorded) failure in substance. Human-corrected captions and a corrected name list in the description are the fix. No audio description for slide-only content (1.2.5). #### Forms and interactions - Present: no forms in the video. - Improve: the audience-engagement poll at 00:00:50 is a raised-hand poll, which is not captured for remote viewers. Offer a text-based equivalent. 3.3.2 applies to any signup page linked later. #### Standards reference - Primary gaps against WCAG 2.2 AA: **1.2.2** (inaccurate captions on proper nouns), **1.2.5** (no audio description for slide-only information), **1.1.1** (slide screenshots with no text alternative), **1.4.4 and 1.4.10** (burned-in text), and **2.4.5** (no chapters). The captions fix has the best return. ## 🌍 DEIB assessment #### Cultural breadth - Present: examples span payroll tax in 50 US states, a BSD device driver and Linux kernel networking. - Improve: every example is US or English-language tooling. Add one non-US case, such as how personalized software treats local payroll rules or non-English UIs. #### Inclusive language - Present: plain, direct second person, with no exclusionary terms found. The speaker apologizes in advance for unpolished delivery. - Improve: "vibe" and "brick my machine" are idioms that translate poorly. "Five-alarm fire" and "cooking show" are US idioms. Gloss for non-native speakers. #### Neurodiversity support - Present: a stated structure (definition, build, timers, thesis, Q&A) and demos that each do one thing. - Improve: the speaker warns the talk "can be a little hard to follow," and false starts and parenthetical tangents (the diff library's missing third product, 00:25:47) add load. Chapters and a one-slide recap would help. #### Representation - Present: one speaker and five audience questioners, with no demographic information in the capture. - Improve: the video shows no voices beyond a single speaker. Cite named people (colleague, Pierre team) with credit when the story depends on them. #### Psychological safety - Present: the speaker invites correction ("please correct me"), admits "I don't understand it either" and doesn't mock questioners. He volunteers failures: the self-deleting VM, the notifications flood, and the near-brick at 00:09:22. - Improve: the speaker says he won't give opinions on languages to avoid "getting me in trouble." That's honest but signals a fear of backlash. A moderated Q&A policy would reduce that pressure. ## ✅ Logical validity #### Strong reasoning - **Causal, 00:07:58.** The self-deleting VM is traced to a cause (a full-account key plus an agent that decides it is done and cleans up) and a targeted fix (tag-scoped keys that can't reach the agent's own VM). This is a clean fault-to-control chain. Strengthen with the full postmortem timeline. - **Deductive, 00:17:58 to 00:22:13.** Premise 1: an agent can modify a program on request. Premise 2: an agent can keep that modification rebased on upstream. Conclusion: personal forks no longer cost ongoing maintenance, so source access matters more than extension APIs. The inference holds if both premises do. Strengthen by reporting more than one week of operation. - **Self-limiting scope, 00:22:57.** He says vibe-coded software is not fine for shipping to customers. Scoping the claim to local software makes the thesis testable and prevents it covering safety-critical or customer-facing code. - **Empirical test, 00:37:22 to 00:38:48.** A matched comparison (same tasks, clean contexts, token count, correctness check) is a sound design for an API-ergonomics claim. It needs a published task list and models to be replicable. - **Candor on failure, 00:09:22 and 00:32:56.** Admits near-bricking the agent, the notifications flood, and counterexamples where agents misled him. Those concessions raise credibility. #### Fallacies - **Anecdotal evidence, 00:15:08.** "about 50% of the time there is a very useful commit in there." One person, no sample size, and the person who reviews is the person who judges usefulness. *Fix:* report review counts and an independent merge-rate. - **Hasty generalization, 00:19:21.** "there there is no work in maintaining this stuff." Based on a week of one agent and one upstream. Large upstream rewrites, security fixes and breaking releases aren't covered. *Fix:* qualify with the conditions (small patch sets, upstream cadence) and report failures. - **Appeal to existing badness (nirvana inversion), 00:25:06.** "it's not going to get worse uh because it's already terrible." That open source is already troubled doesn't show a new pressure won't make it worse, and the questioner asked about direction. *Fix:* answer with data on upstream contribution rates before and after agent adoption. - **Post-hoc rationalization (self-disclosed), 00:21:31.** He says he'll "retcon" the open-sourcing of Shelly as a consequence of the thesis. The disclosure is honest, but it means the thesis and the product decision are not independent evidence for each other. *Fix:* keep the original reason (ownership of what runs in your VM) and the new reason (agent customization) as two separate arguments. - **Undisclosed interest, 00:03:42 and 00:05:06.** Crawshaw runs exe.dev, the demo runs in exe.dev VMs, and the talk concludes that tools must be open source so agents can modify them, which pushes demand toward agent-ready VM hosting. He says the choice of starting point is partly an ad, which is a disclosure, but doesn't connect the conclusion to the business. *Fix:* state the relationship at the conclusion as well. - **Survivorship in the library example, 00:25:47.** One well-maintained diff library is offered as proof that open-source building blocks thrive. Unmaintained building blocks aren't counted. *Fix:* give a count or a counterexample. - **Appeal to personal experience on "design doc" costs, 00:33:40.** "writing the design doc is harder than building the prototype" holds for the speaker's mostly small systems and he concedes distributed systems are the exception, so this one is partly self-limited. #### Tone of voice - Conversational and self-deprecating. Opens with "congratulations and I'm sorry" for anyone who hasn't heard him speak, calls the talk unpolished, and says "I don't understand it either" in reply to a questioner. Candid on failures, hedged on predictions ("I genuinely don't know uh how much and in what direction," 00:35:55). The Q&A register is more relaxed than the main talk and includes jokes ("what can I say about open source code that's not going to get me in trouble?"). Claims are strongest on mechanism (timers, scoped keys) and weakest on forecasts (the fate of open source), and he mostly signals which is which. ## 🛡️ DARVO analysis Not applicable to this source. DARVO describes a response pattern to accusation, and this is a conference talk with audience questions and no allegations. The nearest challenge is the 00:23:39 question that personalization may starve upstream open source. For completeness: - **Deny:** no denial of fault. Crawshaw minimizes the concern partly ("open source is a mess and was a mess before agents," 00:25:06) but doesn't deny the mechanism, and says he won't upstream his changes. - **Attack:** none. He thanks and engages each questioner, calls the product-strategy question "the right question," and calls a design-doc question "fair." - **Reverse victim/offender:** none. He doesn't recast himself as the target of criticism, and he volunteers his own failures (the self-deleting VM, the notification flood). ## 🎯 Conclusion The durable content is a recipe with a mechanism: a self-editing agent, a per-task VM, credentials scoped by tag, and timers that run prompts on a schedule so reviewable work is waiting when you look. The deleted-VM incident at 00:07:58 and the cost figures at 00:15:08 give it enough specifics to replicate. The thesis he draws at 00:17:58 is that agents maintain personal forks, which makes open source necessary for dev tools. It follows from its premises, and the premises rest on one person's week of use. What's missing: sample sizes for the 50% hit rate, the cost method behind $5 a day, any failure log for the nightly rebase, and the token counts from the REST test. The Q&A answers on open-source health and product strategy are reasoned opinions, flagged as such by the speaker. Treat them as hypotheses. **Next steps for readers:** pick one recurring chore that you skip because starting it costs a 45-minute wait (flaky tests, stale issues, release notes) and put it on a nightly timer for two weeks. Log tokens spent, runs reviewed and runs merged. Scope any credentials to a tag. Then run the token-count test on your own docs before you redesign an API for agents. ## 🔄 Refactor Appendix — Building your own software factory — David Crawshaw > Density: 2–5 Literature atoms (H3 pad), 0–3 Permanent candidates (H4 pad). > Emitted: 4 Literature, 2 Permanent. Resonance filter applied: the language-choice and > design-doc answers were dropped as opinion without evidence. > > Konik claim-header rules: present tense, declarative, ≤10 words, searchable, unique. > > Atomicity: one declarative sentence per atom. "and"/"also" → split. > > Tag pool: #bmf · #addendum · #xref · #articleseed · #storystem · #share · #aphorisms ### 📚 Literature Split Pad #### [H1] Agents maintain personal forks, not only prototypes > [!quote] [00:17:58] > agents don't aren't just really good at helping you make a mess with a prototype. They're good at maintaining the mess for you. **Context & use**: - 00:17:58, right after the nightly upstream-rebase job is introduced as "the really interesting part." - This is the talk's central claim. Evidence behind it is one week of operation and no failure log, so record it as a reported result, not a measured one. **Tags**: #xref #articleseed <!-- atomicity: 1 idea ✓ --> #### [H2] Open source is required for agent customization > [!quote] [00:22:13] > the conclusion of those two things is I need my dev tools to be open source **Context & use**: - 00:22:13, summing up two prompts: modify the program, then keep it rebased on upstream. - Caveat: the speaker's company sells the VMs these agents run in, and he admits at 00:21:31 to retconning the reason for open-sourcing Shelly. Treat as a conclusion from premises, not a neutral finding. **Tags**: #xref #addendum <!-- atomicity: 1 idea ✓ --> #### [H3] Scheduled prompts move agent work to your schedule > [!quote] [00:13:42] > all the work has been done for you and is waiting for you **Context & use**: - 00:13:42, explaining the nightly CI-flake timer. The alternative (typing the prompt when you remember) loses to a 45-minute wait and a distraction. - Cost framing at 00:15:08: about $5 a day, about one push-worthy commit per week. Pay tokens on days you don't look in exchange for review-ready work on days you do. **Tags**: #xref #share <!-- atomicity: 1 idea ✓ --> #### [H4] Token counts measure how agent-friendly an API is > [!quote] [00:38:06] > the one that was more for humans and wasn't so restlike uh required fewer tokens for the agent to write code to do **Context & use**: - 00:38:06, result of a test comparing a REST API against a single-endpoint command API with the same tasks and clean contexts. - Method is reusable even though numbers weren't shared. Pair the token count with a check that the code worked, as he did. **Tags**: #xref #articleseed #share <!-- atomicity: 1 idea ✓ --> ### 🧠 Atomic Split Pad #### [P1] Maintenance cost, not writing cost, kills personal software **Claim** (one sentence, Courtney's voice): Most personal software dies because maintaining it costs more than the value it returns, so a tool that removes maintenance changes which software is worth writing. **Evidence pointer**: [[#H1]] **Scope & conditions**: Holds when the maintenance work is mechanical (rebasing, API drift, dependency bumps) and an agent can do it reliably. Breaks when upstream changes direction, when conflicts need design judgment, or when the evidence is only a short run. The source offers one week of operation, so confidence is low. **Links**: - [[Software maintenance economics]] — because extends - [[Plugin and settings systems]] — because contrasts on whether customization needs an official extension surface - [[Open source tooling policy]] — because applies-in decisions about whether to require source access <!-- atomicity: 1 idea ✓ --> #### [P2] Scope agent credentials to what the agent creates **Claim** (one sentence, Courtney's voice): An agent that can delete resources should hold a credential limited to the resources it created, because an agent that finishes a task will clean up everything it can reach. **Evidence pointer**: [[#H3]] · the 00:07:58 self-deleting VM incident in Notes **Scope & conditions**: Applies to any automation that can create and delete infrastructure. Does not prevent harm inside the scoped set, so it needs pairing with review of what the agent creates. Evidence is one incident, though the mechanism is easy to reproduce in a test harness. **Links**: - [[Least privilege for automation]] — because extends - [[Agent sandbox design]] — because applies-in per-task environments - [[Giving agents root]] — because contrasts on where the safety lives: in the sandbox contents, not in withheld tools <!-- atomicity: 1 idea ✓ --> ### ✏️ Fleeting Queue - [F1] Find the official Shelly repository and exe.dev docs; confirm the agent's real name and licence since the auto-captions render "Shelly," "XEVM," "XDEV," and "XI" — https://www.youtube.com/watch?v=bOihWL41m8I #fleeting #inbox - [F2] Look for the promised exe.dev blog post on the REST versus SSH-command token test and copy the task list and token counts — context 00:36:39 to 00:38:48 #fleeting #inbox - [F3] Open question: does a nightly job that rebases onto upstream also import upstream supply-chain risk without review? Context 00:17:16 #fleeting #inbox - [F4] Article seed: "The nightly rebase is the product" — personal forks become viable only when maintenance is a scheduled job #fleeting #inbox #articleseed - [F5] Ask which of my own scripts I abandoned for maintenance cost, mirroring the interview question at 00:18:40 #fleeting #inbox - [F6] Check the 00:09:22 pattern of routing factory-edit requests to a local conversation as a guard against an agent bricking itself #fleeting #inbox ### → Things 3 (max 2 actions) - [ ] Run the token-count test on one docs page pair (reference style versus task style) with a clean-context agent — docs agent-readiness #things - [ ] Draft a nightly agent job for one chore I skip, with a tag-scoped credential and a spend log — two-week trial #things <!-- deep-read:sections end --> 📄 **Full text:** [[Full Document Contents/Articles/Building Your Own Software Factory — David Crawshaw.md|Building Your Own Software Factory — David Crawshaw by David Crawshaw]] ## Highlights - [00:12:59] surprisingly small how uh uh how how the fun starts. So here's a new prompt I fed the thing. Start a set up a nightly systemd timer which is the modern fancy word for chron job. Uh I I don't know why but it's fine. uh and then have and then have that chron job run a prompt and the prompt is each night study the CI history for the past 48 hours because it has access to the CI uh look for test flakes take the top two start a task for each thing attempting to you know and attempted to flake the test this means you can come in in the morning and there's two commits sitting at the top of your factory uh which say like you know in the last 48 hours this test has flaked out seven times in CI here's a [00:13:42] potential fix. You can read it. You can decide if it's any good. If not, you just hit the delete button and you never think about it again. And if you're really busy that day, you don't you don't even bother doing that. Uh this has an enormous advantage uh which is that uh all the work has been done for you and is waiting for you because you could say to yourself, well, it's very easy to just type this prompt in every day when I have time to study the CI and try and make it better. But then you have a workflow problem which is you remember you realize you have a moment. you remember that making the CI better is a thing that you can do and is really good for your team. Uh so you type the prompt in and then the agent goes and works for 45 minutes uh with all these sub agents to try and generate these commits and you go and find something else to do and get distracted by that and do it and by the time this is ready ([View Highlight](https://read.readwise.io/read/01m50s9q83edg77cbjtxxa8wsb)) - Note: [H3] Scheduled prompts move agent work to your schedule - Tags: atom/literature/h3 - [00:17:16] work as a bug because it interrupted you and you need to go and tune the prompts so that you get fewer of these this is super useful and a huge amount of my workflow is is now tasks like this uh so that's that's what that gives us a software factory uh now this is the this is the really interesting part and this is uh uh this this is the something new in my mind. Uh let's do a nightly chron job. Uh and let's let's have it fetch upstream changes to Shelly and rebase all of our local changes on top of upstream. This works. I've had this running for a week now on the little factory I set up. And yes, uh all it it successfully takes all of those prompts I just ran all the commits that were generated from those prompts and gets them running on [00:17:58] whatever changes were upstream uh with Shelly. So my uh my agent doesn't fall behind. It gets new models injected into it. It gets uh UI fixes updated to it. Uh that is new and that changes something really important about software engineering which is that agents don't aren't just really good at helping you make a mess with a prototype. They're good at maintaining the mess for you. And so this is very important because uh traditionally software engineers and I this is this is me working off anecdotes from 5 years ago when I started Tailscale um with uh with Avery and David. We uh we'd go and talk to lots of people about the product ([View Highlight](https://read.readwise.io/read/01m50s9jfe3x0d5f13kdh2yp96)) - Note: [H1] Agents maintain personal forks, not only prototypes - Tags: atom/literature/h1 - [00:21:31] is actually open source. If you use a closed source program, then you don't actually get to have an agent modify it and uh help help you make it just for you. You're you're relying on whatever config file or extension system the company that built the the tool has shipped. Uh, and again, that was fine uh before we had agents 18 months ago. Uh, and now it's it's kind of a problem. I don't want to use a tool like that that I can't customize for myself. And this is why uh this is I'm now going to retcon this is the reason we open source Shelly. Uh because this way you can customize it to your heart's content and do whatever you want. Uh the truth is Shelly was open source because we thought everything that runs in your VM you should own. It should be yours and [00:22:13] it should be open source as part of that. Uh but now I don't see any other way than having it be open source because obviously you're going to modify your agent. Why would you leave it stock? You're going to you're going to add a button that says run now obscure test suite which contacts a server that runs the test suite on your change. Uh that's something I have in my own shell. I have a button to run our test suite because we have custom servers for that stuff. Um and so this I think is the fundamental conclusion I've come to from what is effectively two prompts. One is a prompt that it's uh it's super easy to take a program and modify it. And the second is a prompt that it's super easy to set up a nightly chron job to keep the program up to date with upstream. And the conclusion of those two things is I need my dev tools to be open source ([View Highlight](https://read.readwise.io/read/01m50s9my25cw4qm27hdf0jbtk)) - Note: [H2] Open source is required for agent customization - Tags: atom/literature/h2 - [00:37:22] We prototyped out a full rest API for exi uh because it's easy, you know, it was like a day's work to go and build one of these things um in the classic many endpoints that you can, you know, have get and post and put all these these features for. And then we prototyped out uh the one with a a single endpoint that you pass the existing command to. And one of the tests we ran on these two things is uh in fact we should write a blog post about this um is we uh uh we we structured the docs for both again they were mostly agent generated docs at first uh just while we were playing around with it. Um, we took the docs uh for the APIs and we created a list of half a dozen tasks and we took the docs uh from the REST API and the tasks and [00:38:06] fed it into clean contexts and an agent and then we took the docs from the SSH style interface with each task and fed them into clean context and then we counted the number of tokens it took an agent to complete the task using the two different APIs. uh and the the one that was more for humans and wasn't so restlike uh required fewer tokens for the agent to write code to do. Uh and so you know we said like well look this API is much nicer for humans because you can just read it but it doesn't follow best practices around rest and all this and like maybe that makes it harder for computers. Turns out that wasn't true. The agents found the the human style API easier to use and we could just test that by just counting tokens and like you know we also had to check that the programs actually worked. Uh but that you know they did. Uh these were simple ([View Highlight](https://read.readwise.io/read/01m50s9t6cy87av1cv9nfgpday)) - Note: [H4] Token counts measure how agent-friendly an API is - Tags: atom/literature/h3