Automation at the Speed of Swamp  🔗 **Source:** [Automation at the Speed of Swamp](https://share.snipd.com/episode/acc9810b-0e5a-4c55-8dab-f1fc81e5842d) ## Highlights - **Agents Building The Machines That Build Software** - AI is creating a new development camp that builds the machines that build software. - Adam Jacob calls this a third camp where agents build the software that builds the software, producing radically different workflows and experience. Transcript: Adam Jacob Yeah, not anymore. Not at all, right? Yeah. I mean, I'm bad at time. It's just a thing to know about me since we're friends. I'll get just times and dates. I double-check my daughter's birthday, even though I know her birthday. Just because I might could mess it up like I'm just bad at it. But I feel like, you know, the last eight months, six, eight months, maybe in AI, in particular for software development, really changed the shape of what was possible. And it kind of happened to everyone who was working in that zone sort of all at once. You know, like when I talked to my friends who are like high up at Google or high up at Microsoft or who are running startups or who are, you know, running big teams or who are running small Teams, you know, everybody who was really paying attention and doing that work, there was a tangible change in the outcomes and a tangible change in our ability to start to articulate What a working architecture is. And to get better and to move the industry forward in a meaningful way that's not just the robots are coming to get us and they're going to take all our gerbs and it's all going to end. That happened. It's not evenly distributed. Most people haven't experienced it. I think the repercussions are terrifying and wild. I talk about them to anyone who listened to me like all the time partly because i'm trying to like process on the street corner a little yeah you know because i'm like bible like i don't Know if you know what's happening but like oh no you know like it's gonna it's gonna be bad you know and but good good bad you know good bad and a lot of change good bad change yeah and that's All really happened in the last little bit here and then i think now we're entering an era where you you know back then i think there were like two camps who couldn't talk to each other very Well you know like the ai is stupid and everybody who's talking about it is a dumb dumb and they telling, you know, whatever they're telling lies or they're, they're saying it cause their Investors are making them or whatever the reason is. Yeah. It's a bolt on. It's just a, they have to cause it's a probability machine. They don't have choices, whatever. And then there were people who were using it and they were having trouble talking to each other cause people using it were like, this is good for me. It's helping, you know, I'm using cloud code or whatever. And it's like, and I'm more productive or it's better. And they were struggling to talk to each other. And now we have a third camp that's struggling to talk to each other, which is the version that's using AI to build the software that builds the software that they're trying to build. And that third camp where it's like, I'm building the machine with AI to build the software that is the thing I'm trying to do. That camp can't talk to either of the other two either. You know, because it's just, because it's wildly divergent. ([Time 0:07:11](https://share.snipd.com/snip/0819c05c-1c0c-4aaa-ab62-c48834414925)) - **Begin Agentic Experimentation Outside Work** - Start experimenting with agentic workflows now on your own time because most workplaces are too slow to adopt rapidly. - Learn architecture and how to communicate outcomes so you're indispensable during transition. Transcript: Adam Jacob At you like you're a crazy person. And I think it, I think it probably goes a little both ways that it's just like, Oh no, you've either had that experience or you haven't. And if you've had it, you can talk about it with other people who've had it. But you know, if you haven't had that experience, ([Time 0:11:05](https://share.snipd.com/snip/734f6fe0-8152-4c27-b39c-4421891a253d)) - **Shrank To Five And Shipped 900 Times In Four Weeks** - Adam shrank his company from 18 to 5 and built Swamp, shipping 900 times in four weeks. - He says the team became far more productive by building a machine to build the machine, turning architecture into the primary control point. Transcript: Adam Jacob Thing we did was shrink because the product we had built just failed to find product market fit. And I just needed to face up to that. And so that we, we went from, from, you know, 18 to five. And that was painful. And then we decided to take what we had learned and go build this thing called Swamp, which is very cool. I'm sure we'll talk more about Swamp. But like, you know, that team, we decided we were going to build it as maximalists. We were like, hey, we're just going to do this as if that's just our position. And it turned, and it's wild, crazy, the rate of speed that we're operating. I mean, bananas, the rate of speed we're operating at, the kinds of solutions we're coming up with to solve problems. That would have been, you know, our ability to like estimate how long something takes or what its risk is, is completely decimated. You know, refactors or architecture changes that would have taken months are happening in hours it's all on the table right like whatever direction it has to go is on the table but i couldn't Tell you but like i you know i don't know is that risky or not risky like it used it used to be risky if you got if you're established but if you're building the next thing no even even on my Software that's like you know swampamps only four weeks old it does so much it moves so fast and like and it's so fun and like your emotional ability to process like hey i'm a software engineer You know i've been doing some variation of that my whole life i know what it feels like to end my day having done a good job as a software engineer. Not anymore. Not anymore. I don't know. Like, like, like if, if I had a hard, what's a, what's a hard day at work look like when what I'm doing is telling agents to do the work for me. And then the system is assembling it. Like there is a Delta between a good and a bad day still, but I couldn't tell you what it is. And I could before, you know, I, because I have this accumulation of days and experience and time where I was like, oh yeah, I know that it's a bad day because that bug, I didn't fix the bug Or, you know, it was a bad day because I, you know, whatever I, but I don't, I don't have it anymore. And so like your ability to even like emotionally regulate yourself, much less a team where every single person on the team is also going through this like dramatic change in how they Work and how they think about their job and how they relate to the thing that they do with their hands. And, you know, like it's crazy. And we're more productive than the 18 of us ever were by a huge, by like a real margin. And that's not because I had schmucks before they were incredible software developers. You know, it's, it's just because we built the machine to build the machine. We use those tools in a particular way. And suddenly this acceleration happens. And that's why when you talk about it to people who are like, I fire up cloud code and it does some stuff for me. And it's like, yeah, yeah, we're not the same. You know, like we're not doing the same thing at all. And, you know, yes, there's Claude Code involved, you know, but that's about, that's about where the comparison ends. ([Time 0:13:51](https://share.snipd.com/snip/0773194c-8aac-4925-b526-7ada46eed434)) - **Bring Back Immutable UAT Gates** - Reintroduce outside-in UAT tests that agents cannot modify and block production until they pass. - Use UAT as an immutable gate because agents will otherwise alter tests to make them pass. Transcript: Adam Jacob So then that pull request gets adversarially reviewed as code across the same sort of axes that the plan did, you know, but just with a different agent. And now it has the context of the implementation, not just the plan. And then if those agents find blocking issues, they reply being like, hey, this is a blocking bug, you've got to go fix it. Then you just tell the agent, go look at the pull request, see what their review comments were. So the agents are commenting in the actual PR on GitHub? Yeah, like a person would. They're just doing the code review. They're like, yeah, dog this this didn't match your this didn't match your architecture and then those agents have uh they have accounts on get out there nah they're just like running Out of ci they're just running out of actions you know gotcha uh actions uh and then yeah so at that point it merges so assuming that the robots think it's good it just auto merges so no human Review required If the robots think it's good, it's good. And then we run UAT tests, which this is a thing that we used to do back in the nineties, uh, user acceptance testing, um, where you would have like teams of QA engineers and they'd have Like binders. And every time you were going to release the software, their job was to go through the binder, page one, and like try every feature, you know? And if they found a bug, then they would file a bug, and it would go back to engineering, and then it would all start again, you know? And so like, you're basic, and we did that because there was a separation of concerns. For a long time, people were like, you could never ship software where the same people review the software that do the other thing. That morphed into four-eyed compliance, by the way. But like you know conceptually that's what we used to do so we're doing that again because you need to have a test suite that's outside the agent's control um that defines what the acceptable Outcome is and isn't so it can't just like break you know authentication or login or you know when a feature is stable and there's some rules for that. Yeah. Like UAT testing is only, you're only allowed to do it from the outside in. You can never, you can't, the less, you know, about the internals, the better in UAT testing. So it's not allowed to know the internals, for example. Um, but you just have a big test suite whose job is to exercise the software that the agent can't change. And then if it finds a problem, then it files a new issue that's high priority that goes back to the top and then you stop shipping to production. So nothing ships until the UAT tests pass again. And then, you know, once the UAT tests pass again, then it comes back and you're shipping to production again. And I think we've shipped, I don't know. ([Time 0:25:12](https://share.snipd.com/snip/6042d9bf-aa7d-42bb-9faf-b5dc5a27ae36)) - **Describe Architecture Not Implementation** - Focus on software architecture and domain language when using agents, not low-level coding. - Describe outcomes and bounded contexts so agents produce implementations consistent with intended architecture. Transcript: Adam Jacob Yeah. I'm going to answer a little obliquely. Adam Stacoviak Because you can't share? Adam Jacob No, because I'm not sure how to describe it. I see. Okay. I think for a long time, a lot of software developers learned how to be software engineers from the bottom up. They started attacking code. And you've been able to be a good engineer in our industry by being one of those people. I've hired them. There's many that I really respect, where their first move is always to just, they go straight to that code and they kick the crap out of it until it comes into a shape that makes sense that They like and that's how they program that's essentially dead in the era of agents because you can't understand the bottom like the bottom is moving too quickly for you to attack it with A machete you know what i mean yeah um so instead it's all about software architecture coming down it's do you understand how the components of the system are supposed to fit together? And can you describe those components to the agent in a way that allows the agent to make the implementation have the behavior you desire? And that is very strange because most software engineers have spent most of our career making fun of architects. You know, if you're a software architect, you need like one for every hundred software engineers. Suddenly that's like the main thing you need. So when you look at our plans, the best ones are the ones that start with an outcome. So they say, Hey, we're trying to solve this problem. And then they use the architecture, the language of software architecture to specify how the existing system should change in order to accommodate it. So, you know, you have to do a little less of that when you're like, make this button be greener or, you know, move this left to the right. But when you're like, no, I need to add an anti-corruption layer between these two different bounded contexts. Or like recently I had to describe how we had made a relationship where the authentication layer was separate from how the rest of what we had done was. And we basically said that we were a follower of the authentication layer. And that was a mistake. It turns out I needed to consume the authentication layer. It needed to be an implementation detail of my one bounded context. And so I had to refactor it by saying, what we said was a different bounded context is now an implementation detail of this other one. And I want to completely, I want it to sweep it under the rug with anti-corruption layers. And I had to say it that way to the agent, because that's also how I told the agent to work. I gave it examples of the software architecture I was looking for. And I was like, here's how this works. And it already has read the domain driven design books. It knows what to do. And so if I use that combination of that architecture language and the outcomes I'm looking for in the larger system, it does a great job of generating code that fits to that paradigm In a way that no human would ever be willing to do. Like it's so fucking annoying to write code that way as a human being, but the agent's happy to do it. And so you wind up with this like really clean, supple, easy to maintain code that like, you don't want to write, uh, but you can read it if you needed to, and you understand the architecture. And so the plan phases look like a combination of those two things. It's the high level, here's what we want to do. And then it's the low level, like, uh, this is the architecture by which to accomplish it. ([Time 0:31:06](https://share.snipd.com/snip/cd303c6e-169b-4fbf-8e42-b58ec8c7036a)) - **Provide Agents With An Automation Ontology** - Swamp gives LLMs an automation architecture: models, methods, lifetimes, and reusable workflows. - That architecture lets agents assemble bespoke extensions (e.g., Proxmox) and persist data for later workflows. Transcript: Adam Jacob That's what I do. And a thing that's true is the velocity of engineering causes co-committent increases in the velocity of operations, always. So there's never been an increase in software development velocity that wasn't also accompanied by an increase in operational velocity. Before these changes happened, our ability to operate those systems could barely keep up with the leading edge of software development. I mean, barely. We were hanging on by our fingertips. Like if you were the most peak performance team of software developers, the operational concerns often dragged you down. But if you like got everything right and it was all perfectly aligned, you could hang on, you know, you could not, ops could not be the bottleneck, but you had to be, you had to be doing it, You know, like it had to be designed it had to be you had to come correct okay no way no way those techniques survive an order of magnitude increase in volume none none they will blow immediately Um because they could barely keep up with what we were doing before. It was already slow compared to... It was already slow. It was already bad. Adam Stacoviak Like tortoise versus hare in a way. Adam Jacob Ah, it was already tough. Triple hare, 10X hare. Yeah. If you got everything just so, you could do it. But you mess one thing up and you were cooked, you know? So we had built System Initiative. I've been trying for six years to figure out how to solve that problem. I'd come up with some really smart things. The last iteration, which used the models we had built to build agents, had a problem, which was that it was tied to the same sort of underlying infrastructure we had used to try to solve This visual composing composition problem. And it just was too weird. When people touched it, even though it could do wild things with the agents they were like yeah but it's just too weird you know like it's just whatever it's just too far too far left center Adam Stacoviak Yeah i feel that on some things like you can just have the most next revolutionary thing it could be the most amazing thing but it's just like it's just too different it just wasn't quite Adam Jacob Where i'm at it just wasn't right and and so with Swamp, we had a couple of revel, we had a chance for a couple of revelations. So one was we had discovered the beginnings of this, like we should be building a system to build the system. You know, we should be building, using AI to build the systems that let us build software at this rate. And if that's, if that's true, which it is, that that's what we have to be doing now, the question is, how do you do that for infrastructure? Because all the tools we had to build and manage infrastructure were the worst interfaces for that problem. It's too complicated. And so what Swamp does is it gives the LLM an architecture in the same way that like domain driven design is an architecture to let it think about building reusable automation. And methods that they can run, which then do things. And they produce data, which has different attributes. Like maybe it's a resource that you can serialize. Maybe it's just pure data that you can track. It has lifetimes. You can track multiples of them. And then you can put those models together into reusable workflows that do things for you. And then Swamp is an implementation and a description of that architecture. It says, hey, here's how you put things together. ([Time 0:38:12](https://share.snipd.com/snip/c5e558a7-de22-4af1-8b6a-f46824a56686)) - **Swamp Wrote Its Own Proxmox Extension Live** - A teammate told Swamp to use Proxmox and Swamp wrote its own Proxmox extension and automation on the fly. - Adam watched Swamp auto-generate the model, methods and then execute operations against Proxmox. Transcript: Adam Jacob So in every other age of automation, you would have had to wait. How long would you have had to wait for somebody to build like the Proxmox API support in order to do it? And in Swamp, you don't wait at all. Like we told it how to do it and he said he wanted to use Proxmox. And so Swamp just wrote itself. It was like, great. Here's where I would put the part where I boot a server and Proxmox. And then it would just go, it just goes and does it. ([Time 0:42:28](https://share.snipd.com/snip/af9f366d-5139-40d9-a3d6-bd839ef671db)) - **Rebuilt Config Management As Reusable Primitives** - Adam rebuilt OS-level configuration management inside Swamp as idempotent primitives so agents can reuse them across workflows. - That allowed Swamp to install packages on Ubuntu and reuse those primitives in new automation. Transcript: Adam Stacoviak Just did what it needed to do. Because you kind of gave it tools, right? You're giving it certain tools that can, tools that can write and do different things. I'm giving it building blocks, you know? Adam Jacob Here's the model. Here's eyes, here's hands. Yeah. And here's how you use them. Another thing that I did the other day was while I was shipping other features, I rebuilt configuration management inside a swamp. So it does like, you know, SSH based operating system configuration management. And it has a bunch of, you know, idempotent, convergent config management primitives. And so now whenever you want to do something in an operating system, you can just install that extension and reuse my models to do it. And so it'll be like, oh, I already know how to install packages on Ubuntu. I'll just go grab that extension and I will use that to rebuild this workflow. And what's cool about it is that because we've separated like the information historically, you would have had to have known like what's the right shape of the data you know how do we Share those things or how do my methods work and in swamp you don't you can just be like like the methods produce data and then you can query the data with the llm and then you can use that To feed back into the workflow and sort of around and around it goes so you wind up with this huge this library of repeatable workflows. ([Time 0:43:30](https://share.snipd.com/snip/21948cb6-cb8a-49ae-ae49-02be97de58d5)) - **Treat Created Resources As Queryable Data** - Swamp stores created resources as data so future workflows can query and act on them. - This persistent model-driven approach makes tasks like batch upgrades or inventory trivial for agents. Transcript: Adam Jacob So that moment where you're like, I created this VM. Okay, great. That's awesome. How do I know so I can get back to it later? What if later on you want to write automation that says upgrade all of the VMs that are on an old version of Ubuntu? How do you write that? And in Swamp, it's trivial because you would have just taken the data when you created it and stored it as data in the Swamp. And then later your other workflow would have looked at that data and been like, oh, I know what the Proxmox VMs are that Adam has. I'll just grab that data. Oh, I don't have enough. I need the memory. Great. I'll use the first piece of that data, the ID of that thing in Proxmox, and then I will write a thing that goes and gets it, and then I will write out that data, and then I can use it to do the Next thing. And like- It's like breadcrumbs and blues. Yeah, and that evolutionary nature of it that says, I don't need to know what the model, I don't need to know what you want to do up front. I need to give the LLM the architecture, the shape that says, hey, to make this into reusable automation, think in terms of models, think in terms of data persistence, think in terms Of workflows, think in terms of the relationships in between them, think in terms of inputs. And then I provide tools to make those things get validated across the line. So like, you know, did you write a model that is correct? Call swamp model validate. It'll tell you, you know, it'll look at how you define the model and whether the inputs are right or wrong. And so now the VM doesn't hallucinate whenever creates its own models, because it can just validate that the things that makes up are right or wrong, you know, on and on. ([Time 0:50:33](https://share.snipd.com/snip/26fe70c7-1165-4b37-955f-4517177ee811)) - **Agent-Friendly Transparency And Telemetry Unlock Power** - Agents prefer transparent, persistent context and telemetry; making models auditable improves agent performance. - Swamp records historical data so agents can answer queries like package state on a given date without prior coding. Transcript: Adam Jacob And like the, the side effect of that is that, is that if there's a time where software doesn't work for you, the way you want it to work, the Delta between you doing it the way you want versus The way that the upstream told you to do it is now zero. And even the, even like the counter arguments that people make now where it's like, well, now you have to maintain it and deploy it. And I'm like, well, but if deploying, it just looks like telling swamp bull, you know, if you're just like, deploy it, like, I don't know, man, that seems fine. Seems fine. Seems okay. Like, I don't know that I care, you know? And, and when you think of, I just don't know that i care i know why i used to care you know i know why i used to care i used to care it made sense to care because you had well you had to have your Adam Stacoviak Own personal certainty to have your own confidence because it was your name and that's why you had to care and now i suppose whenever you know something else can move at such a faster pace And potentially dramatically better in almost all cases. Adam Jacob Yeah, and be very specific to what you need. And like, okay, so today that would be – that still feels like a very strange move. But will it tomorrow? I don't know. And like it opens the door to wild things like what's an operating system look like that's designed like swamp. Get out of here, Adam. Right? What's a – I don't even know. I don't either. But you're thinking about it now. Well, I just – I think about what kind of interface does the future human want to use? Adam Stacoviak Yeah. You know what I mean? I don't know. Because that might not even matter. Will we even care about the operating system? Like, sure. There's something there. Adam Jacob But do I care anymore because I'm so distracted away that – The agent will care. The agent will care. Yeah. And the human will care. Someone, some human being will care about the architecture. Oh man. The same way that we care about architecture and houses and you know, like we will care about, we, that is still a skill that will not be removed. You know, you still have to know how to describe how software systems work and how to build them and like all that stuff. But like, you know, people are afraid about how do we train new engineers? I don't know how many like, like new grad engineers you've trained, but I've trained dozens probably. And like, you know, they all come out of school and they know how to program, but they don't know how to program together on a team. And, you know, like they learn over time sort of how all that stuff comes together. Now, I kind of just have to teach those people architecture. And like, if you go to my shelf back here and we grab the like domain-driven design book, like the reason it's frustrating is because he also has to explain how to implement those patterns To you as an engineer. But if I don't have to explain how the patterns are implemented, I just have to explain why you would choose one or the other and what their operational semantics are. I bet I could train you to build software that understands how to implement those architecture patterns and you would be able to build reliable software systems today. That's bananas. ([Time 1:20:56](https://share.snipd.com/snip/f0ce3030-9861-495e-befa-9ed063a8c8e6)) - **Agents Could Collapse Human-Centered Tooling Layers** - Many tooling layers exist primarily for human convenience and may collapse once agents manage operations directly. - Jacob suggests raw fast compute, network, and storage plus agents could replace layers like Kubernetes controllers. Transcript: Adam Stacoviak You're thinking about this more than I am, so I'm just like, I'm curious what you see that. Like, we've got a lot of compression happening and just straight up, you know, gone. It's just not there anymore. Because AI has compressed so many steps from, like, you used to have to do this whole circle. Adam Jacob Now you're just like, straight to, straight to B. I don't know. Um, I think like I could just be wrong. You know what I mean? Like it's like, I think it's, I think it's one of those questions that may be just beyond your ability to see yet, but what will happen is a radical simplification of the stack in general. So today, in order to manage your application in the large or even in the medium, you have to pick a bunch of software whose only real purpose in life is to make it easy for you as a person To relate to it. It's not actually to do the work. It's an abstraction for you to make it easier for you to understand how to do the thing you want to do Right. And most of those things might not be necessary anymore. You know, like if I can just tell swamp to automate something at random and it will do it. When's the moment where what I want is a much more operationally complex thing in the middle? I believe that one exists, but I don't think it's where it used to be. I don't think it's like, when do I want to go? And if you think about this in terms of the NeoClouds, they actually pointed a finger at this, where those data centers they're building for inference and for LLM use, fast networks, Raw compute, fast disks. That's it. Fast compute, fast networks, fast disk. They want as little software as possible in those data centers. They're running almost nothing except operating systems on networks, big, flat networks. And if you try to imagine that with the old tools, you're like, but if you imagine managing it with something like swamp, you're like, yeah, okay. Like, sure. Give me the raw data about every VM in the data center. That's all I care about. That's all agents care about. ([Time 1:40:45](https://share.snipd.com/snip/178ef955-1380-425e-8e25-f66479dcbce6)) - **Accept Issues Not Pull Requests For Agent Systems** - Do not accept raw external pull requests into agent-built systems; accept issues instead. - Adam will never accept pull requests to Swamp and prefers issue-driven, agent-mediated contributions for security. Transcript: Adam Jacob Open to contributions or just open source? Open source, but only open to contributions in the shape of you can file issues or bugs. No one who doesn't, no one can ever accept contributions ever again, if they're going to work at this rate of speed and they want to maintain supply chain security. Your agent can write code at a rate that no human can review. And I can't trust that every agent will catch the security problems that you, And so the only answer is I have to tightly control the inputs to the agents in the first place. So no, you will never, and Swamp will never accept a pull request from anyone ever. What you can do is send me an issue. You can say, hey, I need this feature. And then what I will do is read what you asked me to do and be like, did you ask me to mine Bitcoin for you? You know, did you ask me to do some nefarious stuff? Cause I can do, I can grok that at the level of us, you telling me what you want. I can, I can understand if that's nefarious, but I can't understand it at the level of you sent me a 5,000 line pull request. Did I make sure that you didn't send all the data to someone I didn't want you to send it to? So yeah, it's open source, but it's not open for contributions at all. It's completely closed. And the way you contribute is you file an issue. And when we get around, when we then work that issue with our agents, when we commit the code, I put you on as a co-author. ([Time 1:59:27](https://share.snipd.com/snip/6578f33a-494a-4c3a-a7fd-208344bb359f))