Stop Writing Code, Start Writing Docs  ## Metadata - Author: [[The New Stack Podcast]] - Full Title: Stop Writing Code, Start Writing Docs - Category: #podcasts - URL: https://share.snipd.com/episode/8ba9e38c-e1ba-4538-9847-d915598ceb5b ## Document Notes ## Highlights - **Document Intent Before Using AI** - Slow down to speed up: write a user guide, spec, and technical design before using AI on complex projects. - Treat AI like a coworker and feed it clear plans to get reliable, large-scale results. Transcript: Frédéric Lardinois And like, you know, about that, you know, and tell us about those early days when you were first getting started. Yeah. Keith Ballinger And it's interesting, you know, I, while I've always been involved in like developer tools and technologies, you know, as a teenager, I actually would, I was like an email pen pal of Marvin Minsky. And so I, you know, I had the AI bug very early in my life. And I put together a hack team once, like 2012 or something for Bayes Hack, which did machine learning for nonprofits. So I've always been really interested in that world. And then at GitHub, we started building Copilot. And the idea there was, OpenAI had come along and finally had a pretty good model. Funny enough, they called Codex at the time. And it could do coding tasks pretty well if you managed it carefully. You know, there was a chat soon. And interestingly, I think the code completion, starting with that, starting with something very interactive, was very useful in understanding how people would use AI and kind of Getting them comfortable a little bit with it. And, you know, some people go like a ChatGPT or a Gemini even today to do AI coding. And then, of course, like agents came along, right, where all of a sudden the model was doing things on your behalf with tool usage. And that was like a really interesting, you know, growth in our industry. And, you know, I kind of followed along on it. And I remember like maybe mid last year, all of a sudden tool using models started really taking off. And so, you know, people like cursor all of a sudden were able to make a dent because they weren't just code completion and they weren't just chat. All of a sudden they were, like I said, like doing these kind of semi-autonomous tasks for you with tools. And it was funny because up until then, for like a year prior to that, I would just use chat. So I would go to Gemini, what we called Bart at the time, and I would tell it what I wanted to do. And then it would come back and give me code. And I would take that code and I would paste it into my project. But it would also tell me to do other things like run a test and give it the output of the test. And I was basically acting as the tool for Gemini, for the model itself. And then once that was automated, all of a sudden you could do things so much more quickly and it really expanded on what you could do. Now, it's interesting. One of the things you talk about is how people use it. Right. And there, I think one of the things I'm seeing in the industry right now is there's a big variance in people understanding what the best practices are and even maybe what the best practices Should be. And that's something i think that we have to kind of do more education to help people understand like how to use ai for assistance you know for for coding because if you just single shot Stuff you're not going to really be successful you have to treat the ai you know kind of like a co-worker um and you if you follow best practices from development in general, things like Writing a user guide or a spec, things like writing a technical design, creating a project plan, just those three things. If you do that when you interact with AI, which you should do anyways on anything complex, you actually have really great results with AI. ([Time 0:08:23](https://share.snipd.com/snip/bb347106-b81b-4af4-a76d-63bc33c35c6c)) - **Conductor Repo Walkthrough** - Keith demonstrated Conductor: a repo-driven workflow that creates docs, plans, and prompts to guide agents. - He used it to build Aether, a programming language prototype, by iterating tests and examples. Transcript: Keith Ballinger I will share my screen. Let's see here. So I just want to pull up the right thing here. So this is my desktop. Tell me if you see something called line counter there. Yes. Cool. And you can see right now I just have one file called conductor MD. And if you go to my GitHub, you can find a repository called conductor. This is a file that I use to set up a project. And the way that works is I'll just cap conductor so you can kind of see what's in there um it specifies um to the ai how to set up the project there's like a plan file style guide for pros style Guide for code a workflow a status file the prompt that you can use to launch the AI different times, user guide, architecture, and then with more details. So what I can, sorry, go ahead. Alex Williams So just to go in there, I saw a similar, I mean, this is your project, which is kind of interesting that you're coding yourself, obviously. Yeah, and the code lecture is really just marked on files for what it's worth. Yeah, no, but that's what I was thinking about. It's that it's still all very verbose, and we see similar projects, right, where we had agents.md files. Do you think that is that kind of a, just a step right now in a different direction where we don't have to do that anymore? Keith Ballinger I hope not. How do you see the future of that? I hope not. I don't want to, I think I like creating, you know, I like being a craftsman and creating things. And I want to be involved with that. I don't necessarily want to write every line of code and debug every test failure or whatnot. But I like the act of creation. And for me, the act of creation is you know um thinking of a problem i want to solve you know and then thinking about how i want to solve it and and i think those things you know ai could probably Do that as well but you still have the problem of how do i as the ai um or how do i as a human give you enough of like what i want without the a you know you have to do that somehow it can't read your Mind at least not yet and if it doesn't read your mind you know then you can't get that and i want to be involved in that part um so let me show you i'm going to go back to that um my terminal here I'll just show you how this one works because it's it's it's manual to your point um there are mcps that do this kind of thing. I like doing it manually because I find it to be more enjoyable. Plus, I can really control exactly how things happen. So I can, and I just had this prompt ready to go. You know, I'm like the build tool, smart line counter, counts lines per language, prints out a spark line or histogram. And then I just say, use the process outline in ConductorMD to set everything up. So it's going to read that file, and as you'll see in a second, it's going to set up a folder called.conductor, and it's going to put into that.conductor folder, I'll say, yes, always, It's going to put like all these documents that really will help me and the um the model work really well together like that architecture md and the style guides and whatnot and this means I can be involved in the parts of creation that i'm really interested in without necessarily you know having to do everything and so you know we can see it's now it's creating these. And one of the interesting things about this project, and again, it's just a project, I think the idea is what matters, is how you want to control the agentic experience. Okay, so now it needs to write the user guide. I could just write a user guide if I wanted to, but I actually have it ask me questions. And I won't go through all of this right now. But after I answer all those questions, it then has a very specific task plan and architecture doc and user guide. And then I can iteratively, I can come back anytime I want. I can do a session i can do a couple tasks with it where all i really need to say is what's the next task and it'll continue on with that but these are very detailed which means hey we can um We can achieve a lot of success and you know i have using this methodology and whether it's with conductor or just like similar tools i've been able to build i think very interesting and Complex things. So for instance, if you don't mind, I'll show you one of these. And I got really interested in languages. I've always been interested in languages, but I had the idea, and so I just wanted to prototype. Programming languages. Yeah, programming, sorry, programming languages. So I got really the idea of, like, could we build a programming language that was for LLMs? And one of the problems LLMs have is programming languages aren't, you know, they're ambiguous at times and things like that. That's where a lot of the code generation tasks fall apart. So I created what is basically a really ugly language to humans. It's a Lisp, but it's like incredibly specific and doesn't have much ambiguity at all for the LLM. And I built this using that methodology I was talking about. And it's like a Rust project, right? So here you can see, this is all the Rust code. It's using Rust and the LLVM to compile essentially a systems language that is very, you know, it's very fast and it contains a lot of capabilities. And I did all of this just with that methodology, creating an architecture, you know, creating a plan and going through it. Anyway, someone wants to check out this. I call it Aether. It's on GitHub under the Google Cloud organization. And I have no idea if it actually makes LLMs better. I haven't done that part. But what's interesting is this is a complete language, and it works pretty well. All of those examples work, all of the tests pass, and there's thousands of tests. And, you know, I built it probably in a month, just some nights and weekends. ([Time 0:12:12](https://share.snipd.com/snip/897642a8-1526-4feb-bcb7-c9dd6bea8bc9)) - **Verify With Tests And Examples** - Use tests and examples as verification: have the AI write tests and implement features until tests pass. - Inspect examples and test outputs rather than every line of generated code. Transcript: Keith Ballinger Way i do that is i use the ai to help me with that. So I don't look at every line of code. And with something like Aether, I don't really know Rust all that well. So it isn't like I would have done a good job of that anyway. I know LLVM, okay-ish. But I created that architecture doc. I did collaborate along a lot with it. And then the task plan was very minute. And after each task, I basically had it do like a test driven, it was like TDD. It would write tests and then it would build, implement the feature until the test passed. So, you know, like I would look a lot at the test, but I also spent probably most of my time looking at the examples. I had to create a bunch of examples covering all kinds of scenarios, file IO, HTTP, like a blog, you know, you know, simple, you know, memory management things. And those examples I made sure always ran. So after any task that was done, I would make sure like the examples ran all the tests pass and I would make sure the AI was doing those things too. And so in some sense, it's a black box to me. All I really care about is in the end, did it work? Uh, but in the next times I'd go casually inspect things as well. ([Time 0:19:03](https://share.snipd.com/snip/f5ab8598-7d71-42ca-8ad8-c50002807e06)) - **Leverage CLI For Agent Workflows** - Use the CLI as an integration surface: embed AI tools in the terminal to automate repetitive context transfer tasks. - Prefer terminal workflows for control and focused job-oriented interactions. Transcript: Keith Ballinger And we made it, like, it wasn't as tedious or laborious anymore. And that was, like, I think one of the drivers for Gemini CLI or other tools like that. Alex Williams Why do you think the CLI has become so popular again lately you know we've we've got pretty much all of these agents have some kind of cli version at this point why why is that yeah um i think Keith Ballinger Honestly we never abandoned this the terminal i think as part of it we didn't think about the terminal as a primary interface more and more. I think people, you know, you would go to the terminal to run things like NPM, or just other commands like LS or whatever. But we didn't think about it as like our primary interface more and more. You know, there used to be like a rich ecosystem of like TUI terminal user interfaces, like Midnight Commander, right? Alex Williams I'm afraid I'm old enough to remember that. Keith Ballinger Yeah, right? And it's interesting. We just kind of stopped thinking about those and just had command line script, but we always use the terminal. And especially in modern development, where there's so much OSS, people use the terminal for Git. And so it was really natural, I think, to use the terminal as a way to do these things. It gives you a lot of control. It focuses on the job to be done versus lots of UI, you know. And so it kind of, you know, focuses you in on what you need to do. ([Time 0:24:14](https://share.snipd.com/snip/537580c0-dcb9-4402-a6b6-9cf89730c2c1)) - **Code Becomes A Black Box** - Programming will trend toward higher abstractions where developers express intent and design, not line-by-line code. - New languages and tooling will hide code as a black box while developers focus on architecture and composition. Transcript: Keith Ballinger Is a fascinating question because I think there's a wide variety of opinions on this and no one knows for sure. And I have generally, I think, a very optimistic opinion that's probably in the minority. But Aether was, of course, an example of that. I think we should be building programming languages that are designed for LLMs where we never look at them, so to speak. Like, I think we can get to a place where, as a developer, I'm no longer dealing with lines of code. I'm dealing with the system design, the architecture, the, you know, the user interfaces, all those kinds of things. I'm crafting it together. But the actual language underneath is just a black box to me. And this to me feels like a natural progression of abstractions. We originally started with punch cards, right? Essentially machine code. And then we had languages like C, which were thinly on top of things like Assembler. And then we went to more and more abstract languages like Java or C Sharp, which continued up the abstraction level. And I think the next abstraction level is essentially my intent, what I want, and how that composes together to make an experience, to make an application. ([Time 0:29:52](https://share.snipd.com/snip/a82003c0-e84c-4839-8e1a-92a69288a596)) - **Writing Replaces Syntax As Core Skill** - Natural language may be sufficient as the primary interface because precise, well-written prose can convey exact intent. - Improving writing and problem decomposition becomes a core developer skill for AI-era programming. Transcript: Keith Ballinger Yeah I mean it's a this is a question I ultimately don't know the answer to. That's why I asked it. Yeah, I used to be kind of anti-natural language. I felt like there were going to be other metaphors for interacting with AI that was more precise in the natural language. But, you know, as I've worked in this field longer and longer, I kind of feel like, you know, when you look at a textbook or any, like a nonfiction novel, we don't worry about the precision Of what's been written. You know, no one ever says, well, you know, that, you know, that textbook on calculus was imprecise. It was, we feel like it was pretty precise. And of course, there's bad writing that's vague, but it kind of, I think there's so many examples of people in natural language writing precision and doc language. And I think that that, because you have to kind of meet people where they are, right? And people could learn some other syntax, right? You know, something that is formal that allows them to talk to an AI or something that's visual that allows them to communicate with an AI. But then we're asking people to do, learn something new and do something new. And I think you just have more and more evidence that just have really good writing skills and have really good, have the ability to decompose problems well and understand designs, Right? Like design patterns have existed for a long time now, and they've always turned out to be incredibly useful and not just in coding, right? ([Time 0:32:18](https://share.snipd.com/snip/5fe83dea-7c08-42ba-91e9-300a8df73674)) - **Operationalize Agent Reliability** - We can monitor and audit agents like human collaborators rather than needing full model transparency. - New CICD-style checks and sampling in production are required to catch agent failures and hallucinations. Transcript: Keith Ballinger Gotcha. Yeah. I think there's a couple of thoughts I have about this. One is that in terms of the black box piece, right? We don't know what's going on inside of them. There is, in fact, a lot of research going on that's essentially kind of like an MRI on an LLM to, like, show you the circuits, so to speak, of what's being activated during token generation. So we are gaining more and more insight into kind of, like, how they work, if that makes sense. But the other thing is, like, if I'm working with another programmer, I don't really care how their brain works, right? Like I don't need to understand the, you know, the neurochemistry of their brain. What I need to do is be able to communicate with them effectively and clearly and for them to be able to do what they're doing. And if I kind of humanize AI, well, hey, I just need to know I can communicate effectively and clearly with them. They can do what they're doing. Now, there is a course, you're kind of hitting on it. We need ways to kind of evaluate and understand how these systems are doing so that as they're in production and doing things, like, are they doing the right thing or not? Are they going off the rails? And, you know, there's a bunch of techniques that we're creating in the world. I think we have a long ways to go here because it's far beyond, of course, it's like, you know, HTTP errors. It's like, you know, hallucinations and whatnot. I will say that I feel like a lot of these are happening less and less. I think the teams working on inference, the teams working on the models have done a pretty good job at creating more reliability. ([Time 0:36:36](https://share.snipd.com/snip/5cf3d3eb-f0d4-4f8b-93c9-471598b1aa76)) - **Limit Agent Context Scope** - Manage context deliberately: give agents only the files and scope they need for a task to avoid drift. - Spawn sub-agents with minimal context for focused tasks and clearer coordination. Transcript: Keith Ballinger Yeah. The CLI, I mean, you can generally, you know, people manually give the CLI code context that it cares about, right? You can just add some file. Like if I'm working on something, I might say, look, you know, look at this file. And then I use the at symbol, the kind of IntelliSense to the exact file name and, you know, summarize it for me and then write a unit test. And at that point, once it reads that file, it's in the context and Gemini has this huge context window. So it's kind of always there. But also, you can just tell it to do something, and it will determine. It'll look at readmes or overview files. It'll do LS on different directories, and it'll decide which files to read and then build up its own context. So what's interesting is, as an industry, we need to do a lot more around context management. I don't think models are incredibly good at their own context management and i think that's something that for quite a while we'll need to like continue to iterate on tools because the Bigger context gets the kind of the more things can go off the rails and so we can give the user you know developers who are using Gemini CLI the ability to craft the context, exactly what's In there for any particular task. And that's where sub-agents are actually super useful, because you can like give them exactly what they need for that very specific task. ([Time 0:45:15](https://share.snipd.com/snip/df0e0004-a4ff-45af-9344-03e57a2d0d3a)) - **Cloud Assist Troubleshooting Example** - Keith described Cloud Assist investigations that gather runtime state and logs to diagnose errors with AI help. - He praised the cloud's role in aggregating state for highly effective troubleshooting agents. Transcript: Frédéric Lardinois It's also the cloud service providers, really. Keith Ballinger Yeah, others for sure. But you are right. In GCP, we have something called Cloud Assist, and there's an investigations feature where you can literally go to that from, say, when you're looking at logs and say, what's this error? Or, hey, users are reporting like a bunch of 500 errors or something and it will use ai to gather a bunch of context about the runtime environment to help you diagnose the issue and it'll Look at both like you know uh environmental things that are just with gcp in general but a lot about your environment where you have code running and this is exactly what you're talking About. Like you need a cloud provider to pull off that kind of state. And, but when you have it, the agents are so effective, right? It's amazing to me how, you know, every time I want to troubleshoot an issue at runtime, just how easy it is when you have, you know, using Gemini and using investigations and the way it Gathers the state. And then it just, it always knows what the problem was. It's actually, it might be like the, you know, I think we're sleeping on it. ([Time 0:49:29](https://share.snipd.com/snip/c3601cb6-1084-4017-b2bc-74467fc88a69)) - **Capture Intent For Future Handoffs** - Enable handoffs from nontechnical creators to engineering by capturing intent and design artifacts early. - Preserve project intent so later teams can inherit and maintain AI-built apps without full rewrites. Transcript: Keith Ballinger You know, but on the knowledge worker side, especially, I can see people vibe coding, say, you know, business apps for themselves, right? Like using Gemini Enterprise, a marketer creates an app to help them do copywriting. And I think that, you know, we'll see a lot of those, you know, McKinsey said, there's something like a hundred X software demand for supply. So there's a, that's a lot more programmers are a lot more efficiency we need. So it'd be great to have people, those knowledge workers, for instance, marketers be able to build those things. And then we need to make sure that, you know, there's a handoff, right? If the marketer builds some cool app, you know, at some point it and the internal engineering team will need to take it over. And can we use AI to make that better from step one, you know, like capture the intent that went into what was built, all those kinds of things. ([Time 0:58:29](https://share.snipd.com/snip/fe430970-b9fe-450d-b99f-1c4ef14c47de))