The Good, the Bad, and the Future of AI Agents  ## Metadata - Author: [[Decoder with Nilay Patel]] - Full Title: The Good, the Bad, and the Future of AI Agents - Category: #podcasts - URL: https://share.snipd.com/episode/29332331-102f-4c32-8a00-d61caa7d542c ## Document Notes ## Highlights - **Agents Shine Where Kinks Are Fixed** - Agents excel in domains where their blind spots have been ironed out, with coding being the clearest early win. - Progress is fast but uneven across industries because each task contains many small failure points to fix. Transcript: David Hershey Am less confident in our ability to project the end, but I'm happy to talk about now. I've seen agents come a long ways in the last year working with customers. And my view is there are places where we're starting to see what it looks like when agents work really well. And there are still a lot of places where they don't work really well. And I think that kind of makes it confusing. I think for some people, code is a great example. When you're writing code, and especially with Sonnet 4.5, when you watch the model spend a lot of time as an agent developing software itself, it's incredible. It can do a ton. It's gotten much better literally this week. As we release models, it's really visible and obvious how much better they're getting, sort of if you're plugged into that, at being an agent doing really long-running complex tasks. When you look at other sections of the economy or other jobs that people want agents to do large pieces of, sometimes there's stuff they're still not good at. In some cases, they're not good enough at deciphering what's on a computer screen yet to be able to navigate complex uis or whatever it is and so they fall over and stumble over themselves On on something sort of silly and it's easy to point at that and say what are agents all about like is this kind of fluff or hype or whatever it is and i think the way that I see this sort of generally Happening is we've like slowly been ironing out the kinks of the stuff that models fall over themselves on. And we've done that the best so far in coding. I think the industry has done that the best so far in coding where we're making really fast progress on how much an agent can accomplish when writing code. And we are, I think, starting to make that progress in a lot of other domains. For example, like I talked about clicking on UIs in a computer. This model, Sonic 4.5, is way better at that. I don't know if it's exactly the point we're going to tip into people trusting it to automate a whole bunch of stuff they do when they're clicking around a browser yet, but we're getting There. And so I think my general view is we're making really fast progress. It's just not necessarily visible in every part of the economy yet. And then every job and every person and every individual, I have a feeling sort of each model that comes out will get one bit closer to being something that everybody can sort of interact With and see. ([Time 0:04:58](https://share.snipd.com/snip/ad4ac996-c6c3-496f-9492-774c09976677)) - **Incorporate Specialist Knowledge Directly** - Learn directly from domain specialists by embedding their expertise into models rather than expecting generic data to suffice. - Work with experts and customers to surface the specific gaps that prevent agents from succeeding in niche tasks. Transcript: David Hershey Think we need to learn from specialists and we need great ways to learn from specialists. And I don't know if it's necessarily data in the classical sense that there's some like pile of data that we're going to learn from that makes it better, or there are just better ways we Need to incorporate the intelligence of accountants and lawyers and other people into our models. And yeah, I think, I think that could come from data. Some of it certainly will. I think that can come from talking to and interfacing and working with people in those domains. I think a lot about learning directly from our customers. I think there's a future where more companies can contribute more directly to making the models do the stuff that they care about. It would be really nice if someone saw this big, important thing that they'd like to achieve instead of having to sit around and wait for a lab to hopefully build a model that helps them Do it. They can sort of like directly work with a lab. And I think that's probably somewhere that, well, I know that's probably somewhere that we'll head where we can have more direct mechanisms of working with sort of experts. ([Time 0:13:10](https://share.snipd.com/snip/039b1dff-ddb0-40be-b73c-0af78c1f1e2e)) - **Claude Rebuilt Claude.ai Overnight** - Anthropic asked Claude Sonnet 4.5 to recreate their consumer app Claude.ai and it built a working clone overnight. - The model implemented a complex feature (Artifacts) by itself while the team watched it iterate for hours. Transcript: David Hershey One of my favorites to talk about is my team has been working on seeing how much of a software engineering task a model can take on at a time. And so I think we're all aware that models can be used to write code. That's sort of obvious to a lot of people, at least in this industry, and probably people listening to this podcast at this point. But it's often in pair with a human really directly, like going back and forth using quad code or an IDE, like write one thing at a time, check and review it. And the longer you let a model go trying to implement something big and complex, the more likely it is to do a whole bunch of stuff you didn't want to happen. And so we were really curious, what is the most we could stretch that? If I give quad a really huge task, what can I accomplish? And one thing we've seen with this model is in a way that is sort of like really not true of models I've tested and seen in the past, there doesn't seem to, if you give like a sufficiently Good overview of you, what you want to model accomplish, I haven't really seen a ceiling on how much it can keep like Costa consistently working on making something better. And so my favorite actual example of this, we released a video of this yesterday, but I like I'm going to talk about this until people get bored with me. Is we asked Claude to recreate Claude.ai, like our consumer chat application from scratch. And Sonic 4.5 just like worked overnight. We woke up and it just did it. This beautiful clone of Claude.ai that works incredibly well. In my favorite moment from it, there's a feature that people like of ours called Artifacts, where when you ask Claude to make a document or make a web page, it will make it and render it Next to the chat. So you can play with an app that you built live or whatever it is. And I was saying to the person on my team who was working on this demo, like working on this thing, it'd be really cool if Quad could build artifacts itself. That would be amazing. It's a complex feature. It's pretty hard to figure out. It's like, we should try and see if that happens. And then he messaged me like two hours later and he didn't do anything. He didn't intervene. And he's just like, hey, Quad just built artifacts on its own. And it's like currently testing it live. It's just like trying it out itself. And it's just like we released artifacts a year ago. It was like artisanally created by a whole bunch of like wonderful engineers around me at Anthropic who were doing all of the software engineering to build this really complicated Things themselves. And then a year later, we released a model and we asked it to build Quad.ai and let it go for 12 hours and just did it itself. Instead of this giant team of people at Anthropic that work so hard on this thing, the model is just really capable of biting off really meaty, complicated tasks. Like this is something that would take me months to do if I did not have quad. And overnight we, we sort of looked at it and watched it happen. And that progress of just like, like going from a point where it was neat that a model could write a snippet of code for me a year ago to like, oh, it can do like a big chunk of the stuff of the Complex developer work that I need to do. It's just like, I don't know, it blows my mind. ([Time 0:21:06](https://share.snipd.com/snip/0d06f5b0-0a51-46ba-9f5b-cb72dbeb0d13)) - **Sustained Long-Running Development** - Sonnet 4.5 can sustain long-running, multi-hour autonomous development without obvious ceilings in many tests. - Anthropic observed runs up to 30 hours of continuous development on complex engineering tasks. Transcript: David Hershey If I give quad a really huge task, what can I accomplish? And one thing we've seen with this model is in a way that is sort of like really not true of models I've tested and seen in the past, there doesn't seem to, if you give like a sufficiently Good overview of you, what you want to model accomplish, I haven't really seen a ceiling on how much it can keep like Costa consistently working on making something better. And so my favorite actual example of this, we released a video of this yesterday, but I like I'm going to talk about this until people get bored with me. Is we asked Claude to recreate Claude.ai, like our consumer chat application from scratch. And Sonic 4.5 just like worked overnight. We woke up and it just did it. This beautiful clone of Claude.ai that works incredibly well. In my favorite moment from it, there's a feature that people like of ours called Artifacts, where when you ask Claude to make a document or make a web page, it will make it and render it Next to the chat. So you can play with an app that you built live or whatever it is. And I was saying to the person on my team who was working on this demo, like working on this thing, it'd be really cool if Quad could build artifacts itself. That would be amazing. It's a complex feature. It's pretty hard to figure out. It's like, we should try and see if that happens. And then he messaged me like two hours later and he didn't do anything. He didn't intervene. And he's just like, hey, Quad just built artifacts on its own. And it's like currently testing it live. It's just like trying it out itself. And it's just like we released artifacts a year ago. It was like artisanally created by a whole bunch of like wonderful engineers around me at Anthropic who were doing all of the software engineering to build this really complicated Things themselves. And then a year later, we released a model and we asked it to build Quad.ai and let it go for 12 hours and just did it itself. ([Time 0:21:45](https://share.snipd.com/snip/41a7ddfc-600f-4cb2-99cc-3087cd897b0f)) - **Engineer Justin's 30‑Hour WorkChat Demo** - An engineer named Justin used the Quad agents SDK to give Sonnet 4.5 a Slack-like spec and watched it build a multi-feature chat app. - The app included DMs, threads, channels, search, image uploads, authentication, and AI test users. Transcript: David Hershey So it was someone on my team. His name's Justin. Shout out, Justin. You should give praise to him. He's amazing. Recently joined the team on a side note. He's pretty good. Nice. This was born out of this thing of a lot of people have built demos or proofs of concept. And there's this sort of vibe coding trope that you can, yeah, you can build it, use it to quickly mock something up. But can it really like build a real application? Justin really wanted to like test that. And so he was experimenting using our quad agents SDK, which is just sort of like a more programmatic version of quad code to some extent. He was testing, like, can I give a full spec of like a complex thing, like an app similar to Slack to the model and just like watch it build. And we did some tinkering and experimentation around it to get it right. But yeah, you asked, like, was it impressive? It's impressive. It has DMs and threads and channels and a slick search functionality. And you can upload images and GIFs and render them. And multi-user authentication. And Claude, we didn't ask it to, but implemented a whole bunch of AI users for testing. So if you log in, you can send a message. There's like Alice, the PM is in there that you can send a message to who will respond to back stuff about PM work. It is remarkable. ([Time 0:25:18](https://share.snipd.com/snip/6c8edb8f-10f3-4947-a98f-4091c5701885)) - **Practical Chunking Beats Grand Ambitions** - Sonnet 4.5 favors pragmatic, incremental work: it bites off small achievable chunks and iterates. - That chunked approach makes collaboration feel more natural and reduces wild, off‑path attempts. Transcript: David Hershey So a lot of times models in the past would get really eager and ambitious. They would say, I'm going to build this whole thing, and I have these grand ambitions. And it would kind of just meander everywhere trying to do this miraculous piece of work. And the thing that was really cool about 745 is just pragmatic. It's like, okay, right now I'm going to test, does image upload work? And then it's going to do that. It's going to spend a little while doing that. But it just bites off one little chunk at a time. And that feels a lot more like what I want a coworker to work like or a collaborator work like. It's like, if I ask you like, Hey, I need you to go build or chat. I don't want you to go off on this like crazy escapade trying to make everything magical. I want you to just like bite off a piece at a time, show it to me, like committed to get like that of thing. And it's just a little bit more natural and collaborating with it feels more natural. And funny enough, I think this is unrelated, but we've been chatting in our internal company Slack with Claude a lot lately and just chatting with it. And it's been really natural. Like it feels just a little bit more like working with a coworker does. In terms of the tone? Yeah. Like the tone, how it, how it responds, like how it acts in Slack, how it, how it participates in a conversation, what it tries to do. It's a little less like over the top and eager that has jumped out a little bit, which was surprising. Like's not something that I normally expect. But it's also been funny, I guess, which is a funny thing. It cracks better jokes. It's a little bit more witty, that kind of thing. ([Time 0:27:44](https://share.snipd.com/snip/34af5340-9210-4db2-871b-dfb0721ce7b4)) - **Model As Collaborator, Not Immediate Replacement** - Hershey sees the model as an accelerator and collaborator rather than an immediate job replacement. - He also acknowledges such capability forces teams to rethink roles and workflows in software engineering. Transcript: David Hershey Have a great team and I'm really not worried about, I'm going to frame this two ways. I'm not currently worried. Right now, Quad is a collaborator. Like, it works really well with me. It accelerates me. I think it makes our whole team better and faster writing software. I am general. This is not a now thing. To be really honest, watching Quad go for 30 hours, it does trigger a little bit of like, oh my god. That's a pretty different thing. It is a meaningful step change. I think it does like this technology anthropic is founded on the principle that this technology is going to be hugely impactful in the world and part of that is it's going to change how We do jobs and something like doing a whole week of work for me like that's just like meaningfully going to change the industry of software engineering and so yeah there's like a little Bit of, and I would be like, it'd be goofy for me to say, there's not any amount of like, this is like, how are we going to work next? Like, how do I incorporate this? What is it? My net net though, is like, I just think there's a ton of room to make us better, to make better software for users, to make the world a better place with this technology. ([Time 0:30:17](https://share.snipd.com/snip/cc3de56c-f06e-42e2-8d13-f3421349bc2e)) - **Strong Reasoning, Weak Spatial Sense** - Sonnet 4.5 still struggles with spatial awareness and UI navigation in games and some UI tasks. - High-level reasoning coexists with brittle low-level perception, producing surprising failure modes. Transcript: David Hershey I'm accidentally famous for creating quad plays Pokemon in the past. Oh, that was you. Hayden Field I've heard a lot about that. David Hershey Yes, that is my project. And I have also tried a lot of other games. Like I had quad playing chess this time. And one of my favorite comics people really wants me to make quad play Catan effectively. And it's really bad at spatial awareness still. And they say it annoys me to no end. Like it just like basically doesn't know the difference between left and right and up and down. And this is just like, it's just one of those examples that exists in the world of it can do PhD level math and I can't. And just for it to not really understand that it can't walk straight through a building, it hurts my brain. And models are kind of weird and funny that way. So that one, that one was the one that like probably jumped out the most is I really wanted quad to be this like great chess player. It's so logical and interesting. And then it like doesn't know where the pieces are on the board and where other pieces are. And it's like, ah, you're almost there, Claude, one day. ([Time 0:35:32](https://share.snipd.com/snip/78bb295d-c0a4-4f9d-b0c1-60895b1fcfc5)) - **The Interface Is The Missing Step** - Interfaces matter: AI capabilities outgrow existing UIs, and the next interface could unlock mass production use. - Sonnet 4.5 may be the first model that needs a new interface to let non‑engineers build production apps reliably. Transcript: David Hershey Have noticed like my own personal coding, like every model we release. I am trained as a software engineer, so it's just like cheating. I'm like not the best guinea pig for Vibe coding, but I still do it on the side sometimes. And you notice meaningful improvements on how much it can turn out before it goes off the rails and how much you can trust it. I think this is actually an interface problem. And I think there's something funny about AI, which is that it has this tendency to outgrow interfaces really fast. So if you look at the history of coding with AI, there was probably like Copilot with Ghost Text. So Copilot would automatically complete your code. For a while, there was, you would go to us or ChatGPT or wherever it was, and you would ask it to write some code for you in a browser window, if you were a developer, and then you'd copy and Paste that into your editor. Cursor figured out how to sort of like bridge those two things where it could be side by side. A lot of people have started building this sort of agent that was alongside your ID that lets you build things. I don't think any of that is quite the thing that we need for everybody to build production applications, though. Like there's some interface where I actually do think Sonnet 4.5 is the first model that could be that thing where anybody could build a sort of like production ready application. I've seen enough evidence of when left to its own devices, how it can build complex applications. I've seen it be able to deploy a complex application to AWS, like fully autonomously and do like a security audit on it. ([Time 0:46:21](https://share.snipd.com/snip/58645c62-c020-4c43-af62-3a8b0cd3c9bf))