Sentry Founder: AI Patch Generation Is 'Awful' Right Now  ## Metadata - Author: [[The New Stack Podcast]] - Full Title: Sentry Founder: AI Patch Generation Is 'Awful' Right Now - Category: #podcasts - URL: https://share.snipd.com/episode/83b95a4d-941e-401b-8951-89ddeecbbe2c ## Document Notes ## Highlights - **Humble Origins Of Sentry** - David Cramer built Sentry as a practical tool he needed and accidentally grew it into a company. - He values building useful developer infrastructure and enjoyed the serendipity of product-market fit. Transcript: David Cramer Yeah, sure. Okay. So I think this is actually fun because a lot of then is kind of like now. I always tell people that the internet and so I guess the tech industry, if you will, has been boring for like the last decade. Like most of the problems have been solved. And when I got my start and when I was building things like Sentry, you just kind of, you know, databases were still a problem. For example, we didn't have a lot of distributed systems. And so there was just, it was still a little bit of the wild west, right? There were still big companies kind of getting off the ground and Microsoft still existed, but it was very much like still in that early industry cycle. And Sentry is just one of those random things I built during that and happened to be useful, fortunately, and then also happened to be monetizable, you know, great, great happy accidents There. And then, you know, I just did a lot of open source tech and Sentry. We're not open source anymore. We're something called FairSource, but Sentry is just one of those things. And it was always like dev infra or libraries or something along those lines, basically things that make my life easier and things that I wanted to share with my peers. And so just kept working on it, kept sort of improving it. And, you know, and obviously it's grown a lot since then. But it's great to have product market fit for 15 years. ([Time 0:01:30](https://share.snipd.com/snip/518a9fc2-c829-4be4-bbeb-17a234be72eb)) - **FairSource Balances Openness And Protection** - Sentry switched from open source to a FairSource license to prevent others commercializing their work without contribution. - Commits become Apache 2.0 after two years, balancing openness and protecting the business. Transcript: David Cramer You know? Frédéric Lardinois So why'd you go to FairSource and what do you mean by FairSource? David Cramer Yeah, I'm like one of those, I have this point of view that nothing is like, nothing's free in the world and people shouldn't take advantage of each other at the same time. And so open source is one of those things where it's like a give and take relationship, where you use so many software. If you can, you should try to contribute back or contribute back financially or something like that. Obviously, we're a business. We can easily contribute back. Fair Source was sort of this thing where I was getting plagued with people trying to take Sentry and commercialize it that were not part of our organization. And then we're not part of the open source community either, to be honest with you, as other just commercial entities. They were trying to monetize it. And I'm like, this is just like, it makes me mad. That's all. It's just like, it was a completely, I would argue rational, but irrational in the point of view of like, I didn't really have to do it. But I'm like, look, we're just going to stop them. We're going to change the license to something that we inherently believe achieves the goals we have, which is like, we want everybody to use our software. We don't want people getting like, like a free ride, like especially other venture-backed companies to just like take our software and commercialize it because like we have to fund Our own development. Um, but we also don't want to have two different versions of our software, like the, you know, open source free version and then the really good version that everybody actually wants That costs like an arm and a leg or whatever. And so we came up with this thing called FairSource, which the TLDR, it's a license that has very few strings attached. I got to love the Apple emojis. And so it basically just says you can't like sell a competing cloud service that uses our tech. And so you can't create century2.io or whatever, right? But you get all the same features. It's all source available on the internet. Literally, it's like the same code we run in production for all of this. And after two years, when a commit goes live on GitHub after two years, that commit is Apache license. So you can do whatever you want with it after that. Actually, the majority of our software is Apache license at this point. And we thought that was a pretty good give and take there. ([Time 0:06:32](https://share.snipd.com/snip/2b3339eb-5c37-4829-9a36-afd8231115ff)) - **LLMs Shine With Linked Telemetry** - LLMs excel at pattern matching across large linked telemetry sets like traces and logs. - That makes them very effective for summarizing data and surfacing probable root causes in production errors. Transcript: David Cramer But this was very much for us. It's just proactive. We think there's a possibility that some of these things become real life. Today, they're not really. You can definitely do some cool things with the tech, but code gen, for example, it's super early. You can generate a lot of code, but you're not really building much in terms of true production-grade software with it yet today. Just little bits here and there. But one thing we did find that's really good, which is obvious. So Sentry has this big network of data, right? Like we have errors and then we have a bunch of other data like logs and some other stuff. And we connect all that together via what's called a trace ID. It's just a universal request ID. We did this pre-LLMs because, you know, when you can sort of figure out how these pieces of data are linked, it makes it usually easier to debug complex problems. Again, makes sense. It's just a logical thing. Turns out it's even better with LLMs because they're phenomenal at large scale pattern matching. And so one of the first things we found, actually, there are two first things we found that the technology was super good for our business. One, a core part of Sentry is aggregation. I have an error. I have another error. Are they the same error, right? That is a ton of, from the engineering folks in the room, that is a ton of if statements and regular expressions and things like that to make all that work. Like a lot. It's a huge amount of branching. We improve that massively with embeddings. And this must be 18 plus months ago or something at this point. That's quite old in the grand scheme of how things have been evolving. So that was like the first interesting thing we did, which kind of makes sense because it's able to just figure out these components a little bit more loosely. And to be clear, it doesn't replace any of the tech. It just augments and expands what it does. And so that's really important for folks to understand. The second thing we found is going back to that trace thing is just because they interconnected, they were actually able to summarize a lot of data in the system. And it's able to phenomenally like root cause problems, ([Time 0:11:03](https://share.snipd.com/snip/3ea165db-b884-40e3-9fc3-0e4aa662d30d)) - **Validation Turned Skepticism Into Confidence** - Cramer tested an LLM on a cryptic JavaScript error and initially distrusted the result. - A day later a matching production error validated the model's root-cause suggestion, convincing him to ship the feature. Transcript: David Cramer And so they show it to me. And I'm like, look, it's not going to work. And I gave it, like, if you've ever seen a JavaScript error, they're super cryptic. I gave it just like, maybe not the worst JavaScript error I could find, but like bottom 10 kind of thing. It gave me an answer, like a root cause in what it thought was a problem that I was like, see, it's totally wrong. And then the next day, somebody hit the same error in prod and like DM to me with it. And I'm like, holy shit, it was right. And I think that was the day I'm like, okay, just ship it. It's fine. Like, it's good enough. Let's go. It's not that interesting of a problem. But like, it is one of those things that are like, okay, this is very cool that you can do this. It's just like a minor, like cliff note, arguably in, in what these, these machines are capable of. But, but I think we forget about that because this is not a chat bot, you know, it's just, it's just like stringing together useful data. Frédéric Lardinois So, so, so what do you think happened between day one and day two? David Cramer Uh, what do you mean? Frédéric Lardinois Oh, like you said, this is baloney. And then the next day where I just wasn't convinced because I saw it. David Cramer I'm like, there's no way this makes sense. It's just, it's hallucinating if you will. Right. Frédéric Lardinois What happened in the meantime between day one and day two then? Literally nothing. David Cramer I just, it took validation from a human for me. And then I went and I did the work because JavaScript is one of these, JavaScript in general is quite complex to debug, generally speaking, compared to other programming languages. But when I had that human validation, I'm like, okay, maybe there's something, maybe this is actually correct. And I jumped into it. I'm like, wow, it was actually right. ([Time 0:13:52](https://share.snipd.com/snip/e3f1cd5e-3e35-4725-aec5-7c21e1524b99)) - **Keep Humans In The Loop For Code Gen** - Treat current code generation and patch generation as unreliable and human-supervised. - Use AI to propose fixes but keep humans in the loop to verify and apply changes. Transcript: Frédéric Lardinois Is it just not reliable enough? Does it tend hallucinate? What are the things that you will look for and say, oh, BS, this isn't going to work versus yes, this is going to work? David Cramer Yeah, I think one problem is reliability. And so they have all these benchmarks. So if you think about the way the models are used in sort of developer tooling, which is primarily code gen, but there's also code review and some other stuff that happens. One, there's just like, there's so far from, like, even if you dropped, if it was 90% reliable, 10% is a lot of gap, right? If we're honest with ourselves about daily work. So that, and they're not remotely close to 90%. There's these benchmarks that'll be like, oh, it passed 70% of the assessments. And I think if you put it up against the real world, it passes like 7% of the assessments or something in the sense of like, you ask it for something with the appropriate information, and It actually gives you the result you wanted. And I think sometimes that's human error. ([Time 0:16:22](https://share.snipd.com/snip/197f1c71-459f-45c3-b34c-3dbd1e275c92)) - **Context And Recall Limit AI Reliability** - LLMs are strong on pattern matching but weak when prior art or precise context is missing. - Reliability and recall within large contexts are major barriers to production autonomy. Transcript: David Cramer Because again, it's's mostly just like, I always, I always reason about his pattern matching. Like if it can't see sort of prior art, if you will, to scope a problem, how is it going to possibly reason through it? And so I think the problem is you just end up where it's really good at certain things. And because it can do certain things where like, okay, what if it could do the next thing? And it just goes off the rails in the sense of it will, I'll give you an example. One of my hobbies is astrophotography. Take a telescope with a camera, point it at the sky, right? And I was vibe coding, if you will, using codec CLI to just generate like an overlay for a Twitch stream that would show like telescope stats and all this. I gave it the API spec, as in I gave it the exact endpoint and I gave it the payload that would be returned. It completely fabricated everything from there. Like it fabricated how, not just what data might be returned. It just assumed magically some other data might return. But it went beyond that and it fabricated like the disconnect between, okay, I've got this data and then i want to put it here just made up like completely different sort of payload exchanges In there which honestly i was surprising because that usually doesn't happen when you give it like very very specific context um but i think it also goes back to like the reliability Is an issue and so like you can't really use it to build production systems at least from a like a purely agentic way if you will um i think it's much like we are very much at the phase, and I think we're going to be at this phase for quite a long time, where it is truly human in the loop nonstop. But it's a great enabler for moving faster on some things. ([Time 0:17:24](https://share.snipd.com/snip/f572d715-b963-4ea5-8f66-7ee2c42b9a6d)) - **Enforce Rules By Prefixing Every Prompt** - Force important rules into every prompt when agents ignore instructions in longer sessions. - Prefix each prompt with enforced constraints to improve adherence despite costs. Transcript: David Cramer So for example, in one of my agents files, because the context is important, I say, you must always follow docs or something like this. Always look up the docs in the repo before doing a task. It will not do that. It'll completely ignore that probably after like the third sort of step in a prompt, like the third time you prompt the agent. Right. So what I do instead is I actually just prefix every single prompt with what I want to be enforced. And I just use like a cloud code hook to do this and it works, but it breaks a number of other things. ([Time 0:24:05](https://share.snipd.com/snip/fd686b0b-562d-4caa-814e-0fd7bf00e8c0)) - **Agent Monitoring Is Familiar But Non‑Deterministic** - Monitoring agents resembles existing observability work but adds non‑determinism and qualitative concerns. - Token usage and curated detectors become important new metrics for agent monitoring. Transcript: Alex Williams Because these, just all the concepts of, you know, they're similar but different from what you've been doing in the past. David Cramer Important. They're similar and they're very, very similar. They're probably like 99% similar and 1% different. And so when you look at things like agent monitoring, well, it's like, you know, what is actually different? Okay. Well, some might say, well, like you actually really care about the messages, right? Okay. I always cared about the inputs. Inputs were always how I debug systems. Now there's some qualitative stuff that's very, very different, right? Like, because like these are non-deterministic systems. So I think that's the 1%, which probably is worth a lot more than 1%. But the way the systems work and machines operate and communicate, still exactly the same as all technology before it. We use a lot of fancy words to describe, like agents are just services, right? Tools are just RPC calls, if you will, or IPC calls. And so I think once you start rationalizing, it's like, oh, it's actually just technology at the end of the day. How would you debug or monitor or like build technology? And it's the same thing. So we focus on a lot of like, okay, make this work within the context of like, we just build software. These are still software fundamentals. Errors still exist. They're still the same concept, right? And then on the call, and so we have a bunch of stuff that will monitor these and then just use like trace semantics or, I mean, logs are still logs, errors are still errors. They haven't changed at all. Metrics are still metrics, right? There are things you obviously care about that are like curated. So we've, we've tried to like a little bit is like open telemetry, a little bit's our flavor on top of it, but things like, yeah, you obviously want to understand token consumption and Things like this. If nothing else, it's not always about pricing. Token consumption is actually one way you optimize context. So the fewer tokens you can use, obviously for anything, the better you can manage the accuracy of that context window that you provided. So we do a bunch of like sort of curated stuff in that, but it looks like the rest of our monitoring, all honesty. ([Time 0:25:12](https://share.snipd.com/snip/c2b11755-0e72-48fc-920a-4841ce7b4af2)) - **Treat Evals Like Traditional Testing** - Evals for LLMs map directly to standard testing concepts like synthetic uptime and QA. - Using LLMs as judges enables new kinds of detectors and automated evaluation for traces and logs. Transcript: David Cramer So takeaway agents, synthetic uptime monitoring is testing. Testing is testing, like in CI, CD, right? Smoke test, there's all these flavors of that kind of thing. And then there's sort of like traditional QA. All of this is exactly the same kind of thing. It's like I craft a way to sort of ideally repeatedly test a thing and repeatedly look for the same result, right? And if you look at evals, that's all it is. And so it's an interesting way because we've actually never had a great way to describe that concept. But there's like online evals and offline evals and all this stuff. But if you look at the nature of evals, mostly they're just like, okay, I have an LLM that is actually going to be the test. Like it's just going to validate that thing, does it right? And sure, you can do it to do qualitative testing. But we're looking at it from the point of view of like, well, actually, if you use this LLM as a judge concept, there's a lot of ways you could do it. Imagine if you could just stick a trace into it that has nothing to do with a chat bot. Like, I've just got a trace that's a normal, traditional CRUD application. Could I write a rule using an LLM that better evaluates if there's a problem in that trace or in a set of logs or something like that? And so I think there's a lot of cool approaches there, and we're starting to explore some of that. ([Time 0:27:15](https://share.snipd.com/snip/9292e8af-d027-4de5-a320-58962fd4c358)) - **Feeding Agents Curated Telemetry** - Sentry aims to find bugs and evolve toward helping fix them, using curated telemetry to support agents. - Cramer validated integrations where Sentry data feeds coding agents to speed debugging, sometimes almost hands‑free. Transcript: David Cramer Error monitoring. You know, that one's going forever, I think. But I don't know, honestly. I think a lot of what we look at, so Sentry has two clean narratives to reason about the way we view the world. One is we find bugs and we find bugs from the context of what does a bug look like to us? Like when I open the Uber application and it crashes, clearly it's a bug. Okay. If it's slow, that's a bug too. If it doesn't work correctly, that's a bug. You know, those are all different kinds of things. So we, we care about find bugs in abstract. Right. And then we're like, okay, let's, let's evolve sort of the vision of Sentry or the constraints of the company a little bit and say like, what if we find and help you fix bugs? And maybe find and help fix bugs slowly becomes find and just fix bugs, like if the technology sufficiently advances, right? And so a lot of that is like, okay, we're like, we have all this telemetry that's super useful. That's the known quantity. That's like the traditional thing that matters still. And then there's like, well, how does century change if you are super optimistic about the future and whatnot and agents are like a big deal? And one of those things we look at is like, well, how do you make interfaces consumable by headless agents just as well as they're consumable by humans? Or how do you plug in the data? It's like I built the MCP and I am super passionate about the MCP. I think it has great product market fit already. And the reason I like it is because I'm in a coding agent. All I want to do is fix a problem. I don't care how I fix the problem. I care about correctness and all this, but like with the combination of these things, what happens is I open like cursor or cloud code. I say, help me fix. And I drop in the Sentry URL. That query Sentry and it queries like a very curated thing, like a journey, if you will, where it has the right information from the error. It has our sort of AI root cause stuff also embedded in there. And it does just enough to feed this into the coding agent, which isn't ours, right, which is great, to help me cycle through that whole thing. Almost hands-free, and in some cases fully hands-free, which is amazing when it works. Again, it doesn't work all the time. And that's phenomenal. ([Time 0:29:14](https://share.snipd.com/snip/4da5341b-ec54-49fd-ae8d-72e625054ffd)) - **Leverage Distribution For Focused AI Bets** - Use your existing distribution to explore adjacent AI opportunities but avoid irrelevant bets. - Be bold in building with AI, but prioritize areas that leverage your domain and users. Transcript: Alex Williams The dedicated cursor engineer that'll be interesting in a company how much pressure are you getting from your investors for example to move faster with all of this stuff because we Talked to a bunch of these startups on the show too and you know they're they're they're hearing from their investors, we have to move. We have to, we want this, right? And there's a lot of FOMO in the industry. Like, how are you pushing against that? David Cramer Yeah, I mean, I think that's like, that's the average sentiment all the time, no matter what. And also, like, on one hand, it's the average sentiment, but it's more like the talked about hypothetical sentiment than the practical sentiment. Investors, especially like the board, the job is like, hey, you should make more money more fast, right? That is the job at the end of the day. Nothing's changed here. But there is a real thing of like, you see a lot of these companies, which I would definitely be, I'm highly skeptical of, that have grown really fast. And I'm skeptical because I don't think the TAM is real in those. I don't think they've captured markets. I think it's a little bit, it's definitely inflated right now, but they're able to grow. Lots of companies now have achieved like a hundred million in revenue off of like shaky product value. And if you see that and you're driven and you want to succeed, you have to say, how do we capture market fast? Because clearly right now it doesn't matter if the value is real or not. There is a little bit of this game. And when you're building a business is like, you need to capture the opportunity and then catch up to the opportunity, right? And so we think about that a lot because there's sort of two truths, you know. You need to adapt the technology to your use cases in the best way you can. Never get left behind, but that's not different than any technology. But the other truth, which is more real on this one, is it's a disruptive technology, which means nobody's safe anymore, which is very, very interesting. But because of that, you don't have to stay in your own lane. And this is what I keep telling people at the company, you can build anything you want. Now, you might already have distribution, and you might want to leverage that distribution. ([Time 0:36:44](https://share.snipd.com/snip/f4d21de3-c162-49e7-bd22-d7c6dd654947)) - **No‑Code Hype Often Outpaces Deliverable Quality** - Many no‑code and low‑code startups show promise but lack durable, maintainable product quality today. - Cramer views these offerings as prototypes of future value rather than proven PMF. Transcript: David Cramer The thing I like to particularly pick at is the low code slash no code code gen platforms. And so and it's not a slight. I think they're phenomenal. Like, it's very cool. But as I look at this just purely logically as a business, I look at things like Bolt or we're lovable or replet. And I'm like, I just don't see it. I don't see. I see what could be, but I don't see what exists today, if that makes sense. And and then I look at my own irrational behavior and it makes more sense. Like I last month spent a thousand dollars personal personal on random AI services. It's like ChatTBT Pro and Cloud Code and Google's Ultra crap or whatever, you know, some of those I did get value, but I also have a corporate subscription on half of these things. And the Google ultra, I got zero value out of, I just made funny videos. Right. And that was the most expensive one. And so that's where it becomes interesting. You're like, okay, there's something here, but everybody's really pushing to like, figure out what it could be, which is totally fine. Right. But I think the mistake is thinking that those markets exist right now today and that they've, they've achieved PMF, if you will, because I don't think they have, because if you look At these no-code platforms, I'm just going to call them no-code, for lack of better words, they don't really produce a quality product. They don't produce something that I could kind of maintain and stuff. They produce sort of a belief in something that could exist. And I think there's something that could exist, right? But it's just very interesting in that regard. And then when you look at that and you're like, okay, if those things are a little bit, you know, questionable and like, you know, they're a little hand wavy in the actual value and how Well they work. Then what stops you from building something that's hand wavy that works kind of okay, but maybe doesn't quite deliver on the promise that maybe it could in the future. And I'd actually argue our Sears stuff, our root cause and patch fixing, whatever, is exactly that. It's less interesting. It's like less mainstream, if you will. But the patch generation doesn't work at all. I don't know what the percentage, but it's definitely like low double digit percentage of the time. Is it like a correct enough good patch that you'd want to just use it? But it could exist. And so I think it's OK to make those bets. ([Time 0:40:26](https://share.snipd.com/snip/11403c69-ee53-4b7a-9a22-ff3a87377004)) - **Don't Rely Entirely On Future Models** - Expect next models to improve but avoid banking business plans solely on an unreleased model. - Build incrementally while monitoring vendor and frontier model concentration risks. Transcript: David Cramer ChatGPT is a clear value extraction already. Everybody gets lots of value out of it today. Alex Williams A lot of the companies placing those bets would say, well, the bet we're really placing is on the next model to get so much better, right? I feel like that's the thing we keep hearing all the time. Like, it's not great yet but just wait for cloud five or chat gpp well five wasn't that great but ([Time 0:42:31](https://share.snipd.com/snip/702442d2-9830-4b94-bad4-a134e1093773))