Are You Ready for Agentic Observability?  All right, welcome everyone to our p panel. Are you ready for a agentic observability? Uh we've all been very busy writing all kinds of code with agents. Uh I'm a big fan of of a few of them. I'm not going to talk about which ones right now because I'm we're here to hear our guests. So, what I'm going to do is have these guys introduce themselves. We're going to get right into the conversation talking about how agentic coding uh agentic tools like Claude and Gemini and Codeex are changing how we work and especially how they're changing observability. So, to start with Mikele, could you introduce yourself briefly? >> Yeah. Hi, I am Mikuela uh co-founder of Derum Open Tentry maintainer. Um I have been doing observability for the last 15 years and I've recently taken a liking to to VIP coding. >> Thomas, >> great. Hi, I'm I'm Tom, co-founder and CTO of Multiplayer. Um been a software developer for 25 plus years. Got my start in machine learning and robotics. a lot of passion about observability, building distributed systems, and adding AI to the mix. >> All right. So, we're we're all very excited about this. So, I'm going to say what what I got most excited about when I started using uh aentic coding and then I want you guys to tell a little bit and then we'll get into the observability part. So, for me, it was being able to drop back into the terminal and get out of VS Code and I could run, you know, I mostly use Claude or Gemini CLI. I can run that. I can control Z it. I can run it in T-Max. I can switch around, go to Vim. I was back in my my old stomping grounds from early in my career and it was it's so exciting sometimes. Now, now we're getting better at this. It's like maybe the terminal is limiting, but that's what really just woke up a new kind of version of software development for me. So, uh to you Thomas, what what what got exciting for you in the last year or so about all this? >> So, that's that's a great question. For me, it was, you know, not having to do all the grunt work that I I have to do e over and over and over with every new project that I have. I know I have to write this spoiler plate and this scaffolding and this and if I could just ask for it and it shows up and maybe I have to correct a little bit, but just that that for me felt magical and and it it allowed me to sort of get to the more interesting things faster. >> For sure. Michaela, >> uh, for me, as somebody who sits 26 hours a day meetings, being able to to let stuff work, uh, in the background and then be able to to to test it the way I would test uh, the rest of my product and then be able to adjust it in in between changes has a way of uh, uh, making me productive in the in the nicks of time that are left between uh, one meeting, the other. It's uh, it's very very helpful. That's a huge one for me. Just I don't have to stop, put my tools down. I just let it go on cruise for a bit and just tab over and check on it. Being naughty during meetings, coding. Um so let's get into observability. Uh we we had kind of just over the last what five so years a all the talk was about kind of well one observability as opposed to monitoring. Um, and then getting to, you know, there's the tracing surge and open telemetry and now we're here, we're through kind of this the hump of like, hey, we found this really great use case for AI. What what's changing in observability then? And uh, I'll start with you Michaela like what do you see is the the thing that's coming this year um, and then maybe a couple years out. >> It's actually a few things. I mean for for the first time in probably uh 101 15 years uh the industry is poised to deliver on the promise of obserability tools they actually figure out what's wrong. Um but that is not actually the part that excites me. It's finally the fulfillment of of a long-held promise. What excites me is that the um not really the conversational paradigm. the fact that you chat with the tool because that is usually lackluster but the fact that you can u express in in a text form complex concepts in in a very flexible way that's very exciting the what I uh what I tell often my designers is that that entire dashboard could have been a telegram so being able to have a a very concise representation what you need to know instead of throwing all the semantic pixels and lines and numbers that we have been throwing at at humans so that they would make sense and being able to write it down uh in a concise way where you can get more information on demand as for example evidence and factchecking. Uh that's super exciting to me. >> Awesome. Awesome. Uh Thomas, >> well I I think for me um you know we have an opportunity to deliver on the dream of self-healing systems, you know, where I mean because I take this uh I mean so this is this is happening. I think it's really happening. is pretty exciting and it it um you know it's the I think to be able to go from because I use you know observability for fixing bugs because I I as a developer I create bugs and I have to fix them you know so you know when you have a big distributed system it becomes really really hard to gather all the data in one place trace through all the things to find out that it's this and this and this causing that weird bug that you see every once in a while. Those are the really toughest things, you know, the little exceptions that's kind of easy, you know, to fix, but it's the tougher things in distributed systems that I want help with. So, you know, what I see is, you know, it's go we're going in the direction where, you know, you've got observability, you've got open standards, they're being embraced, there's more data being collected, it's being put in one place, and you can add that as context to an agent that helps you to to debug. And then the next step is the PR. You know, maybe you're yoloing the PR and it's going straight in your system depending on how brave you are. Probably not. That's a future thing. But I I like the dream of, you know, I'm not even involved in the bug fix, the fixing of the bug that I created potentially or, you know, frankly, the AI might have created on their own. So it's a loop of >> they create the bugs. >> Are you writing bugs anymore or are you having the AI write them for you? >> Right. Exactly. I I kind of do both now. I would say there are more bugs in production because of AI. That's one of the drawbacks. >> Well, I'm shipping more code and if my rate of bugs is the same, just give me more bugs because I'm shipping. >> No, I I notice myself, for example, that for I'm working on a on a CLI between things and code that looks plausible, it fails in uh uh very unexpected ways. And uh luckily I uh I still have the the testing. So I never I test stuff within an inch of of reality. But um it's u I I do see that a lot of these bugs would have been prevented if I've been writing them because I would have thought like what happens if the iterator returns more than this many results. And uh I feel that I'm somewhat afraid that our capability of um testing systems is going to be outstripped by the AI proclivity for coding bugs. >> Yeah, let's explore that because that that goes back to like what people like Charity Major say about uh observabilitydriven development is you you don't actually know if your code works until you've seen it in production and you've looked at the metrics and and validated that they're doing what they're supposed to do. Um, so I kind of what I heard you say is testing is we we've had TDD my whole career, right? Right. Showed up in the late 90s, I think. Um, not not everybody does it. Like how many how many people you work with actually does TDD, right? >> Nobody does it. >> But I think we're nobody agrees on what TDD is. >> We have now the the cost of testing is collapsing to nearly zero, right? >> Um, especially unit testing, it's basically zero now. And then integration testing is getting pretty close to zero or at least way lower than it used to be. Nobody did integration testing before because it's huge such a huge pain in the butt to build and test. But now we're at this point where the all that testing is getting cheaper and that last mile of reliability falls to what's happening in production. >> Yeah. >> And so we're back to observability again which I wanted to go. Um, so what what do you guys see changing like from that conversation we just had about more bugs, more code, higher velocities than we've ever seen. Um, what needs to change in our observability stacks to make this thing keep going to keep moving that velocity. >> So I I would say um that the data needs to change. There's not enough I I'm a developer. Uh I it so when I go to my observability tool to find evidence of the thing that's happening I'm I um I I there's a lot of missing data. Okay there that I need as a developer. So I end up having to try to reproduce a bug in order to get the data that I want. That shouldn't happen. So so I need requests, responses, headers. I need session correlated data. Right now observability is kind of a big bucket of data great for like overall system but it but a bug is happening to a person typically typical application it happens because maybe a complex thing is I did this first then this second then this third and then this weird thing happened in the system I need it all together can't get that with traces so I think we need to have better data we the data needs to evolve um and that's an important thing because for agents to work Well, to actually fix a bug for you, you know, do the work that you really want it to do, you have to provide it with good context. And uh if I as a developer don't have enough context from observability out of the box, there's no magic that will come from passing it to an agent. Hika what do you think >> to some extent that there are uh what Thomas said about some sources of data missing like the reticence for the COT tracing uh uh approaches to capture responses and requests so inputs and outputs uh which is almost universal with the exception of um some niches like uh the serverless observability industry where you need since you cannot put debuckers in there with little control the runtime you need the data I am I'm not entirely sure I agree with Thomas so the the correlation of uh of sessions uh yes it's uh there is technically uh real user monitoring which uh goes beyond uh just >> tracing in the client application >> basically yeah yeah >> that's effectively the the holy grail of distributed tracing is to tell you what is the impact of issues right and uh the actual impact that you should care about is what is on the end users. If your database kills over but nobody is affected, was it really a problem? Right? On the other hand, there is also too much telemetry. So the uh something that we see a lot is not only humans drown in bajillion metrics that are never queried, uh traces that are never seen, logs that are never read, but uh AI does it as well. So the our hay stacks have become truly large and this is a negative side effect of the real positive empowering revolution that opens brought around. I I can tell you I've been I've been doing um proprietary observability uh before uh with the we went all in with open telemetry and u we spent at instana uh we fet were competing with other companies about who had the best telemetry and now we start to compete of who does who does the best with the telemetry that everybody has. Point is everybody having telemetry and means of collecting telemetry become so ubiquitous uh that people are hoarding like smog under the mountain and uh >> so I kind of heard two things sorry so I I hear you saying like I I agree with you right there is this problem of if we trace everything and we log everything and we metric everything we get buried in data that 99% of it never gets seen by anyone not even bots we also have what Thomas is trying to say which is we at the long tail of these problems that users experience. Um it's really hard to get that telemetry even if you have tracing because it's often in parts of the code that are not often instrumented. It's a hot path or something where or it has the problem of like you said where it just generates so much telemetry we can't leave that on. So does that open opportunities though um for more intelligent ways to to tap into our systems? Oo well something that I would be profoundly excited to see and it has not happened yet is for the creation to outstrip the creator. The um AI LLM models are trained on on code that that people write especially public facing code and uh there is very little bespoke telemetry in them. It's uh I would love to see like the moment that uh an application will be generated with uh meaningful uh custom instrumentation to actually capture the business context that's going to be massive. >> Um yeah and I I would say that um you know just to sort of put a a finer point on what I was sort of suggesting is I'd like to see the right data not more data. Um, and in fact, right now we've got a lot of noise because you and we've got these big switches like sampling. Okay, like you I agree with you. There's too much data. Nobody looks at it, right? That's because we're trying to collect we're keeping the wrong data. Um, keeping the wrong data and we're missing data. So, it's it's really two problems. So, I'd like to have the the the right data for when I'm debugging. I need to have requests response and headers for certain problems. it is a reality as a developer you know what because otherwise you're guessing what was this called with I don't know it was called I don't know what it was called with so certain cases I need that but I don't mean collect it for everything because most of the time you're just going to throw it away so better data collection options than just uh random sampling or tail sampling or things like that so I think the industry has to move in a direction that provides better data collection opt uh more it's you know I'm interested in these sessions this user is having a problem right now let me look at their data let me turn on debug logging for my session because I'm developing something and I want to see every not not debugging for everybody but debugging for me or debugging for a certain person or this error happened let me escalate for their session more control and also options about more data because there are you know I I've I've tried many times to use open telemetry I love open telemetry you know but I've tried to use open telemetry traces and logs on their own uh with AI agents to fix bugs and they and it gets lost because even I as a developer don't can't figure out exactly what the what the issue is unless it's very simple. So that's that's how I see that's where I'd like to see the industry going more more. >> I guess what you're saying is like if if the trace has the exception in it, I don't need an agent to tell me there's an exception right there. >> Right. Have it read the exception and chase it down for you. But >> yeah, >> but if it's Yeah. >> But you need >> Yeah. If you don't know what it was called with, if you don't know what happened three or four clicks away and you know, real user monitoring is good. That's front end only. You know, you want to see what's happening in your system. I don't know if you've ever, you know, tried to debug an issue in a in a you know, microservices architecture that's just sprawling, you know, where you got to go deep into the system and you you know, your your user is way back over there and what they sent to the black box of your platform. No, I don't know. Lots of other things happened. >> It's 20. Yeah. >> Yeah. So unless you've got a very kind of almost trivial system where you know what's sent to your blackbox is all you need then uh then you need sort of more information and also better collection options. I think >> the funny part is that uh distributed tracing was born actually as an ondemand debug signal. When you go and read the original Dapper Tracer, the the sampling levels were like epic, like one out of five. >> Yeah. Yeah. >> And I hear you saying that what you would like is to be able to on demand for some uh some traces turn on the verbosity plus+ and then for example collect u payloads of requests and responses. >> Well, an agent will need that, you know, an agent will need that. >> So maybe like we shift back to agents for a second, right? Like for me the the thing that I get excited about most is tool use. Like I think when you when you mentioned earlier, right? Like I think Michaela, you know, the chat thing I could take or leave for the most part for myself, but tool calling >> with with that that's super cool to me. And so does that start to like what what opens up in your imagination in terms of like when these things are better at calling tools um to solve these problems we were just talking about like opening up a remote debugger and gathering some information for you without the client ever knowing or stuff like that. What what what's in your imagination right now? >> That's something we're actually building right now. Uh >> you're building that right now. >> Yeah. Yeah. For there is observability like the the classic signals. um they go a long way but uh not hard enough as we have established in this session by now. Uh for example, being able to to go and connect from an operator to a Kubernetes cluster and get on demand information that would be expensive to collect and statistically useless all of the time because when you're when you're needing it like only 0.001% of the times you have the data then it's it's freaking useless. Uh that's important context and uh one of the reasons why our haste tags have been grown massive is since our collection of telemetry is preventative. So we we hoard all the telemetry that we think we might need in an issue since it is almost impossible to go and collect telemetry on demand live. Um yeah that's something that uh that agents can uh can can definitely need. I something that sometimes also keeps me thinking is that in a future where uh AI actually does operate applications and complex ones most of the telemetry collection is going to be live. So we're probably going to have just enough to do baselines and then uh uh most of the things are going to to be turned on and off because the reaction time of machines is going to be orders of magnitude better than humans and machines are effectively always available as long as there's compute. Instead we get stuck in meetings and then we need something >> pages us to to get us back. >> Yeah. Yeah. And I think I think the tool calling is is probably the most exciting part of you know what we're experiencing now. You know uh it opens up so many possibilities. Um you know I mean we we're working on a debugging agent ourselves and you know integrated with Visual Studio Code and what I'd love to see is uh you know issue is detected data is provided to the agent you know code is updated run the code with debug with debugging enabled step through it provide you know the state of the code that you fixed back to the language model for verification. So beyond you know unit tests or or other things actually as a do what a developer would do. >> So the more exploratory part right like exactly >> we could we could write debugger scripts they just they weren't useful 20 years ago they aren't useful now because >> I'm only in a debugger when something weird has happened. >> Yeah. >> So what you're saying is like bring the agent with tool calling in. Tell me if I got this right >> to to go do that debugging for you. Put set the break points step through all those data structures. pull out the useful information and then synthesize a summary for you >> for verification because whatever code you want to verify what it did >> exactly like this like here is the issue I'm changing this with the expectation that with these inputs this path and this state of the application will exist at the end of this function that I just fixed. Okay, I could create a test that that treats that code as a black box and test input output or I could step through it and just sometimes I need to do that as a developer because just to look at this if statement that I've been staring at for 30 minutes because I I don't know my mind is just switching the the thing but I you know I I'm able to like look at the state of the code the actual state of the code to verify that uh that it's actually a fixed then you have confidence that it's that it's done and you can move on. So it's, you know, so tool calling is necessary there. Um, but you know, again, it's this goal of like self-healing systems. You know, there's a bug detected, fix, verification, repeat. >> Yeah, we have a question about that. I I can see Michaela has a thought, so I'm going to let you talk and then we'll get to the question. >> For for the people listening, Thomas is completely right. Don't think however that that replaces tests. That is something that No, I'm not suggesting that. >> I just want for the people in the back. >> Yeah, please. >> I don't think that's enough. >> In fact, the reason I was thinking it is because I've talked to dudes who are like, I don't need tests. I can just open a debugger and just see if it's right. >> Yeah, sure. I don't remember what my code does yesterday. The problem is that agents do not have collective memory. So you you can check it now and then nobody's going to remember the next time the code starts that that was supposed to keep happening. >> That's a good point. I'm >> so thank you for for clarifying that because I I agree with you. In fact, I think that test-driven development I I hated it. I never really worked that way, but I think I'm going to I forced my coding agents to work that way. And that's part of that breath work that I love. You still do what I do. Do do what I tell you. Yes. >> Yeah. >> I'm doing TD now. TDD now because something else is doing it. >> Um. >> Right. Exactly. I could take credit for it now. Yeah. >> So, so we got a question from Grant um said a few minutes back um a great point about a long-held dream of having self-healing program. Um obviously agents can write code and can also receive an error back and try to diagnose but in li my limited experience these are still separate. Are there any examples of even a simple self-healing routine that you have se used or seen in action? >> Not yet but but it's coming. It's It's coming. Absolutely. Yeah. >> I have a little thing I built that's that sort of does it didn't quite do the repair, but it's a it was basically like running a a py a model on a GPU and it wasn't uh doing the the accelerated attention and I could it's a black box, right? like it's hard to debug and I was like >> let me just go and see what the model thinks and then it started exploring and like looking at memory sizes and uh memory size and CPU usage and then there was one other dimension I can't remember right now um and then it would like infer um that SDPA wasn't on so the the accelerated detention wasn't on >> based on those parameters and so now it's like this little can prompt I have I just run it anytime I think that's happening like is this happening and it goes >> nah you're fine or it goes Yep, it's happening again. So, it's but that next step, which is what Grant asked about. >> I think the verification step is the thing that's holding it back. Like I I don't think >> people are ready for the PR >> YOLO deployment, you know, uh because the verification is not quite there. There's still hallucination. There's still, you know, needs to be better tooling around the verification. >> Confidence. Exactly. Right. >> Yeah. Because like if we have all these functional tests and and great unit tests that are free now, we should should my old therapist said no sword, right? No Um, right. But we should be able to just let this stuff rip and let it ship to production. >> We still need a human in the loop for this. >> We do. >> How how much longer? Not sure. >> Either you want to take a guess. >> Also, the I think that something. And by the way, we are so far off the topic of of the webinar. We're going to come back to web to observability. >> No, the one one last point about testing. >> I already see the effect. So the same thing that observability is now infinite telemetry and you can either have infinite telemetry or good insights, we're going to have the same about tests because if everything is tested, who knows which ones you need to keep, right? That I don't know how that is going to shake out. Well, to shift this back into the agentic observability, we have a question from Marvin. Um, and Marvin asks, "In an eneic observability model where agents consume large volumes of traces and signals, how do you decide what is data worth ingesting and retaining versus what should be sampled or dropped to manage cost?" >> Who wants to go first? >> Great question. I I have strong opinions about this, but go ahead. Okay. Um okay so uh so as so I think that um I think the way that data is collected has to change right now it's big bucket throw into the big bucket and you'll never be able to tell what's important what's signal and noise because you got these little needles in the haystack and you can't there's no correlation there's no other stuff so I think I think that there has to be better correlation of data there has to be better um choices about what to be saving and that happens at the application level you know and then in the system itself um ra and not with these big switches of uh am I s you know sample drop 50% just because I don't want to like we have to move away from that with better controls um and uh you know we we have opinions about that at multiplayer you know you know we're trying to work in that direction um there are others you know, charity majors talks about shifting left, better developer oriented uh focus for observability. I think that really has to do with, you know, data collection um choices and and and other things making it more developer friendly uh as opposed to like system level summaries and dashboards in which is where we kind of live today where you only have the choice of sampling uh and that doesn't really get you >> uh places. So I think I think a number of things have to change u and evolve in observability for for for things to be better have better choices about what to get more signal less noise. >> Michaela I we've only known each other for a little bit. I already know you have something to say. A face I can tell. So go ahead. >> I find that um it's not necessarily collection problem. It's a retention problem. So we have these huge haststacks but the problem with the hay stack is only if you start piling it up the um I take a rather dervenient approach to the usefulness of telemetry the um and I I I have TED talks about uh how telemetry to sample what works what doesn't in open telemetry the the tragedy of the commons that it is uh the holy grail tail sampling uh completely not fitting the collector architecture of open telemetry If still sampling is a time deferred centralized decision, the collector is a streaming live compon distributed live component and the two things really don't smash. But um I find that it's not really the collection. So the art of creating telemetry is more which one you want to store. Every single piece of telemetry exists because somebody somewhere at some point would have been helped by the data existing. Nobody creates metrics for funk. The point is that statistically they are not use useful. They are only occasionally useful. Something that for example AI could do is uh given the fact that telemetry is easy to be collected and that those instrumentations exist in the applications uh turn on and off the the dropping of telemetry very close to the application. It's not directly in the process in the closest collector knowing that data would exist. For example, um your Java application starts having spikes in response latency. We all are going to think about garbage collection unless you use the new one. Let's say we're a bit out. We immediately think stop the world, right? The fact that you know that you could get by by turning off uh like opening the dam, getting information about garbage collection, that gives you the freedom to go investigate on on the fly. And this is something that AI would be able to do. Something else uh is for example there is telemetry that is proven to be useful is what users are being shown in dashboards they actually access that is as close to the declaration of usefulness that a user can give you. Not only I created a dashboard for it but I'm actually opening the dashboard. Right? So I um I don't know if it's a matter of how we collect telemetry. I think it's more what do we do with it. So I so I would um you know say that you know for I think what most people do is they auto instrument their applications you know because >> yeah so they auto instrument and what you're talking about is when the data is received have AI decide what to do with it drop it what I think that's I think that's good I agree with you that's good but it still requires a lot of data to be sent and ingested okay so what I'm talking about in collecting options are like you have your observability agents and other stuff like that centrally connected in real time with your applications. So you don't the app doesn't even send the data if it doesn't if it's it knows it's going to be dropped because what one of the big problems that people have is the cost of using these tools. So you can't just send the data and then like okay we'll save part of it with this agent that decides I mean it's good but you're going to charge for that. So what I think the the best the best way to to reduce noise, reduce cost is not send the data at all if you know that it's not going to be used and that's those are the kinds of controls I think we need to like be you know innovative about and and also things like you know things like I agree with you that that tail sampling is you know is non-starter um >> no it is a starter just not in open tele >> for for a lot for open sle whatever But but I think that we also have to think beyond the trace. You know a trace is useful. It's depth into the system. Um but but you know agents for them to do real work as coders. I think if we're connecting observability with actually coding and debugging and fixing um they have to have more uh and what I'd love to see is a direction where we're having more controls real-time connection with the application and also so don't even send the data if it's not going to be used various escalation based on on hey the garbage collection is having some issues let's go around and see start collecting some without overwhelming the system >> you know and and also uh you know more context I would say. >> Yeah, I I I see the two perspectives coming together here, right? Like I Michaela says is pointing out that there's a lot we can do by looking at what users actually do with the data that we've already collected, >> right? And then you're saying that like we also need to push some controls deeper into the applications so we can turn the tap on and off or or get a core dump or whatever to our agent for that analysis so that or not the data at all really going to be necessary. They sound complimentary to me. >> Yeah, I I agree with you. Yeah. >> So you can control costs. Yeah. And get the right data. >> The I think the the discussion about where the tap should be and how to control it, it doesn't actually have a general answer. For example, my answer about hey, you have the the collector running close to you that can do the filtering for you and then you don't have an application that changes its logic at runtime. >> Yes. I I think more people should do that. I really do. I I I don't know what you experience whether people do that. Yeah. Yeah. >> What you're saying is actually the correct approach in mobile because the end user has significant cost. >> Absolutely. Yeah. >> Because of bandwidth. So it's it really depends on the nature of the application where where the tabs should be. Yes. Yeah, definitely. >> Cool. We have a couple questions. So, we have about 10 minutes left together. Um, so I'm going to try to get through some of these questions and uh let's let's try to make these Yeah, we got plenty of time. Let's just go through them. All right. So, not entered. That's your name now. Um, you mentioned self- remediation, but considering the rise of vibe coding and overall use of agents to code, how does agentic co observability I think that's a mistype. How does a agentic observability help these agents like cloud code now in developers in their daily workflows? So I think this is like how are you using cloud code in in an observability workflow or similar whatever agent you use. Um so uh so at at multiplayer we have a full stack session recording and debugging agent. So we we capture it's built on open telemetry. So we c we add the missing data the request responses and headers correlate the data by session provide it directly to to the agent um and are able to say fix this. Okay. So I'm using this all the time. Our customers are using this this all the time. Um, we're working on an automated piece uh to this so that right now you look at the session, you click fix and it it does things uh for you. So it takes a lot of that grunt work of trying to collect the data and you know the missing stuff or replaying and um all that kind of stuff. Uh so we use that all the time. The goal is to try to get it to self-healing. And again that that verification step is still a like work in progress. I don't know that anybody has a a a great solution. So, but the coding tools are great for that. We can say, you know, here's a session. Here's what happened. You know, write some unit tests, right? You know, let's see whether it works. >> I do that all the time, right? Like there's a problem here. I can't tell. Let's just keep writing tests until it shakes out, >> right? Yeah. Yeah, but but in that case, you know, in order to write good tests, you have to have the the the good context, the data, what actually happens, you know, that's the important thing. >> Cool. Do you want to take a shot at that one, Michaela, or >> No, I don't. Uh >> All right, let's go to the next question. >> I I actually would like to take this one. The one about autonomous agents? >> Yeah, go for it. You want me to read it first? I'll read it first so everybody hears it. Thank you. As autonomous agents begin to make real decisions across tools, systems, and clouds, what do you believe is the minimum observable build observable unit regulators, security teams, and operators will need, for example, beyond logs and traces to confidently answer who acted under what authority with what constraints and why? And where do you see today's observability stacks fundamentally failing to provide that level of agent level accountability? I um I do not believe that the problem is the types of telemetry we collect. I believe it's how we use the tools we're given. So uh logs are fantastic to describe events in time and in open ter you can correlate them with traces. Uh the problem is most of the logs are useless. So I actually had a talk at the last CubeCon where I showed that of the logs getting into their zero less than 1% are actually error and fatal and all the rest is probably a waste of money especially if they're sent all the time. The um we had um a discussion about traces and sampling. Tracing sampler are great tools. It's just that on the one hand it's difficult to use them correctly uh because of limitations of the current technology and on the other hand the literacy about um how observability works is uh not great in terms it's complex stuff something that I I used to say is that um and this is where for example my view with respect to to uh for example charity that you mentioned is a bit different Um some people say you should shift left. So every developer should become in some sense an observability expert. And I look at it and I'm thinking most developers are challenged in making their current systems work without getting more responsibility and having the cognitive burden >> of everybody else's crap can make my brain ball. Right. So the um it's more a matter of um getting better at generating telemetry that is useful. And this goes back to my initial comment where I I'm hoping to see soon that AI actually gets good not only generating application code but it telemetry to explain what it does and what it shouldn't. >> Okay. >> And where >> I guess we what I'm curious about and we have a couple more questions I want to get to so we'll go a little quicker. Um, what do we need to add to the observability stack that our current suite of the the three-legged stool isn't getting us? What do you want to add on a quick answer? >> Do I go first? >> Yeah, go for it. >> There is profiling. Uh, for example, it's in the works with open telemetry. Thomas has a great point about input and outputs, especially not for every single one. Uh, reuser monitoring is is a a completely overseen hero. Um the uh there are of course concerns about the cost and things but in reality there are solutions to that. It's just that the current industry has effectively grown um very successful in uh exploiting people that generate too much telemetry instead of generating the correct one. Uh so there we need probably um somebody to to change the rules of the game to to price the signal and the quality rather than just how big is your hay stack. >> Fair enough. Did you want to add anything Thomas before we go? >> No, I I think I totally agree. >> Cool. Well, Benny asks, "Have you seen Python generated knowledge graphs and querying recent PR release notifications? Guessing via GitHub can be uh about a hundred in a day successfully used to better outcomes." I think this is asking about using knowledge graphs with with the agents. Have either of you experimented with that? >> I know there's a bunch of people working in that space. >> Interesting. I mean, that's an interesting idea. Um I have not seen knowledge graphs generated from PRs in order to I guess this would be to to to kind of get you know to do the you know the review itself to understand the relationships between the code and the and the pull request and maybe to get uh to detect whether there's an issue with what's being suggested. I mean, boy, would I like that in some of the huge PRs that I've had to review. Uh um but I guess if you're trying to pull out a knowledge graph for a PR, then you probably want to cut that PR down into bite-sized nibbles rather than like, you know, maybe I would take a more human approach to um to that, but no, I haven't I haven't seen knowledge graphs being used in that way. But but since we're talking about GitHub and history, uh the something that has been historically pretty pretty separate is uh the history of how software evolves and the infrastructure changes that accompany deployment of software >> and observability on the other side. >> Yes. >> And there is a huge potential in effectively marrying the versioning data you have like service. in open telemetry with matching code changes because that is something that agents at scale could probably infer um uh how the changes are related. >> That's a great idea of marrying the observability with the changes in the code and being able to use an agent to connect the two because probably the bug that you're seeing today is something that was injected in code in in you know in the history in recent week or so different PR most of the time. >> Yeah, exactly. Exactly. And that's I think that's important thing. >> It's kind of strange how infrequently my agent will actually go look at the git history and I'll be like, >> "Yeah, yeah, >> hey, we did that 3 days ago, bud. Go take go look at the G." They'll be like, "Oh, look, I discovered this problem." >> This is what I said about agents missing collective memory, right? >> Yeah. >> For them, it's always day one. It's always the new intern. >> Yes, >> we could be so lucky. Um, we have I think we're just about at the end of our time together. Um, I'm going to give you each. We have about two minutes left. Uh just sum up your thoughts if you'd like to and then we'll let our audience go. We really appreciate you all coming. >> Um okay, great. I'll just jump. Yeah. So, uh you know, I I think this is such an exciting area. We're at the very beginning. I think to it's it's a wonderful time to be a software developer. We're going to see not so we're starting to see how co changing coding is changing for us. We see that happening today. We're going to see you know uh running a system debugging things also change greatly in the coming you know months uh and it's what a time to be alive as they say >> what a time yeah Michaela >> we indeed live in interesting times uh there is going uh there is learning that has to be had but it's also um the first time in my career where there is something entirely utterly novel the uh the fact that now We can instead of shoving in the faces of users all the time static dashboards overripe with information hoping that something catches the eye of the person but instead having a sort of I like to use the expression like premortem >> and that's good >> because humans were excellent at ingesting what is going on and what to do about it. Um when I say these things that usually people ask me do you think static dashboards are dead? Uh the answer is uh no but uh their usefulness is going to be curtailed to fewer use cases because when you can rely on AI actually um having a pretty good analysis and showing you effectively build you dashboards in real time that explain the situation then the static ones are more for the kind of um checking how much money you're doing. >> Yeah. the more the more static. >> I totally agree with you by the way. I think that's >> Yeah, I do too. We're at time. Um >> thank you so much everyone. Thank you too for the discussion. Uh this was really fun and we hope to see you at the next uh lead dev panel. >> Have a great day. >> Thank you. >> Bye., Are You Ready for Agentic Observability?, YouTube, 01kg84v8czb2p2amm7kfsgg01p.