Are you ready for agentic observability - CourtneyR.dev's 2nd Brain

Are you ready for agentic observability? 🔗 **Source:** [Are you ready for agentic observability?](https://www.youtube.com/watch?v=FSo33neZL_Q) > [!note]+ Document Notes > tags: [agentic-observability, ai-debugging, observability-tooling] > author: [MISSING] > source: [MISSING] > reference: https://www.youtube.com/watch?v=FSo33neZL_Q > date created: 2024-12-19 16:20 > > # 🏷️ Title: Are you ready for agentic observability? >  ## 🧰 Action Inbox - [H1] Implement on-demand telemetry collection with agent-controlled verbosity — distributed systems debugging #observability - [H2] Integrate debugging agents with IDE for automated bug reproduction and verification — self-healing systems #development - [H3] Explore tool-calling capabilities for live system interrogation during incidents — real-time diagnostics #automation ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ### 🔗 References - [Multiplayer](https://multiplayer.dev) - [OpenTelemetry](https://opentelemetry.io) ## 📝 Summary - **What happened**: Panel discussion on how AI agents are transforming observability practices, featuring experts from Derum and Multiplayer discussing agentic coding's impact on debugging and system monitoring - **Why it matters**: Traditional observability tools are evolving to support AI-driven development workflows, promising self-healing systems and more intelligent data collection - **Key stakeholders**: DevOps engineers, platform teams, observability vendors, and developers using AI coding assistants - **Risks and tradeoffs**: Higher bug velocity from AI-generated code vs improved debugging capabilities; data collection costs vs comprehensive system visibility - **Bottom line**: Observability must evolve beyond static dashboards to support agent-driven debugging, requiring smarter data collection and real-time system interrogation capabilities ## 📖 Main Content The panel explores how AI agents are revolutionizing both software development and observability practices. Michaela, co-founder of Derum and OpenTelemetry maintainer, and Thomas, CTO of Multiplayer, discuss their experiences with agentic coding tools like Claude and Gemini CLI. Key themes include the shift from reactive monitoring to proactive, conversational interfaces with observability data. Rather than overwhelming developers with dashboard metrics, AI can synthesize complex system states into concise textual summaries with evidence on demand. The discussion highlights critical gaps in current observability stacks: missing request/response data, poor session correlation, and ineffective sampling strategies. Tool-calling capabilities enable agents to perform live system interrogation, connecting to Kubernetes clusters or debugging sessions to gather targeted information only when needed. Self-healing systems emerge as a realistic near-term goal, where agents detect issues, generate fixes, and verify solutions through automated debugging. However, verification remains the bottleneck preventing fully autonomous remediation. ## 🗒️ Notes #### ⏱️ [00:00](https://youtu.be/FSo33neZL_Q?t=0) — [H1] Panel introductions and agentic coding excitement - Key idea: Experts share what excites them most about AI-powered development tools - Evidence: "Being able to drop back into the terminal and get out of VS Code...back in my old stomping grounds" #### ⏱️ [03:45](https://youtu.be/FSo33neZL_Q?t=225) — [H2] Productivity gains from eliminating grunt work - Key idea: AI handles boilerplate and scaffolding, allowing focus on interesting problems - Evidence: "Not having to do all the grunt work...just ask for it and it shows up" #### ⏱️ [04:30](https://youtu.be/FSo33neZL_Q?t=270) — [H2] Background processing during meetings - Key idea: Agents enable asynchronous development during fragmented schedules - Evidence: "Being able to let stuff work in the background...making me productive in the nicks of time" #### ⏱️ [06:15](https://youtu.be/FSo33neZL_Q?t=375) — [H1] Conversational observability interfaces - Key idea: Text-based system summaries replace overwhelming dashboard pixels - Evidence: "That entire dashboard could have been a telegram...concise representation of what you need to know" #### ⏱️ [08:30](https://youtu.be/FSo33neZL_Q?t=510) — [H1] Self-healing systems becoming reality - Key idea: Agents can close the loop from bug detection to automated fixes - Evidence: "Opportunity to deliver on the dream of self-healing systems...I'm not even involved in the bug fix" #### ⏱️ [11:20](https://youtu.be/FSo33neZL_Q?t=680) — [H2] AI generates more bugs but ships faster - Key idea: Higher velocity development increases total bugs despite debugging assistance - Evidence: "Code that looks plausible, it fails in very unexpected ways" #### ⏱️ [15:45](https://youtu.be/FSo33neZL_Q?t=945) — [H2] Missing data problem in observability - Key idea: Critical debugging information absent from current telemetry stacks - Evidence: "There's a lot of missing data...I need requests, responses, headers, session correlated data" #### ⏱️ [18:50](https://youtu.be/FSo33neZL_Q?t=1130) — [H2] Too much telemetry creates noise - Key idea: OpenTelemetry's success created data hoarding problem for both humans and AI - Evidence: "Our hay stacks have become truly large...AI does it as well" #### ⏱️ [22:10](https://youtu.be/FSo33neZL_Q?t=1330) — [H1] On-demand telemetry collection - Key idea: Agents should enable targeted data gathering instead of preventative hoarding - Evidence: "Turn on and off the verbosity plus+ and collect payloads of requests and responses" #### ⏱️ [26:30](https://youtu.be/FSo33neZL_Q?t=1590) — [H1] Tool calling enables system interrogation - Key idea: Agents can connect to live systems and gather expensive-to-collect data on demand - Evidence: "Connect from an operator to a Kubernetes cluster and get on demand information" #### ⏱️ [29:15](https://youtu.be/FSo33neZL_Q?t=1755) — [H2] Automated debugging workflows - Key idea: Agents can step through debuggers and verify fixes automatically - Evidence: "Issue detected, data provided, code updated, run with debugging enabled, step through it" #### ⏱️ [32:45](https://youtu.be/FSo33neZL_Q?t=1965) — [H2] Tests remain essential despite debugging agents - Key idea: Automated debugging supplements but cannot replace comprehensive testing - Evidence: "Don't think that replaces tests...agents do not have collective memory" #### ⏱️ [38:20](https://youtu.be/FSo33neZL_Q?t=2300) — [H2] Retention vs collection problem - Key idea: Focus on storing the right data rather than collecting less data - Evidence: "It's not necessarily a collection problem. It's a retention problem" #### ⏱️ [42:10](https://youtu.be/FSo33neZL_Q?t=2530) — [H1] Static dashboards becoming obsolete - Key idea: AI-generated real-time analysis will replace most static monitoring dashboards - Evidence: "Their usefulness is going to be curtailed...when you can rely on AI building you dashboards in real time" ## 📚 References - [OpenTelemetry](https://opentelemetry.io) - [Multiplayer](https://multiplayer.dev) - [Dapper Tracing Paper](https://research.google/pubs/pub36356/) ## 🗣️ Paraphrase - AI coding tools are transforming both development velocity and observability needs, requiring smarter data collection strategies rather than traditional sampling approaches - Self-healing systems are becoming feasible through agent tool-calling capabilities, but verification and testing remain critical human-in-the-loop requirements - Observability interfaces will shift from static dashboards to conversational AI that synthesizes system state into actionable insights with evidence on demand ## 📰 Newsletter **Agentic Observability: The Future of Debugging is Here** The latest panel discussion reveals how AI agents are revolutionizing not just how we write code, but how we monitor and debug our systems. With tools like Claude and Gemini CLI enabling faster development cycles, observability must evolve to keep pace. **Key Takeaways:** - Static dashboards are giving way to conversational AI interfaces that synthesize complex system data - On-demand telemetry collection reduces costs while providing targeted debugging insights - Self-healing systems are becoming reality through agent tool-calling capabilities **What This Means for Your Stack:** Consider how your observability tools can support AI-driven workflows. Look for platforms that offer programmatic access and can integrate with debugging agents rather than just human operators. **Resources:** [OpenTelemetry](https://opentelemetry.io), [Multiplayer](https://multiplayer.dev), [AI Debugging Best Practices](https://learn.wordpress.org/) ## ✅ Action Items #### DevRel - Developer advocacy: Create demos showing agent-driven debugging workflows, measure adoption metrics for agentic observability tools - Education: Design workshops on integrating AI coding tools with observability stacks, office hours for debugging best practices - Developer experience: Improve documentation for programmatic observability APIs, update SDKs to support agent tool-calling - Developer marketing: Campaign highlighting self-healing system capabilities, content strategy around AI-native monitoring - Community management: Survey developers on observability pain points with AI tools, moderate discussions on verification approaches #### OSPO - Policy & Governance: Develop contribution policies for AI-generated observability code, update project charters for agent integration - License Compliance: Review AI training data usage in observability tools, establish scanning protocols for generated telemetry code - Security Management: Assess vulnerabilities in agent-controlled system access, create attestation frameworks for automated fixes - Software Inventory: Track AI dependencies in observability stacks, maintain SBOMs for agent-generated components - Developer Support: Training on secure agent debugging practices, tooling for verification of AI-generated fixes - Community Engagement: Fund upstream work on OpenTelemetry agent integrations, negotiate agreements with AI observability vendors #### Community - Maintainers: Support roadmap development for agentic observability features, provide governance templates for AI integration - Contributors: Create onboarding programs for AI-assisted contributions, establish first-issue programs for agent debugging tools - Sponsors: Transparent budgets for AI observability infrastructure, impact reports on self-healing system outcomes - Ecosystem: Coordinate cross-project standards for agent tool-calling in observability platforms #### WordPress - Core Performance: Integrate agent-driven profiling tools, establish metrics for AI-assisted performance optimization - Core AI: Implement guardrails for observability agents, create evaluation sets for debugging accuracy - Documentation: Fill gaps in AI debugging workflows, create style guides for agent-generated telemetry documentation - Meta: Improve .org API reliability for agent integrations, roadmap support for programmatic monitoring access #### The WPCC - Programs: Fund fellowships focused on agentic observability research, provide microgrants for AI debugging tools - Projects: Support governance clinics for AI integration policies, sponsor documentation sprints for agent workflows #### FAIR - Technical independence: Ensure metadata transparency in AI observability tools, implement provenance checks for agent-generated fixes - Governance artifacts: Create charters for responsible AI in monitoring systems, establish operator agreements for self-healing capabilities ## 🧪 Hypotheses - On-demand telemetry collection controlled by AI agents will reduce observability costs by 60% while improving debugging effectiveness - Self-healing systems with proper verification frameworks will resolve 80% of common production issues without human intervention - Conversational observability interfaces will replace 70% of static dashboards within 3 years as AI synthesis becomes standard - Agent-controlled debugging will identify root causes 5x faster than traditional manual investigation methods ### 📊 Methodology - A/B testing of on-demand vs traditional telemetry collection measuring cost and resolution time metrics - Longitudinal study tracking self-healing system success rates and false positive frequencies across different application types - User experience research comparing dashboard vs conversational interfaces for incident response effectiveness ### ❓ Questions - How can verification frameworks ensure agent-generated fixes don't introduce regressions while maintaining development velocity? - What privacy and security models are needed when agents have real-time access to production system internals? - How will observability pricing models adapt to AI-driven data collection that focuses on quality over quantity? ### 🔮 Future Research - Investigation of agent collective memory systems to maintain debugging context across incidents and deployments - Development of formal verification methods for AI-generated system fixes in production environments - Exploration of multi-agent observability systems where specialized agents handle different aspects of system health ### 💡 Implications - **Practice**: DevOps teams must develop new skills in agent management and verification rather than just dashboard interpretation - **Policy**: Organizations need governance frameworks for automated system changes and agent access controls - **Security**: New attack surfaces emerge when agents have production system access, requiring enhanced authentication and authorization - **Education**: Training programs must evolve to cover AI-assisted debugging and observability agent management - **Funding**: Budget allocation shifts from storage costs to compute costs for real-time analysis and agent processing ### 🧠 Learning Styles Assessment Reading level estimate: Flesch-Kincaid Grade Level 12 #### 👀 Visual learners - Present: Architecture diagrams showing agent-system interactions, data flow charts for telemetry collection - Improve: Add visual debugger screenshots, system topology diagrams showing agent integration points #### 🔊 Aural learners - Present: Panel discussion format provides conversational learning - Improve: Create 5-minute audio summary highlighting key concepts, provide speaker identification cues #### 🗣️ Verbal learners - Present: Technical definitions provided in context, analogies like "hay stacks" for data volume - Improve: Add glossary of agentic observability terms, provide more real-world examples and case studies #### 🧩 Kinesthetic learners - Present: Discussion of hands-on debugging experiences and tool usage - Improve: Create step-by-step labs for setting up agent debugging workflows, provide configuration checklists #### 🧠 Logical learners - Present: Clear problem-solution progression, cost-benefit analysis of different approaches - Improve: Add decision trees for choosing observability strategies, provide KPI frameworks for measuring success #### 🧑‍🤝‍🧑 Social learners - Present: Panel format encourages discussion and different perspectives - Improve: Add discussion prompts for teams to evaluate their observability needs, create peer review rubrics #### 🧘‍♀️ Solitary learners - Present: Comprehensive notes enable independent study and reflection - Improve: Add self-assessment questions for observability maturity, provide worksheets for planning agent integration #### 🌿 Naturalistic learners - Present: Discussion of system evolution and adaptation mirrors natural processes - Improve: Connect observability ecosystem health to environmental monitoring analogies ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Clear heading hierarchy, timestamped sections with descriptive titles - Improve: Add ARIA landmarks for navigation, enhance alt text for technical diagrams #### Keyboard navigation - Present: Logical tab order through content sections - Improve: Add skip links for major sections, ensure all interactive elements are keyboard accessible #### Contrast and readability - Present: Plain text format ensures high contrast compatibility - Improve: Specify minimum contrast requirements for any visual elements, test with high contrast mode #### Text resizing and reflow - Present: Markdown format scales well across devices and zoom levels - Improve: Test content at 200% zoom, ensure code blocks don't create horizontal scrolling #### Media and captions - Present: Full transcript provided for video content - Improve: Add timestamps to transcript for easier navigation, provide audio descriptions for --- ## Full Transcript [00:12] All right, welcome everyone to [00:16] our p panel. Are you ready for a agentic [00:19] observability? Uh we've all been very [00:21] busy writing all kinds of code with [00:24] agents. Uh I'm a big fan of of a few of [00:26] them. I'm not going to talk about which [00:28] ones right now because I'm we're here to [00:30] hear our guests. So, what I'm going to [00:31] do is have these guys introduce [00:32] themselves. We're going to get right [00:34] into the conversation talking about how [00:36] agentic coding uh agentic tools like [00:39] Claude and Gemini and Codeex are [00:42] changing how we work and especially how [00:44] they're changing observability. So, to [00:46] start with Mikele, could you introduce [00:48] yourself briefly? [00:49] >> Yeah. Hi, I am Mikuela uh co-founder of [00:53] Derum Open Tentry maintainer. Um I have [00:57] been doing observability for the last 15 [00:59] years and I've recently taken a liking [01:02] to to VIP coding. [01:06] >> Thomas, [01:07] >> great. Hi, I'm I'm Tom, co-founder and [01:09] CTO of Multiplayer. Um been a software [01:12] developer for 25 plus years. Got my [01:14] start in machine learning and robotics. [01:16] a lot of passion about observability, [01:19] building distributed systems, and adding [01:22] AI to the mix. [01:24] >> All right. So, we're we're all very [01:26] excited about this. So, I'm going to say [01:29] what what I got most excited about when [01:30] I started using uh aentic coding and [01:33] then I want you guys to tell a little [01:34] bit and then we'll get into the [01:34] observability part. So, for me, it was [01:38] being able to drop back into the [01:39] terminal and get out of VS Code and I [01:42] could run, you know, I mostly use Claude [01:44] or Gemini CLI. I can run that. I can [01:47] control Z it. I can run it in T-Max. I [01:49] can switch around, go to Vim. I was back [01:51] in my my old stomping grounds from early [01:53] in my career and it was it's so exciting [01:55] sometimes. Now, now we're getting better [01:57] at this. It's like maybe the terminal is [01:59] limiting, but that's what really just [02:01] woke up a new kind of [02:05] version of software development for me. [02:06] So, uh to you Thomas, what what what got [02:09] exciting for you in the last year or so [02:11] about all this? [02:12] >> So, that's that's a great question. For [02:13] me, it was, you know, not having to do [02:16] all the grunt work that I I have to do e [02:19] over and over and over with every new [02:21] project that I have. I know I have to [02:22] write this spoiler plate and this [02:24] scaffolding and this and if I could just [02:26] ask for it and it shows up and maybe I [02:28] have to correct a little bit, but just [02:30] that that for me felt magical and and it [02:34] it allowed me to sort of get to the more [02:36] interesting things faster. [02:39] >> For sure. Michaela, [02:41] >> uh, for me, as somebody who sits 26 [02:44] hours a day meetings, being able to to [02:47] let stuff work, uh, in the background [02:50] and then be able to to to test it the [02:52] way I would test uh, the rest of my [02:54] product and then be able to adjust it in [02:57] in between changes has a way of uh, uh, [03:00] making me productive in the in the nicks [03:03] of time that are left between uh, one [03:05] meeting, the other. It's uh, it's very [03:07] very helpful. That's a huge one for me. [03:10] Just I don't have to stop, put my tools [03:12] down. I just let it go on cruise for a [03:14] bit and just tab over and check on it. [03:16] Being naughty during meetings, coding. [03:19] Um so let's get into observability. Uh [03:22] we we had kind of just over the last [03:24] what five so years a all the talk was [03:26] about kind of well one observability as [03:29] opposed to monitoring. Um, and then [03:31] getting to, you know, there's the [03:33] tracing surge and open telemetry and now [03:35] we're here, we're through kind of this [03:37] the hump of like, hey, we found this [03:40] really great use case for AI. What [03:43] what's changing in observability then? [03:45] And uh, I'll start with you Michaela [03:47] like what do you see is the the thing [03:49] that's coming this year um, and then [03:51] maybe a couple years out. [03:52] >> It's actually a few things. I mean for [03:55] for the first time in probably uh 101 15 [03:58] years uh the industry is poised to [04:01] deliver on the promise of obserability [04:03] tools they actually figure out what's [04:04] wrong. Um but that is not actually the [04:08] part that excites me. It's finally the [04:10] fulfillment of of a long-held promise. [04:12] What excites me is that the um not [04:16] really the conversational paradigm. the [04:18] fact that you chat with the tool because [04:19] that is usually lackluster but the fact [04:23] that you can u express in in a text form [04:27] complex concepts in in a very flexible [04:30] way that's very exciting the what I uh [04:32] what I tell often my designers is that [04:35] that entire dashboard could have been a [04:37] telegram [04:38] so being able to have a a very concise [04:41] representation what you need to know [04:43] instead of throwing all the semantic [04:46] pixels and lines and numbers that we [04:48] have been throwing at at humans so that [04:51] they would make sense and being able to [04:55] write it down uh in a concise way where [04:58] you can get more information on demand [05:00] as for example evidence and [05:01] factchecking. Uh that's super exciting [05:04] to me. [05:05] >> Awesome. Awesome. Uh Thomas, [05:09] >> well I I think for me um you know we [05:12] have an opportunity to deliver on the [05:15] dream of self-healing systems, you know, [05:18] where I mean because I take this uh I [05:21] mean so this is this is happening. I [05:24] think it's really happening. is pretty [05:25] exciting and it it um you know it's the [05:29] I think to be able to go from because I [05:32] use you know observability for fixing [05:35] bugs because I I as a developer I create [05:38] bugs and I have to fix them you know so [05:40] you know when you have a big distributed [05:42] system it becomes really really hard to [05:45] gather all the data in one place trace [05:49] through all the things to find out that [05:51] it's this and this and this causing that [05:53] weird bug that you see every once in a [05:55] while. Those are the really toughest [05:56] things, you know, the little exceptions [05:59] that's kind of easy, you know, to fix, [06:00] but it's the tougher things in [06:02] distributed systems that I want help [06:03] with. So, you know, what I see is, you [06:06] know, it's go we're going in the [06:08] direction where, you know, you've got [06:10] observability, you've got open [06:11] standards, they're being embraced, [06:13] there's more data being collected, it's [06:15] being put in one place, and you can add [06:17] that as context to an agent that helps [06:20] you to to debug. And then the next step [06:23] is the PR. You know, maybe you're [06:26] yoloing the PR and it's going straight [06:28] in your system depending on how brave [06:30] you are. Probably not. That's a future [06:32] thing. But I I like the dream of, you [06:35] know, I'm not even involved in the bug [06:38] fix, the fixing of the bug that I [06:39] created potentially or, you know, [06:41] frankly, the AI might have created on [06:43] their own. So it's a loop of [06:45] >> they create the bugs. [06:46] >> Are you writing bugs anymore or are you [06:48] having the AI write them for you? [06:50] >> Right. Exactly. I I kind of do both now. [06:52] I would say there are more bugs in [06:54] production because of AI. That's one of [06:56] the drawbacks. [06:57] >> Well, I'm shipping more code and if my [06:58] rate of bugs is the same, just give me [07:01] more bugs because I'm shipping. [07:02] >> No, I I notice myself, for example, that [07:05] for I'm working on a on a CLI between [07:07] things and code that looks plausible, it [07:12] fails in uh uh very unexpected ways. And [07:15] uh luckily I uh I still have the the [07:18] testing. So I never I test stuff within [07:21] an inch of of reality. But um it's u I I [07:26] do see that a lot of these bugs would [07:28] have been prevented if I've been writing [07:29] them because I would have thought like [07:31] what happens if the iterator returns [07:34] more than this many results. And uh I [07:37] feel that [07:39] I'm somewhat afraid that our capability [07:41] of [07:43] um testing systems is going to be [07:45] outstripped by the AI proclivity for [07:48] coding bugs. [07:49] >> Yeah, let's explore that because that [07:50] that goes back to like what people like [07:52] Charity Major say about uh [07:54] observabilitydriven development is you [07:57] you don't actually know if your code [07:58] works until you've seen it in production [08:00] and you've looked at the metrics and and [08:02] validated that they're doing what [08:03] they're supposed to do. Um, so I kind of [08:06] what I heard you say is testing is we [08:09] we've had TDD my whole career, right? [08:11] Right. Showed up in the late 90s, I [08:13] think. Um, not not everybody does it. [08:16] Like how many how many people you work [08:17] with actually does TDD, right? [08:19] >> Nobody does it. [08:20] >> But I think we're nobody agrees on what [08:23] TDD is. [08:23] >> We have now the the cost of testing is [08:26] collapsing to nearly zero, right? [08:28] >> Um, especially unit testing, it's [08:29] basically zero now. And then integration [08:31] testing is getting pretty close to zero [08:34] or at least way lower than it used to [08:36] be. Nobody did integration testing [08:37] before because it's huge such a huge [08:38] pain in the butt to build and test. But [08:41] now we're at this point where the all [08:42] that testing is getting cheaper and that [08:44] last mile of reliability falls to what's [08:48] happening in production. [08:50] >> Yeah. [08:50] >> And so we're back to observability again [08:52] which I wanted to go. Um, so what what [08:55] do you guys see changing like from that [08:57] conversation we just had about more [08:59] bugs, more code, higher velocities than [09:01] we've ever seen. Um, what needs to [09:05] change in our observability stacks to [09:07] make this thing keep going to keep [09:09] moving that velocity. [09:12] >> So I I would say um that the data needs [09:16] to change. There's not enough I I'm a [09:17] developer. Uh I it so when I go to my [09:21] observability tool to find evidence of [09:23] the thing that's happening I'm I um I I [09:28] there's a lot of missing data. Okay [09:30] there that I need as a developer. So I [09:32] end up having to try to reproduce a bug [09:34] in order to get the data that I want. [09:35] That shouldn't happen. So so I need [09:38] requests, responses, headers. I need [09:41] session correlated data. Right now [09:43] observability is kind of a big bucket of [09:45] data great for like overall system but [09:49] it but a bug is happening to a person [09:51] typically typical application it happens [09:54] because maybe a complex thing is I did [09:57] this first then this second then this [09:58] third and then this weird thing happened [10:00] in the system I need it all together [10:02] can't get that with traces so I think we [10:04] need to have better data we the data [10:07] needs to evolve [10:09] um and that's an important thing because [10:11] for agents to work Well, to actually fix [10:13] a bug for you, you know, do the work [10:15] that you really want it to do, you have [10:17] to provide it with good context. And uh [10:20] if I as a developer don't have enough [10:23] context from observability out of the [10:25] box, there's no magic that will come [10:27] from passing it to an agent. [10:31] Hika what do you think [10:33] >> to some extent that there are uh what [10:36] Thomas said about some sources of data [10:38] missing like the reticence for the COT [10:42] tracing uh uh approaches to capture [10:45] responses and requests so inputs and [10:47] outputs [10:48] uh which is almost universal with the [10:50] exception of um some niches like uh the [10:53] serverless observability industry where [10:56] you need since you cannot put debuckers [10:59] in there with little control the runtime [11:01] you need the data I am I'm not entirely [11:04] sure I agree with Thomas so the the [11:06] correlation of uh of sessions [11:09] uh yes it's uh there is technically uh [11:13] real user monitoring which uh goes [11:16] beyond uh just [11:18] >> tracing in the client application [11:20] >> basically yeah yeah [11:21] >> that's effectively the the holy grail of [11:24] distributed tracing is to tell you what [11:25] is the impact of issues right and uh the [11:29] actual impact that you should care about [11:30] is what is on the end users. If your [11:33] database kills over but nobody is [11:35] affected, was it really a problem? [11:38] Right? On the other hand, there is also [11:40] too much telemetry. So the uh something [11:43] that we see a lot is not only humans [11:46] drown in bajillion metrics that are [11:49] never queried, uh traces that are never [11:51] seen, logs that are never read, but uh [11:54] AI does it as well. So the our hay [11:57] stacks have become truly large and this [12:01] is a negative side effect of the real [12:04] positive empowering revolution that [12:06] opens brought around. [12:09] I I can tell you I've been I've been [12:11] doing um proprietary observability uh [12:14] before uh with the we went all in with [12:16] open telemetry and u we spent at instana [12:21] uh we fet were competing with other [12:23] companies about who had the best [12:26] telemetry and now we start to compete of [12:28] who does who does the best with the [12:30] telemetry that everybody has. [12:33] Point is everybody having telemetry and [12:36] means of collecting telemetry become so [12:38] ubiquitous [12:40] uh that people are hoarding like smog [12:43] under the mountain and uh [12:45] >> so I kind of heard two things sorry so I [12:48] I hear you saying like I I agree with [12:49] you right there is this problem of if we [12:51] trace everything and we log everything [12:53] and we metric everything we get buried [12:55] in data that 99% of it never gets seen [12:57] by anyone not even bots [13:00] we also have what Thomas is trying to [13:02] say which is we at the long tail of [13:04] these problems that users experience. Um [13:07] it's really hard to get that telemetry [13:09] even if you have tracing because it's [13:10] often in parts of the code that are not [13:12] often instrumented. It's a hot path or [13:14] something where or it has the problem of [13:16] like you said where it just generates so [13:18] much telemetry we can't leave that on. [13:20] So does that open opportunities though [13:23] um for more intelligent ways to to tap [13:26] into our systems? Oo well something that [13:29] I would be profoundly excited to see and [13:31] it has not happened yet is for the [13:34] creation to outstrip the creator. The um [13:38] AI LLM models are trained on on code [13:42] that that people write especially public [13:44] facing code and uh there is very little [13:47] bespoke telemetry in them. It's uh I [13:51] would love to see like the moment that [13:55] uh an application will be generated with [13:57] uh meaningful uh custom instrumentation [14:00] to actually capture the business context [14:03] that's going to be massive. [14:06] >> Um yeah and I I would say that um you [14:09] know just to sort of put a a finer point [14:13] on what I was sort of suggesting is I'd [14:15] like to see the right data not more [14:17] data. Um, and in fact, right now we've [14:21] got a lot of noise because you and we've [14:24] got these big switches like sampling. [14:28] Okay, like you I agree with you. There's [14:30] too much data. Nobody looks at it, [14:32] right? That's because we're trying to [14:33] collect we're keeping the wrong data. [14:36] Um, keeping the wrong data and we're [14:38] missing data. So, it's it's really two [14:40] problems. So, I'd like to have the the [14:41] the right data for when I'm debugging. I [14:44] need to have requests response and [14:45] headers for certain problems. it is a [14:48] reality as a developer you know what [14:50] because otherwise you're guessing what [14:52] was this called with I don't know it was [14:54] called I don't know what it was called [14:55] with so certain cases I need that but I [14:58] don't mean collect it for everything [15:00] because most of the time you're just [15:01] going to throw it away so better data [15:03] collection options than just uh random [15:06] sampling or tail sampling or things like [15:09] that so I think the industry has to move [15:12] in a direction that provides better data [15:16] collection opt [15:18] uh more it's you know I'm interested in [15:21] these sessions this user is having a [15:23] problem right now let me look at their [15:25] data let me turn on debug logging for my [15:28] session because I'm developing something [15:30] and I want to see every not not [15:33] debugging for everybody but debugging [15:35] for me or debugging for a certain person [15:38] or this error happened let me escalate [15:40] for their session more control and also [15:44] options about more data because there [15:46] are you know I I've I've tried many [15:49] times to use open telemetry I love open [15:52] telemetry you know but I've tried to use [15:54] open telemetry traces and logs on their [15:58] own uh with AI agents to fix bugs and [16:01] they and it gets lost because even I as [16:04] a developer don't can't figure out [16:06] exactly what the what the issue is [16:08] unless it's very simple. So that's [16:11] that's how I see that's where I'd like [16:13] to see the industry going more more. [16:15] >> I guess what you're saying is like if if [16:16] the trace has the exception in it, I [16:20] don't need an agent to tell me there's [16:21] an exception right there. [16:23] >> Right. Have it read the exception and [16:25] chase it down for you. But [16:26] >> yeah, [16:28] >> but if it's Yeah. [16:29] >> But you need [16:30] >> Yeah. If you don't know what it was [16:32] called with, if you don't know what [16:33] happened three or four clicks away and [16:37] you know, real user monitoring is good. [16:39] That's front end only. You know, you [16:41] want to see what's happening in your [16:42] system. I don't know if you've ever, you [16:44] know, tried to debug an issue in a in a [16:48] you know, microservices architecture [16:50] that's just sprawling, you know, where [16:53] you got to go deep into the system and [16:55] you you know, your your user is way back [16:58] over there and what they sent to the [16:59] black box of your platform. No, I don't [17:02] know. Lots of other things happened. [17:04] >> It's 20. Yeah. [17:05] >> Yeah. So unless you've got a very kind [17:08] of almost trivial system where you know [17:10] what's sent to your blackbox is all you [17:13] need then uh then you need sort of more [17:16] information and also better collection [17:18] options. I think [17:20] >> the funny part is that uh distributed [17:22] tracing was born actually as an ondemand [17:25] debug signal. When you go and read the [17:28] original Dapper Tracer, the the sampling [17:30] levels were like epic, like one out of [17:32] five. [17:33] >> Yeah. Yeah. [17:33] >> And I hear you saying that what you [17:35] would like is to be able to on demand [17:38] for some uh some traces turn on the [17:41] verbosity plus+ and then for example [17:43] collect u payloads of requests and [17:46] responses. [17:47] >> Well, an agent will need that, you know, [17:49] an agent will need that. [17:51] >> So maybe like we shift back to agents [17:53] for a second, right? Like for me the the [17:55] thing that I get excited about most is [17:57] tool use. Like I think when you when you [18:00] mentioned earlier, right? Like I think [18:01] Michaela, you know, the chat thing I [18:03] could take or leave for the most part [18:04] for myself, but tool calling [18:07] >> with with that that's super cool to me. [18:09] And so does that start to like what what [18:11] opens up in your imagination in terms of [18:13] like when these things are better at [18:15] calling tools um to solve these problems [18:18] we were just talking about like opening [18:20] up a remote debugger and gathering some [18:22] information for you without the client [18:24] ever knowing or stuff like that. What [18:25] what what's in your imagination right [18:26] now? [18:27] >> That's something we're actually building [18:29] right now. Uh [18:30] >> you're building that right now. [18:32] >> Yeah. Yeah. For there is observability [18:35] like the the classic signals. um they go [18:38] a long way but uh not hard enough as we [18:42] have established in this session by now. [18:45] Uh for example, being able to to go and [18:47] connect from an operator to a Kubernetes [18:50] cluster and get on demand information [18:52] that would be expensive to collect and [18:55] statistically useless all of the time [18:58] because when you're when you're needing [18:59] it like only 0.001% of the times you [19:03] have the data then it's it's freaking [19:04] useless. Uh that's important context and [19:07] uh one of the reasons why our haste tags [19:09] have been grown massive is since our [19:12] collection of telemetry is preventative. [19:14] So we we hoard all the telemetry that we [19:17] think we might need in an issue since it [19:19] is almost impossible to go and collect [19:21] telemetry on demand live. Um yeah that's [19:25] something that uh that agents can uh can [19:28] can definitely need. I something that [19:31] sometimes also keeps me thinking is that [19:34] in a future where uh AI actually does [19:37] operate applications and complex ones [19:41] most of the telemetry collection is [19:42] going to be live. So we're probably [19:45] going to have just enough to do [19:46] baselines and then uh uh most of the [19:49] things are going to to be turned on and [19:51] off because the reaction time of [19:53] machines is going to be orders of [19:56] magnitude better than humans and [19:58] machines are effectively always [20:00] available as long as there's compute. [20:02] Instead we get stuck in meetings and [20:03] then we need something [20:05] >> pages us to to get us back. [20:08] >> Yeah. Yeah. And I think I think the tool [20:10] calling is is probably the most exciting [20:13] part of you know what we're experiencing [20:16] now. You know uh it opens up so many [20:19] possibilities. Um you know I mean we [20:22] we're working on a debugging agent [20:24] ourselves and you know integrated with [20:26] Visual Studio Code and what I'd love to [20:28] see is uh you know issue is detected [20:33] data is provided to the agent [20:36] you know code is updated [20:39] run the code with debug with debugging [20:42] enabled step through it provide you know [20:45] the state of the code that you fixed [20:48] back to the language model for [20:50] verification. So beyond you know unit [20:53] tests or or other things actually as a [20:56] do what a developer would do. [20:58] >> So the more exploratory part right like [21:00] exactly [21:01] >> we could we could write debugger scripts [21:02] they just they weren't useful 20 years [21:04] ago they aren't useful now because [21:06] >> I'm only in a debugger when something [21:07] weird has happened. [21:09] >> Yeah. [21:09] >> So what you're saying is like bring the [21:10] agent with tool calling in. Tell me if I [21:12] got this right [21:14] >> to to go do that debugging for you. Put [21:16] set the break points step through all [21:18] those data structures. pull out the [21:19] useful information and then synthesize a [21:21] summary for you [21:22] >> for verification because whatever code [21:25] you want to verify what it did [21:27] >> exactly like this like here is the issue [21:31] I'm changing this with the expectation [21:33] that with these inputs this path and [21:35] this state of the application will exist [21:38] at the end of this function that I just [21:40] fixed. Okay, I could create a test that [21:43] that treats that code as a black box and [21:45] test input output or I could step [21:48] through it and just sometimes I need to [21:50] do that as a developer because just to [21:52] look at this if statement that I've been [21:54] staring at for 30 minutes because I I [21:56] don't know my mind is just switching the [21:57] the thing but I you know I I'm able to [22:00] like look at the state of the code the [22:03] actual state of the code to verify that [22:06] uh that it's actually a fixed then you [22:08] have confidence that it's that it's done [22:10] and you can move on. So it's, you know, [22:12] so tool calling is necessary there. Um, [22:15] but you know, again, it's this goal of [22:17] like self-healing systems. You know, [22:20] there's a bug detected, fix, [22:23] verification, [22:24] repeat. [22:25] >> Yeah, we have a question about that. I I [22:26] can see Michaela has a thought, so I'm [22:28] going to let you talk and then we'll get [22:29] to the question. [22:32] >> For for the people listening, [22:35] Thomas is completely right. Don't think [22:37] however that that replaces tests. That [22:40] is something that No, [22:43] I'm not suggesting that. [22:44] >> I just want for the people in the back. [22:46] >> Yeah, please. [22:47] >> I don't think that's enough. [22:49] >> In fact, the reason I was thinking it is [22:51] because I've talked to dudes who are [22:52] like, I don't need tests. I can just [22:53] open a debugger and just see if it's [22:55] right. [22:57] >> Yeah, sure. I don't remember what my [22:59] code does yesterday. The problem is that [23:01] agents do not have collective memory. So [23:04] you you can check it now and then [23:06] nobody's going to remember the next time [23:08] the code starts that that was supposed [23:09] to keep happening. [23:10] >> That's a good point. I'm [23:11] >> so thank you for for clarifying that [23:13] because I I agree with you. In fact, I [23:15] think that test-driven development I I [23:17] hated it. I never really worked that [23:19] way, but I think I'm going to I forced [23:21] my coding agents to work that way. And [23:24] that's part of that breath work that I [23:25] love. You still do what I do. Do do what [23:27] I tell you. Yes. [23:28] >> Yeah. [23:28] >> I'm doing TD now. TDD now because [23:30] something else is doing it. [23:32] >> Um. [23:32] >> Right. Exactly. I could take credit for [23:35] it now. Yeah. [23:36] >> So, so we got a question from Grant um [23:39] said a few minutes back um a great point [23:43] about a long-held dream of having [23:44] self-healing program. Um obviously [23:46] agents can write code and can also [23:48] receive an error back and try to [23:49] diagnose but in li my limited experience [23:52] these are still separate. Are there any [23:54] examples of even a simple self-healing [23:56] routine that you have se used or seen in [23:58] action? [24:01] >> Not yet but but it's coming. It's It's [24:04] coming. Absolutely. Yeah. [24:08] >> I have a little thing I built that's [24:09] that sort of does it didn't quite do the [24:11] repair, but it's a it was basically like [24:13] running a a py a model on a GPU and it [24:17] wasn't uh doing the the accelerated [24:19] attention and I could it's a black box, [24:21] right? like it's hard to debug and I was [24:23] like [24:24] >> let me just go and see what the model [24:26] thinks and then it started exploring and [24:28] like looking at memory sizes and uh [24:30] memory size and CPU usage and then there [24:32] was one other dimension I can't remember [24:33] right now um and then it would like [24:36] infer [24:37] um that SDPA wasn't on so the the [24:40] accelerated detention wasn't on [24:42] >> based on those parameters and so now [24:44] it's like this little can prompt I have [24:46] I just run it anytime I think that's [24:47] happening like is this happening and it [24:49] goes [24:50] >> nah you're fine or it goes Yep, it's [24:52] happening again. So, it's but that next [24:55] step, which is what Grant asked about. [24:57] >> I think the verification step is the [24:59] thing that's holding it back. Like I I [25:01] don't think [25:02] >> people are ready for the PR [25:05] >> YOLO deployment, you know, uh because [25:09] the verification is not quite there. [25:11] There's still hallucination. There's [25:12] still, you know, needs to be better [25:15] tooling around the verification. [25:17] >> Confidence. Exactly. Right. [25:20] >> Yeah. Because like if we have all these [25:21] functional tests and and great unit [25:23] tests that are free now, we should [25:25] should my old therapist said no sword, [25:28] right? No Um, right. But we should [25:31] be able to just let this stuff rip and [25:32] let it ship to production. [25:33] >> We still need a human in the loop for [25:35] this. [25:35] >> We do. [25:36] >> How how much longer? Not sure. [25:40] >> Either you want to take a guess. [25:42] >> Also, the I think that something. And by [25:45] the way, we are so far off the topic of [25:47] of the webinar. We're going to come back [25:49] to web to observability. [25:51] >> No, the one one last point about [25:54] testing. [25:55] >> I already see the effect. So the same [25:57] thing that observability is now infinite [25:59] telemetry and you can either have [26:00] infinite telemetry or good insights, [26:03] we're going to have the same about tests [26:05] because if everything is tested, who [26:07] knows which ones you need to keep, [26:09] right? [26:13] That I don't know how that is going to [26:14] shake out. [26:17] Well, to shift this back into the [26:18] agentic observability, we have a [26:20] question from Marvin. Um, and Marvin [26:24] asks, "In an eneic observability model [26:26] where agents consume large volumes of [26:28] traces and signals, how do you decide [26:30] what is data worth ingesting and [26:32] retaining versus what should be sampled [26:34] or dropped to manage cost?" [26:38] >> Who wants to go first? [26:39] >> Great question. I I have strong opinions [26:42] about this, but go ahead. Okay. Um okay [26:46] so uh so as so I think that um I think [26:52] the way that data is collected has to [26:54] change right now it's big bucket throw [26:57] into the big bucket and you'll never be [26:59] able to tell what's important what's [27:01] signal and noise because you got these [27:02] little needles in the haystack and you [27:04] can't there's no correlation there's no [27:06] other stuff so I think I think that [27:09] there has to be better correlation of [27:10] data there has to be better um choices [27:13] about what to be saving and that happens [27:16] at the application level you know and [27:18] then in the system itself [27:20] um ra and not with these big switches of [27:24] uh am I s you know sample drop 50% just [27:28] because I don't want to like we have to [27:30] move away from that with better controls [27:34] um and uh you know we we have opinions [27:37] about that at multiplayer you know you [27:40] know we're trying to work in that [27:41] direction um there are others you know, [27:44] charity majors talks about shifting [27:45] left, better developer oriented uh focus [27:49] for observability. I think that really [27:52] has to do with, you know, data [27:55] collection um choices and and and other [27:59] things making it more developer friendly [28:01] uh as opposed to like system level [28:03] summaries and dashboards in which is [28:07] where we kind of live today where you [28:09] only have the choice of sampling uh and [28:12] that doesn't really get you [28:14] >> uh places. So I think I think a number [28:16] of things have to change u and evolve in [28:19] observability for for for things to be [28:23] better have better choices about what to [28:26] get more signal less noise. [28:28] >> Michaela I we've only known each other [28:30] for a little bit. I already know you [28:31] have something to say. A face I can [28:33] tell. So go ahead. [28:35] >> I find that um [28:41] it's not necessarily collection problem. [28:43] It's a retention problem. [28:45] So we have these huge haststacks but the [28:47] problem with the hay stack is only if [28:48] you start piling it up [28:51] the um I take a rather dervenient [28:53] approach to the usefulness of telemetry [28:56] the um and I I I have TED talks about uh [29:01] how telemetry to sample what works what [29:03] doesn't in open telemetry the the [29:05] tragedy of the commons that it is uh the [29:07] holy grail tail sampling uh completely [29:11] not fitting the collector architecture [29:13] of open telemetry If still sampling is a [29:15] time deferred centralized decision, the [29:18] collector is a streaming live compon [29:21] distributed live component and the two [29:22] things really don't smash. But um I find [29:27] that it's not really the collection. So [29:28] the art of creating telemetry is more [29:31] which one you want to store. Every [29:33] single piece of telemetry exists because [29:36] somebody somewhere at some point would [29:39] have been helped by the data existing. [29:41] Nobody creates metrics for funk. The [29:44] point is that statistically they are not [29:46] use useful. [29:48] They are only occasionally useful. [29:50] Something that for example AI could do [29:52] is uh given the fact that telemetry is [29:55] easy to be collected and that those [29:56] instrumentations exist in the [29:58] applications [29:59] uh turn on and off the the dropping of [30:02] telemetry very close to the application. [30:04] It's not directly in the process in the [30:06] closest collector knowing that data [30:09] would exist. For example, um your Java [30:12] application starts having spikes in [30:14] response latency. We all are going to [30:17] think about garbage collection unless [30:20] you use the new one. Let's say we're a [30:22] bit out. We immediately think stop the [30:24] world, right? The fact that you know [30:27] that you could get by by turning off uh [30:30] like opening the dam, getting [30:32] information about garbage collection, [30:33] that gives you the freedom to go [30:35] investigate on on the fly. And this is [30:38] something that AI would be able to do. [30:41] Something else uh is for example there [30:43] is telemetry that is proven to be useful [30:45] is what users are being shown in [30:48] dashboards they actually access that is [30:51] as close to the declaration of [30:52] usefulness that a user can give you. Not [30:55] only I created a dashboard for it but [30:56] I'm actually opening the dashboard. [30:58] Right? So I um I don't know if it's a [31:01] matter of how we collect telemetry. I [31:03] think it's more what do we do with it. [31:06] So I so I would um [31:10] you know say that you know for I think [31:12] what most people do is they auto [31:14] instrument their applications you know [31:15] because [31:17] >> yeah so they auto instrument and what [31:20] you're talking about is when the data is [31:21] received have AI decide what to do with [31:24] it drop it what I think that's I think [31:26] that's good I agree with you that's good [31:28] but it still requires a lot of data to [31:30] be sent and ingested okay so what I'm [31:33] talking about in collecting options are [31:36] like you have your observability [31:40] agents and other stuff like that [31:41] centrally connected in real time with [31:44] your applications. So you don't the app [31:47] doesn't even send the data if it doesn't [31:50] if it's it knows it's going to be [31:51] dropped because what one of the big [31:53] problems that people have is the cost of [31:56] using these tools. So you can't just [31:59] send the data and then like okay we'll [32:01] save part of it with this agent that [32:03] decides I mean it's good but you're [32:05] going to charge for that. So what I [32:07] think the the best the best way to to [32:11] reduce noise, reduce cost is not send [32:14] the data at all if you know that it's [32:16] not going to be used and that's those [32:18] are the kinds of controls I think we [32:20] need to like be you know innovative [32:23] about and and also things like you know [32:25] things like I agree with you that that [32:27] tail sampling is you know is non-starter [32:31] um [32:32] >> no it is a starter just not in open tele [32:34] >> for for a lot for open sle whatever But [32:36] but I think that we also have to think [32:38] beyond the trace. You know a trace is [32:41] useful. It's depth into the system. Um [32:44] but but you know agents for them to do [32:48] real work as coders. I think if we're [32:50] connecting observability with actually [32:52] coding and debugging and fixing um they [32:55] have to have more uh and what I'd love [32:58] to see is a direction where we're having [33:01] more controls real-time connection with [33:03] the application and also so don't even [33:06] send the data if it's not going to be [33:07] used [33:08] various escalation based on on hey the [33:12] garbage collection is having some issues [33:13] let's go around and see start collecting [33:16] some without overwhelming the system [33:18] >> you know and and also [33:22] uh you know more context I would say. [33:24] >> Yeah, I I I see the two perspectives [33:26] coming together here, right? Like I [33:29] Michaela says is pointing out that [33:31] there's a lot we can do by looking at [33:32] what users actually do with the data [33:34] that we've already collected, [33:36] >> right? And then you're saying that like [33:37] we also need to push some controls [33:39] deeper into the applications so we can [33:42] turn the tap on and off or or get a core [33:44] dump or whatever to our agent for that [33:46] analysis so that or not the data at all [33:49] really going to be necessary. They sound [33:51] complimentary to me. [33:52] >> Yeah, I I agree with you. Yeah. [33:54] >> So you can control costs. Yeah. And get [33:57] the right data. [33:58] >> The I think the the discussion about [34:00] where the tap should be and how to [34:02] control it, it doesn't actually have a [34:04] general answer. For example, my answer [34:06] about hey, you have the the collector [34:08] running close to you that can do the [34:10] filtering for you and then you don't [34:12] have an application that changes its [34:14] logic at runtime. [34:16] >> Yes. I I think more people should do [34:18] that. I really do. I I I don't know what [34:20] you experience whether people do that. [34:22] Yeah. Yeah. [34:23] >> What you're saying is actually the [34:24] correct approach in mobile because the [34:26] end user has significant cost. [34:28] >> Absolutely. Yeah. [34:28] >> Because of bandwidth. So it's it really [34:30] depends on the nature of the application [34:32] where where the tabs should be. [34:34] Yes. Yeah, definitely. [34:36] >> Cool. We have a couple questions. So, we [34:38] have about 10 minutes left together. Um, [34:40] so I'm going to try to get through some [34:41] of these questions and uh let's let's [34:43] try to make these Yeah, we got plenty of [34:45] time. Let's just go through them. All [34:47] right. So, not entered. That's your name [34:48] now. Um, [34:51] you mentioned self- remediation, but [34:52] considering the rise of vibe coding and [34:54] overall use of agents to code, how does [34:56] agentic co observability [34:59] I think that's a mistype. How does a [35:01] agentic observability help these agents [35:03] like cloud code now in developers in [35:06] their daily workflows? [35:11] So I think this is like how are you [35:13] using cloud code in in an observability [35:15] workflow or similar whatever agent you [35:17] use. [35:20] Um so uh so at at multiplayer we have a [35:24] full stack session recording and [35:26] debugging agent. So we we capture [35:29] it's built on open telemetry. So we c we [35:33] add the missing data the request [35:34] responses and headers correlate the data [35:37] by session provide it directly to to the [35:39] agent um and are able to say fix this. [35:43] Okay. So I'm using this all the time. [35:45] Our customers are using this this all [35:47] the time. Um, we're working on an [35:49] automated piece uh to this so that right [35:52] now you look at the session, you click [35:54] fix and it it does things uh for you. So [35:57] it takes a lot of that grunt work of [35:59] trying to collect the data and you know [36:00] the missing stuff or replaying and um [36:04] all that kind of stuff. Uh so we use [36:06] that all the time. The goal is to try to [36:09] get it to self-healing. And again that [36:10] that verification step is still a like [36:13] work in progress. I don't know that [36:14] anybody has a a a great solution. So, [36:18] but the coding tools are great for that. [36:20] We can say, you know, here's a session. [36:22] Here's what happened. You know, write [36:23] some unit tests, right? You know, let's [36:27] see whether it works. [36:28] >> I do that all the time, right? Like [36:30] there's a problem here. I can't tell. [36:31] Let's just keep writing tests until it [36:33] shakes out, [36:34] >> right? [36:36] Yeah. Yeah, but but in that case, you [36:37] know, in order to write good tests, you [36:39] have to have the the the good context, [36:41] the data, what actually happens, you [36:43] know, that's the important thing. [36:47] >> Cool. Do you want to take a shot at that [36:49] one, Michaela, or [36:50] >> No, I don't. Uh [36:52] >> All right, let's go to the next [36:53] question. [36:54] >> I I actually would like to take this [36:56] one. The one about autonomous agents? [36:58] >> Yeah, go for it. You want me to read it [36:59] first? I'll read it first so everybody [37:01] hears it. Thank you. As autonomous [37:02] agents begin to make real decisions [37:04] across tools, systems, and clouds, what [37:06] do you believe is the minimum observable [37:07] build observable unit regulators, [37:10] security teams, and operators will need, [37:12] for example, beyond logs and traces to [37:15] confidently answer who acted under what [37:17] authority with what constraints and why? [37:20] And where do you see today's [37:21] observability stacks fundamentally [37:23] failing to provide that level of agent [37:25] level accountability? [37:27] I um I do not believe that the problem [37:30] is the types of telemetry we collect. I [37:33] believe it's how we use the tools we're [37:35] given. So uh logs are fantastic to [37:38] describe events in time and in open ter [37:41] you can correlate them with traces. Uh [37:43] the problem is most of the logs are [37:44] useless. So I actually had a talk at the [37:47] last CubeCon where I showed that of the [37:49] logs getting into their zero less than [37:52] 1% are actually error and fatal and all [37:54] the rest is probably a waste of money [37:56] especially if they're sent all the time. [37:59] The um we had um a discussion about [38:03] traces and sampling. Tracing sampler are [38:06] great tools. It's just that on the one [38:08] hand it's difficult to use them [38:10] correctly uh because of limitations of [38:13] the current technology and on the other [38:15] hand the literacy about um how [38:18] observability works is uh not great in [38:22] terms it's complex stuff something that [38:25] I I used to say is that um and this is [38:30] where for example my view with respect [38:32] to to uh for example charity that you [38:34] mentioned is a bit different Um some [38:37] people say you should shift left. So [38:39] every developer should become in some [38:40] sense an observability expert. And I [38:43] look at it and I'm thinking most [38:45] developers are challenged in making [38:47] their current systems work without [38:48] getting more responsibility and having [38:51] the cognitive burden [38:54] >> of everybody else's crap [38:58] can make my brain ball. Right. [39:00] So the um it's more a matter of um [39:06] getting better at generating telemetry [39:08] that is useful. And this goes back to my [39:11] initial comment where I I'm hoping to [39:13] see soon that AI actually gets good not [39:17] only generating application code but it [39:19] telemetry to explain what it does and [39:21] what it shouldn't. [39:24] >> Okay. [39:24] >> And where [39:25] >> I guess we what I'm curious about and we [39:27] have a couple more questions I want to [39:28] get to so we'll go a little quicker. Um, [39:30] what do we need to add to the [39:31] observability stack that our current [39:33] suite of the the three-legged stool [39:35] isn't getting us? [39:37] What do you want to add on a quick [39:39] answer? [39:41] >> Do I go first? [39:42] >> Yeah, go for it. [39:43] >> There is profiling. Uh, for example, [39:45] it's in the works with open telemetry. [39:47] Thomas has a great point about input and [39:49] outputs, especially not for every single [39:51] one. Uh, reuser monitoring is is a a [39:55] completely overseen hero. Um the uh [40:00] there are of course concerns about the [40:02] cost and things but in reality there are [40:04] solutions to that. It's just that the [40:05] current industry has effectively grown [40:10] um very successful in uh exploiting [40:14] people that generate too much telemetry [40:16] instead of generating the correct one. [40:18] Uh so there we need probably um somebody [40:21] to to change the rules of the game to to [40:24] price the signal and the quality rather [40:26] than just how big is your hay stack. [40:29] >> Fair enough. Did you want to add [40:30] anything Thomas before we go? [40:31] >> No, I I think I totally agree. [40:34] >> Cool. Well, Benny asks, "Have you seen [40:37] Python generated knowledge graphs and [40:39] querying recent PR release [40:41] notifications? Guessing via GitHub can [40:44] be uh about a hundred in a day [40:46] successfully used to better outcomes." [40:49] I think this is asking about [40:54] using knowledge graphs with with the [40:56] agents. Have either of you experimented [40:58] with that? [40:59] >> I know there's a bunch of people working [41:00] in that space. [41:03] >> Interesting. I mean, that's an [41:04] interesting [41:05] idea. Um I have not seen knowledge [41:08] graphs generated from PRs in order to I [41:12] guess this would be to to to kind of get [41:16] you know to do the [41:18] you know the review itself to understand [41:21] the relationships between the code and [41:22] the and the pull request and maybe to [41:25] get uh to detect whether there's an [41:29] issue with what's being suggested. I [41:32] mean, boy, would I like that in some of [41:33] the huge PRs that I've had to review. Uh [41:37] um but I guess if you're trying to pull [41:39] out a knowledge graph for a PR, then you [41:42] probably want to cut that PR down into [41:45] bite-sized nibbles rather than like, you [41:49] know, maybe I would take a more human [41:51] approach to um to that, but no, I [41:54] haven't I haven't seen knowledge graphs [41:56] being used in that way. But but since [41:58] we're talking about GitHub and history, [42:00] uh the something that has been [42:02] historically pretty pretty separate is [42:05] uh the history of how software evolves [42:08] and the infrastructure changes that [42:10] accompany deployment of software [42:12] >> and observability on the other side. [42:14] >> Yes. [42:14] >> And there is a huge potential in [42:16] effectively marrying the versioning data [42:19] you have like service. in open telemetry [42:22] with matching code changes because that [42:25] is something that agents at scale could [42:27] probably infer um uh how the changes are [42:30] related. [42:31] >> That's a great idea of marrying the [42:32] observability with the changes in the [42:34] code and being able to use an agent to [42:36] connect the two because probably the bug [42:39] that you're seeing today is something [42:42] that was injected in code in in you know [42:45] in the history in recent week or so [42:49] different PR most of the time. [42:51] >> Yeah, exactly. Exactly. And that's I [42:53] think that's important thing. [42:54] >> It's kind of strange how infrequently my [42:56] agent will actually go look at the git [42:57] history and I'll be like, [42:58] >> "Yeah, yeah, [42:59] >> hey, we did that 3 days ago, bud. Go [43:01] take go look at the G." They'll be like, [43:02] "Oh, look, I discovered this problem." [43:05] >> This is what I said about agents missing [43:06] collective memory, right? [43:08] >> Yeah. [43:08] >> For them, it's always day one. It's [43:09] always the new intern. [43:11] >> Yes, [43:13] >> we could be so lucky. Um, we have I [43:16] think we're just about at the end of our [43:17] time together. Um, I'm going to give you [43:19] each. We have about two minutes left. Uh [43:21] just sum up your thoughts if you'd like [43:22] to and then we'll let our audience go. [43:24] We really appreciate you all coming. [43:27] >> Um okay, great. I'll just jump. Yeah. [43:29] So, uh you know, I I think this is such [43:32] an exciting area. We're at the very [43:33] beginning. I think to it's it's a [43:35] wonderful time to be a software [43:37] developer. We're going to see not so [43:39] we're starting to see how co changing [43:41] coding is changing for us. We see that [43:43] happening today. We're going to see you [43:45] know uh running a system debugging [43:48] things also change greatly in the coming [43:51] you know months uh and it's what a time [43:55] to be alive as they say [43:56] >> what a time yeah Michaela [43:58] >> we indeed live in interesting times uh [44:01] there is going uh there is learning that [44:04] has to be had but it's also um the first [44:07] time in my career where there is [44:09] something entirely utterly novel the uh [44:12] the fact that now We can instead of [44:15] shoving in the faces of users all the [44:17] time static dashboards [44:20] overripe with information hoping that [44:22] something catches the eye of the person [44:24] but instead having a sort of I like to [44:27] use the expression like premortem [44:33] >> and that's good [44:34] >> because humans were excellent at [44:35] ingesting what is going on and what to [44:37] do about it. Um when I say these things [44:40] that usually people ask me do you think [44:42] static dashboards are dead? Uh the [44:44] answer is uh no but uh their usefulness [44:47] is going to be curtailed to fewer use [44:49] cases because when you can rely on AI [44:51] actually [44:53] um having a pretty good analysis and [44:55] showing you effectively build you [44:57] dashboards in real time that explain the [44:58] situation then the static ones are more [45:01] for the kind of um checking how much [45:04] money you're doing. [45:06] >> Yeah. the more the more static. [45:07] >> I totally agree with you by the way. I [45:09] think that's [45:09] >> Yeah, I do too. We're at time. Um [45:13] >> thank you so much everyone. Thank you [45:14] too for the discussion. Uh this was [45:16] really fun and we hope to see you at the [45:18] next uh lead dev panel. [45:21] >> Have a great day. [45:22] >> Thank you. [45:23] >> Bye.