Do We Need AI Agents to Fix Our AI Agents - CourtneyR.dev's 2nd Brain

Do We Need AI Agents to Fix Our AI Agents? ![rw-book-cover](https://img.youtube.com/vi/UGWD1zN__xU/maxresdefault.jpg) 🔗 **Source:** [Do We Need AI Agents to Fix Our AI Agents?](https://www.youtube.com/watch?v=UGWD1zN__xU) > [!note]+ Document Notes > tags: [ai-agents, observability, devops, incident-response, sre, automation] > author: [MISSING] > source: Lead Dev > reference: https://www.youtube.com/watch?v=UGWD1zN__xU > date created: {{date:YYYY-MM-DD}} {{time:HH:mm}} > > # 🏷️ Title: Do we need AI agents to fix our AI agents? >  ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ## 📝 Summary - A panel discussion on AI agents in observability and operations, featuring experts from Google Cloud, Delivery Hero, and Palo Alto Networks. - Key theme: Moving from reactive to proactive observability using AI agents while managing non-deterministic systems. - Organizations are experimenting with agents for incident response, auto-remediation, and root cause analysis. - Tracing and evaluation frameworks are essential for building trust in agent systems. - SRE roles are evolving toward business-aware generalists supported by technical agents. ## 📖 Main Content The panel explored how AI agents are transforming observability and operations. The core challenge is that "writing code was never the bottleneck" - but AI-generated code creates new operational complexities requiring proactive rather than reactive approaches. ### Key Perspectives on Agentic Operations **Google's Jason Davenport** emphasized that observability fundamentals haven't changed, but teams now must manage non-deterministic systems. The key is applying rigorous software engineering principles while handling agents that may behave differently each time. **Delivery Hero's Rodrigue Schaefer** outlined multiple dimensions of agent observability: - Using agents for incident response and auto-remediation - Observing customer-facing agents for drift detection - Managing agent-generated code in production - Monitoring the code generation process itself **Palo Alto's Ryan Pierce** focused on the operational shift where practitioners work with software agents rather than performing all tasks manually. ### Implementation Approaches The panelists shared practical deployment strategies: 1. **Start Simple**: Begin with basic tasks like incident summarization and executive communication during outages 2. **Build Trust Gradually**: Progress from advisory roles to executing well-defined runbooks in controlled scenarios 3. **Enable Comprehensive Tracing**: Use OpenTelemetry and similar tools to capture agent reasoning chains and tool calls 4. **Develop Strong Evaluation Frameworks**: Create datasets and continuous testing to validate agent behavior ### Technical Requirements Key technical infrastructure includes: - **Tracing**: Capturing every agent step, reasoning chain, and tool interaction - **Guardrails**: Rate limiting, API backoffs, and permission boundaries - **Evaluation**: Continuous testing against expected outcomes - **Context Engineering**: Managing what information agents receive ## 🗒️ Notes #### ⏱️ [00:01:30](https://www.youtube.com/watch?v=UGWD1zN__xU&t=90) Introduction and Problem Statement - Jennifer Rigggins introduces the core challenge: AI-generated code creates more operational complexity - Need to move from reactive to proactive observability approaches - Panel includes experts from major organizations at different stages of adoption #### ⏱️ [00:05:12](https://www.youtube.com/watch?v=UGWD1zN__xU&t=312) Ryan Pierce on Agentic Era Operations - Shift from manual practitioners to software agents augmenting or replacing human tasks - Agents can solve problems teams haven't yet defined processes for - Continuous understanding and iteration throughout the process #### ⏱️ [00:06:45](https://www.youtube.com/watch?v=UGWD1zN__xU&t=405) Jason's View on Managing Non-Determinism - Core observability principles remain the same - Main change: managing systems that don't behave identically each time - Apply extreme software engineering best practices to manage agent behavior - Dashboard fatigue problem gets solved with agents taking action on metrics #### ⏱️ [00:08:30](https://www.youtube.com/watch?v=UGWD1zN__xU&t=510) Rodrigue's Multi-Dimensional Agent Challenge - Four aspects: incident response agents, observing business-facing agents, agent-generated production code, monitoring code generation itself - Engineers losing familiarity with code they didn't write personally - Challenge of detecting agent drift in non-deterministic systems #### ⏱️ [00:12:15](https://www.youtube.com/watch?v=UGWD1zN__xU&t=735) Delivery Hero's Scale and Approach - 4,000+ engineers across global tech hubs (Dubai, Riyadh, Barcelona, Singapore, Seoul, Buenos Aires, Montevideo) - Experimenting with autonomous software delivery agents and AI ops - Focus on preventing engineer burnout from increased incident load #### ⏱️ [00:15:30](https://www.youtube.com/watch?v=UGWD1zN__xU&t=930) Building Trust in Agents - Start with advisory roles, leave decisions to human engineers - Progress to executing runbooks in well-defined scenarios - Eventually handle more critical operations based on evidence - Most organizations not ready to let agents delete infrastructure resources #### ⏱️ [00:18:45](https://www.youtube.com/watch?v=UGWD1zN__xU&t=1125) Poll Results on Agent Tracing - 18% have adopted tracing for agents - 30% unsure about current tracing implementation - 52% have not adopted agent tracing yet #### ⏱️ [00:20:00](https://www.youtube.com/watch?v=UGWD1zN__xU&t=1200) Ryan on Tracing Implementation - Tracing shows agent reasoning, tool usage, latency, and security issues - Use OpenTelemetry to trace every step and request - Risk-based approach: higher-risk use cases need more comprehensive tracing - Control plane helps keep only valuable traces #### ⏱️ [00:23:15](https://www.youtube.com/watch?v=UGWD1zN__xU&t=1395) Google's Tracing and Evaluation Strategy - Focus on evaluation frameworks to determine expected outcomes - Challenge: building training data when no agent exists yet - Use synthetic data generation and verbose logging initially - Capture reasoning chains to understand unexpected behavior (like 500 database calls) #### ⏱️ [00:27:30](https://www.youtube.com/watch?v=UGWD1zN__xU&t=1650) Implementing Guardrails - Apply good software engineering: API limits, backoffs, service defense - Let agents pressure-test systems to find undefended services - Need anomaly detection as anomalies happen, not 10-15 minutes later - Enabling telemetry can be as simple as 1-2 lines of Python code #### ⏱️ [00:30:45](https://www.youtube.com/watch?v=UGWD1zN__xU&t=1845) Choosing First Agent Deployments - Focus on undocumented incident response processes - Document decision-making chains that happen in incident rooms - Start with low-risk scenarios and maintain human oversight - Map agents into existing harnesses with defined signals and mitigation actions #### ⏱️ [00:34:00](https://www.youtube.com/watch?v=UGWD1zN__xU&t=2040) Rodrigue's Practical Starting Points - Simple wins: automated incident summarization for late-joining responders - Executive communication during incidents to free up technical teams - Experiment with existing SRE agent solutions rather than building from scratch - Industry players (observability companies, hyperscalers, startups) all experimenting #### ⏱️ [00:37:15](https://www.youtube.com/watch?v=UGWD1zN__xU&t=2235) Measuring Agent Performance - Business KPIs as primary metrics for agent success - Technical metrics: token usage, context window utilization, model performance - Security metrics: preventing unauthorized actions (jailbreaking) - Evaluation-centric approach to measuring outcomes vs. inputs #### ⏱️ [00:40:30](https://www.youtube.com/watch?v=UGWD1zN__xU&t=2430) Ryan's Layered Metrics Approach - Business outcome layer: independent success vs. human escalation vs. failure - Model optimization layer: right-sizing models for cost vs. performance - Technical performance layer: latency, time-to-first-token, total tokens - Quality layer: treating hallucinations and bias as error conditions #### ⏱️ [00:42:45](https://www.youtube.com/watch?v=UGWD1zN__xU&t=2565) Evolution of SRE Roles - SREs becoming business experts, not just technical specialists - Need deeper understanding of customer workflows and expectations - Agents help provide technical context for unfamiliar codebases - Shift from "you built it, you run it" to broader-scoped, agent-supported teams #### ⏱️ [00:45:30](https://www.youtube.com/watch?v=UGWD1zN__xU&t=2730) Future Organizational Structure - Move away from teams centered around single product capabilities - Single teams managing 5-10 different capabilities with agent support - Generic on-call teams with broader scope enabled by agent-provided context - Responders focus on business flows while agents handle technical deep-dives ## 📚 References - [Lead Dev Community](https://leaddev.com/) - Engineering management community - [Chronosphere](https://chronosphere.io/) - Observability platform sponsor - [Google Cloud](https://cloud.google.com/) - Cloud platform and AI services - [OpenTelemetry](https://opentelemetry.io/) - Observability framework mentioned for tracing - Manning Publications - "Cloud Observability in Action" book offer ## 🗣️ Paraphrase This panel discussion explores how AI agents are reshaping observability and operations practices. The speakers argue that while AI hasn't changed core observability principles, it has introduced the challenge of managing non-deterministic systems that behave differently each time they run. The solution involves applying rigorous software engineering practices while building comprehensive tracing and evaluation frameworks. Organizations are starting with low-risk use cases like incident summarization before progressing to automated remediation. The role of SREs is evolving from deep technical specialists to business-aware generalists who work with agents to manage broader system scopes. Success requires treating agent deployment as a gradual trust-building process rather than an immediate replacement of human judgment. ## 📰 Newsletter 🤖 **Agents Are Coming to Your Ops Team** - This fascinating Lead Dev panel reveals how AI agents are transforming observability and incident response. The key insight? Start small with boring tasks like incident summarization, then gradually build trust toward auto-remediation. Google's doing it with 1-2 lines of Python for telemetry, while Delivery Hero's 4,000 engineers are experimenting with autonomous delivery agents. The secret sauce: comprehensive tracing with OpenTelemetry, strong evaluation frameworks, and treating your SREs as business consultants rather than code archaeologists. Plus, hot tip: if you've been struggling to get observability budgets approved, just say it's for AI! 😉 Perfect timing as we're all figuring out how to make agents helpful rather than just another thing to monitor. ## ✅ Action Items #### DevRel - Create educational content on OpenTelemetry setup for AI agent tracing - Develop evaluation framework templates for common agent use cases - Build demo scenarios showing agent incident response workflows - Document best practices for gradual agent trust-building in production #### WordPress - **Core Performance**: Investigate agents for automated performance regression detection and basic remediation - **Support**: Experiment with agents for first-level ticket triage and response summarization - **Community**: Create agent-assisted moderation tools for forums and Slack channels - **Documentation**: Deploy agents for keeping documentation synchronized across repositories #### Community - **Maintainers**: Develop agent assistants for routine issue triage and contributor onboarding - **Contributors**: Create agents that help match contributor skills with appropriate first issues - **Sponsors**: Build agent-driven impact reporting and milestone tracking systems #### OSPO - **Security Management**: Implement agent-assisted vulnerability analysis and coordinated disclosure workflows - **License Compliance**: Deploy agents for automated license scanning and policy violation detection - **Community Engagement**: Create agents for monitoring upstream health and contribution opportunities ## 🧪 Hypotheses - Organizations that start with low-risk agent deployments (summarization, communication) will have higher success rates with critical automation later - Comprehensive agent tracing reduces time-to-resolution for agent-related incidents by 40-60% - SRE teams using agent-assisted workflows can manage 3-5x broader system scopes without proportional headcount increases - Agent evaluation frameworks that include business outcome metrics predict production success better than technical metrics alone - Organizations treating agent deployment as trust-building rather than replacement have higher team satisfaction scores ### 📊 Methodology - Longitudinal studies tracking organizations through agent adoption phases - A/B testing of agent-assisted vs. traditional incident response workflows - Survey research on SRE role satisfaction and scope changes - Quantitative analysis of tracing data correlation with incident resolution times - Comparative case studies across different organizational scales (startup vs. enterprise) ### ❓ Questions - How do you measure business impact of agent-driven observability improvements? - What organizational changes are needed to support broader-scoped SRE teams? - How can smaller organizations compete with hyperscaler agent capabilities? - What are the failure modes when agent trust-building is rushed? - How do you maintain human expertise when agents handle routine tasks? ### 🔮 Future Research - Long-term studies on SRE skill evolution in agent-augmented environments - Development of standardized agent evaluation benchmarks for observability - Investigation of agent coordination patterns in complex incident scenarios - Research on optimal human-agent collaboration models for different organization sizes - Analysis of agent-driven organizational structure changes across industries ### 💡 Implications - **Practice**: Shift from reactive monitoring to proactive agent-driven operations requires new skills and organizational structures - **Policy**: Need for governance frameworks around agent decision-making in production systems - **Security**: Agent permissions and guardrails become critical infrastructure components - **Education**: SRE training must expand beyond technical skills to include business domain knowledge - **Funding**: Organizations should budget for increased telemetry and evaluation infrastructure costs ### 🧠 Learning Styles Assessment - Reading level estimate: College level (Flesch-Kincaid ~12th grade) - technical panel discussion with industry jargon #### 👀 Visual learners - Present: Transcript format with clear speaker attribution and timestamps - Improve: Add system architecture diagrams, agent workflow visualizations, metrics dashboards screenshots #### 🔊 Aural learners - Present: Original video format with multiple speakers and conversational flow - Improve: Chapter markers, audio-only version, speaker introduction audio cues #### 🗣️ Verbal learners - Present: Clear definitions of terms like "non-deterministic," detailed explanations of concepts - Improve: Glossary section, more analogies for complex technical concepts, plain-language summaries #### 🧩 Kinesthetic learners - Present: Actionable implementation steps, practical examples from real organizations - Improve: Step-by-step setup guides, hands-on labs for OpenTelemetry configuration, interactive examples #### 🧠 Logical learners - Present: Structured progression from simple to complex use cases, clear cause-effect relationships - Improve: Decision trees for agent deployment, flowcharts for trust-building processes, metrics frameworks #### 🧑‍🤝‍🧑 Social learners - Present: Panel discussion format, multiple perspectives, community references - Improve: Discussion prompts, peer collaboration exercises, community case study sharing #### 🧘‍♀️ Solitary learners - Present: Comprehensive transcript for individual study, reflection-worthy insights - Improve: Self-assessment questionnaires, individual experimentation guides, journaling prompts #### 🌿 Naturalistic learners - Present: Organic evolution of organizational practices, ecosystem thinking about agent interactions - Improve: Biological analogies for agent behavior, sustainability considerations for compute resources ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Clear heading structure, speaker attributions, timestamp navigation - Improve: More descriptive alt text for any embedded media, proper ARIA labels for timestamp links #### Keyboard navigation - Present: Standard web navigation patterns - Improve: Skip links to different speakers, keyboard shortcuts for timestamp navigation #### Contrast and readability - Present: Standard text formatting meets basic contrast requirements - Improve: Ensure timestamp links maintain sufficient contrast, consider dark mode compatibility #### Text resizing and reflow - Present: Responsive text formatting - Improve: Test at 200% zoom, ensure timestamp links don't break layout #### Media and captions - Present: Full transcript provided for video content - Improve: Downloadable transcript, speaker identification improvements #### Forms and interactions - Present: No forms in this content - Improve: N/A for this content type ### 🌍 DEIB Assessment #### Cultural breadth - Present: Global perspective from Delivery Hero's distributed teams - Improve: Include perspectives from smaller organizations, non-Western operational practices #### Inclusive language - Present: Professional technical language, avoids exclusionary jargon mostly - Improve: Define more acronyms, expand on industry-specific terms, provide context for newcomers #### Neurodiversity support - Present: Well-structured content with clear sections and timestamps - Improve: More frequent headings, consistent formatting patterns, reduced cognitive load summaries #### Representation - Present: Male-dominated panel typical of current industry leadership - Improve: Highlight diverse voices in follow-up content, ensure inclusive examples in action items #### Psychological safety - Present: Professional tone, acknowledges uncertainty and experimentation - Improve: More explicit encouragement for questions, validation of different experience levels ### ✅ Logical Validity #### Strong reasoning - **Inductive reasoning**: Panel draws general principles from specific organizational experiences (Google, Delivery Hero, Palo Alto Networks) - **Causal reasoning**: Clear connections between agent implementation challenges and proposed solutions like comprehensive tracing - **Analogical reasoning**: Effective comparison of agent risk assessment to manufacturing quality control (toys vs. airplanes) #### Fallacies - **Appeal to authority**: Some claims rely heavily on speaker credentials without supporting evidence - could strengthen with more concrete data - **Hasty generalization**: Limited sample size of organizations (3 companies) used to make broad industry claims - **Survivorship bias**: Focus on successful implementations without discussing failure cases #### Tone of voice - **Collaborative and exploratory**: "It's a journey, not a light switch" - acknowledges uncertainty while maintaining optimism - **Pragmatic**: Emphasis on starting small and building trust gradually rather than revolutionary change ### 🛡️ DARVO Analysis - **Deny**: No evidence of denial patterns in this technical discussion - **Attack**: No personal attacks or credibility undermining observed - **Reverse victim/offender**: Not applicable to this content type - **Assessment**: Clean technical discussion focused on problem-solving rather than blame assignment ## 🎯 Conclusion The panel reveals that successful agent adoption in observability requires a fundamental shift in thinking - from revolutionary replacement to evolutionary augmentation. The key insight is treating agent deployment as a trust-building journey rather than a technology switch. Organizations should start with boring, low-risk tasks like incident summarization, invest heavily in tracing and evaluation infrastructure, and gradually expand agent capabilities based on demonstrated reliability. The future of SRE work involves business-aware generalists working with specialized agents rather than deep technical specialists working alone. This transformation requires new organizational structures, expanded skill sets, and a commitment to measuring both technical performance and business outcomes. ## 🔄 Refactor Appendix — Do we need AI agents to fix our AI agents? ### 📚 Literature Atoms (H3) #### Agent observability requires managing non-deterministic systems with traditional engineering principles > [!quote] ID:rw_panel_001 > "Observability hasn't really changed other than the fact that we now have to manage non-determinism in addition to the rest of the stack... It's not that we're trying to do all these new things, but what it does mean is that we have to manage things that may not behave the same way every time." **Context & use**: - Core insight from Google Cloud's perspective on agent integration - Applies to any organization implementing agents in production systems - Foundation for understanding why traditional monitoring approaches need enhancement **Tags**: #xref #operations **Highlight**: Panel discussion at ~6:45 timestamp  #### Agent trust-building should progress through advisory, runbook, and autonomous phases > [!quote] ID:rw_panel_002 > "I think in the beginning it's more about supporting with advice... But we leave the decision to the on call engineer to actually act... The next step would really be to give agents the ability to execute runbooks in well-defined scenarios" **Context & use**: - Delivery Hero's phased approach to agent deployment in critical systems - Risk mitigation strategy for production agent implementation - Practical framework for organizational change management **Tags**: #xref #implementation-strategy **Highlight**: Panel discussion at ~15:30 timestamp  #### Comprehensive agent tracing enables debugging non-deterministic behavior patterns > [!quote] ID:rw_panel_003 > "Tracing is going to be the mechanism to let you see how it's reasoning, what tool calls it's using, what it's not using that you think it should, where there might be latency or performance issues or even security issues" **Context & use**: - Technical requirement for agent observability implementation - Addresses the debugging challenge of non-deterministic systems - Foundation for building confidence in agent decision-making **Tags**: #xref #debugging **Highlight**: Panel discussion at ~20:00 timestamp  ### 🧠 Permanent Candidates (H4) #### Agent deployment success correlates with gradual trust-building approach **Claim**: Organizations achieve higher agent adoption success rates when they treat deployment as iterative trust-building rather than immediate automation replacement. **Evidence pointer**: [[#Agent trust-building should progress through advisory, runbook, and autonomous phases]] **Scope & conditions**: Applies to critical systems where failure has significant business impact; may not apply to isolated or non-critical use cases **Links**: - [[Change management]] — because applies-in technology adoption - [[Risk mitigation]] — because extends proven strategies - [[Production systems]] — because contrasts with development experimentation  ### ✏️ Fleeting Queue - How do you measure business impact of improved incident response times with agent assistance? #fleeting #metrics - What are the failure modes when organizations skip the advisory phase and jump to autonomous agents? #fleeting #risk-analysis - How does agent-assisted observability change team hiring and skill requirements? #fleeting #workforce-planning ### → Things 3 (max 2 actions) - [ ] Research OpenTelemetry integration options for agent tracing in WordPress infrastructure — platform-engineering #things - [ ] Document current incident response process to identify automation candidates — devops #things --- ## Full Transcript [00:12] Hello. Hello everyone and a huge thank [00:15] you to joining us today. This is a lead [00:18] dev online panel discussion created in [00:20] partnership with Chronosphere and Google [00:22] Cloud. The discussion will last about 45 [00:25] minutes. Writing code was never the [00:28] bottleneck. If anything, AI generated [00:31] code deployed by AI agents adds more [00:34] roadblocks to production and makes it [00:36] harder than ever to debug. [00:39] Ops and site reliability engineers are [00:41] stuck being reactive. [00:44] So today we're diving into observability [00:46] or 011y [00:49] if you're cool monitoring and operations [00:52] in response to this AI boost and where [00:55] AI can help your observability move into [00:58] this proactive idea. So, not just [01:02] observing to be reactive and to kind of [01:05] anticipate change, which we hope those [01:07] unknown unknowns as it's called, but [01:10] also if you have the right guard rails [01:12] around it, maybe we can start using AI [01:15] to help where the real bottlenecks were [01:17] and to be proactive. So, let's get [01:19] started with some introductions. I'm [01:20] Jennifer Rigggins. I'm a freelance [01:22] technology journalist, including with [01:24] lead dev. And today, I am joined by [01:27] Rodrig, Jason, and Ryan. ready to share [01:29] their experiences and insights. So, can [01:32] you please introduce yourselves? Rodri, [01:35] if you want to go first. [01:37] >> Sure. So, my name is Rodri Schaefer. I'm [01:39] working for Delivery Hero, uh, one of [01:42] the world's leading delivery platforms [01:44] out there. And my responsibility there [01:47] is uh mainly leading tech foundations, [01:49] which is an uh internal organization [01:52] that provides infrastructure and tooling [01:54] for software engineers, data engineers, [01:56] ML engineers and so on. so that they can [01:59] do their job more efficiently with [02:01] higher quality and with with higher [02:04] reliability overall. [02:07] >> Okay, thank you very much Rodri Jason. [02:11] >> Hey, nice to meet everyone out there. My [02:12] name is Jason Davenport. I'm an area [02:14] tech lead with Google Cloud. My focus is [02:17] on making sure that developers have a [02:18] great developer experience when working [02:20] with Google Cloud products and that we [02:23] help developers understand all the great [02:25] things they can do and then also take [02:27] friction back to our team so we can [02:29] improve the products even further. So [02:31] super cool to be a part of this. Thank [02:33] you Jennifer and the panel for the [02:35] opportunity. [02:36] >> Thank you so much. And finally, Ryan, [02:38] please. [02:40] >> Hi, I'm Ryan Pierce. I'm a systems [02:41] engineer at Palo Alto Networks. actually [02:44] joined PaloAlto um from Chronosphere was [02:47] recently acquired to bring observability [02:49] to PaloAlto's existing product portfolio [02:52] been working in observability for about [02:54] 10 years across different vendors. [02:56] Really excited to talk today. Uh we've [02:58] been able to do a lot of cool work with [03:00] AI vendors, companies building AI native [03:02] applications. Uh so it's a really [03:04] exciting topic to dig into. [03:06] >> Okay, great. Thank you so much all of [03:08] you and thank you again listeners. So [03:11] Ryan, let's kick off by I want everyone [03:13] to answer this question please. What is [03:16] operations in this to quote Taylor Swift [03:20] agentic era? [03:23] Uh for me and for our customers, I think [03:26] the agentic era, it's switching away [03:28] from having manual practitioners and [03:31] operators doing every task and looking [03:33] at dashboards and now understanding how [03:36] to actually plug in pieces of software [03:38] that are agents that can augment or take [03:41] on some of those responsibilities uh or [03:43] even solve things that maybe the team [03:45] hasn't thought about as a process ahead [03:47] of time. Uh and then being able to [03:49] continually understand and iterate [03:51] throughout [03:53] Okay, Jason, what's Google's view on [03:56] this [03:57] or yours? [04:00] >> It's a It's a good question. To me, um, [04:05] observability hasn't really changed [04:07] other than the fact that we now have to [04:09] manage non-determinism [04:11] in addition to the rest of the stack. [04:14] And you know what I mean by that? all [04:16] the things that we've learned about and [04:17] we know, you know, from software [04:19] engineering and and S sur and and all [04:22] those stacks. It's not that it's [04:24] different, right? It's not that we're [04:25] trying to do all these like new things [04:27] and we have like new platforms and all [04:29] this, but what does mean is that we have [04:31] to manage things that may not behave the [04:34] same way every time. So how do we you [04:37] know leverage the things that we know [04:38] take software engineering principles [04:40] really you know to the best practice [04:42] side in in the extreme to manage agents [04:46] and manage things that happen with [04:47] agents and all this stuff with it. So [04:50] you I think you'll hear a lot of great [04:52] things from you know folks here today [04:53] who are implementing that but you know a [04:56] lot of it is right how do we just manage [04:57] the non-determinism with the things that [04:59] we have or or extending them so we can [05:01] also manage it as a part of that. I [05:04] guess we could objectively say it could [05:06] be harder in this time period. [05:10] >> Yeah, I think you could say you on the [05:12] one hand it's uh harder. On the other [05:15] hand, you know, take take even the [05:16] example of a dashboard. You know, one of [05:18] the things we always struggle with a [05:19] dashboard is the last mile delivery. You [05:22] with agents it makes that last mile, you [05:24] know, heart easier, right? Because we [05:25] can have an agent that takes action [05:27] based on a metric rather than having to [05:29] sit through, you know, dashboards of [05:30] things to find, you know, a bug, right? [05:33] or something in that. So, you know, the [05:35] AI agents world is is very interesting [05:37] to think about it and it's how do we [05:39] build trust in the platform and the [05:41] system? So, you know, we're working as [05:43] as a pair with the agent in in many of [05:45] these cases. [05:47] >> Absolutely. Dashboard fatigue is real. [05:50] Rodrigue, what does operations mean in [05:53] the aenic era for you and your team? [05:56] Yeah, I think there are several aspects [05:58] uh that are quite different from each [06:00] other in a way when we talk about agents [06:02] and observability or operations. one is [06:06] uh using agents for uh instant response [06:09] for example Jason touched on this right [06:11] I think that's an extremely promising [06:13] technology and is also absolutely needed [06:17] because the more production code we [06:20] produce with agents right in gentic [06:22] engineering in general the more we will [06:25] need to operate it in production and [06:28] just if let's say if we stick with to [06:31] the same quality level as before it [06:34] would just linearly scale the number of [06:36] incidents. So we have to have something [06:39] that helps us there and uh that's where [06:42] I'm very optimistic about uh agents that [06:46] can help with triaging root cause [06:48] analyzis even executing runbooks in uh [06:52] predefined scenarios for example. So [06:54] that's one aspect. Then there is the [06:56] aspect of observing [06:59] agents that are embedded in consumer or [07:02] business facing products, right? For [07:04] example, a customer service agent that's [07:07] um that's answering uh customer service [07:09] requests. That's that's a different [07:11] story there. We need to to figure out uh [07:14] what what are we looking at? How do we [07:16] uh detect if the agent is drifting in [07:19] some direction? If if we have some sort [07:22] of regression, right? um and due to the [07:24] nondeterministic nature it's uh [07:26] definitely harder to do that than with [07:29] uh traditional software. So that's one [07:32] thing. Then we have um essentially the [07:35] question of um and I think you touched [07:38] on this in the beginning uh of agent [07:41] generated code landing in production [07:43] right which um on one hand means it's [07:48] more code than ever uh and the second is [07:51] people are not that deep into that [07:55] generated code as they used to be before [07:57] with code they've written themselves. [07:59] Right? So uh we will see that uh [08:02] engineers will lose some muscle in terms [08:04] of really being able to uh quickly [08:07] understand code in a high stress station [08:10] like an incident. So um this is a an [08:13] interesting aspect in itself. It's also [08:15] interesting to see how uh topics like [08:17] maintenance will evolve over time as [08:20] agents get better, right? And how we [08:23] position ourselves in terms of this [08:26] agent generated code uh in the future. [08:29] Um and then there is last but not least [08:31] there's also the thing where you want to [08:33] maybe observe agents that produce code [08:36] right so not the output but actually the [08:39] the process of producing code and seeing [08:43] if uh the the code produced becomes [08:46] better if it uh becomes more efficient [08:49] if it's more reliable and so on. So [08:51] these kind of aspects also play a role [08:54] in terms of having essentially somewhat [08:57] under control what um models are doing [09:00] within your engineering organization [09:02] right [09:03] >> to give a concept like how big is [09:06] delivery hero in the engineering [09:07] organization approximately [09:10] >> yeah I mean we are uh globally [09:12] distributed in total we have over 4,000 [09:14] engineers and uh uh yeah [09:20] separated in in and tech hubs across the [09:22] world like Dubai uh Riad Barcelona [09:27] Singapore Se uh Buenos Iris Monte Vido [09:31] right so it's we have uh several of [09:34] these uh hubs out there and they're also [09:36] acting quite uh autonomously they have [09:39] uh their own way of working their own [09:41] tax stack um and that's also what what [09:44] makes this whole setup quite interesting [09:46] to solve right and to to support [09:50] >> how is your role as like VP of tech [09:53] foundations which I guess is like [09:55] platforms and developer enablement and [09:59] the enablement segment if we're talking [10:01] team topologies. [10:03] How are you are you experimenting with [10:07] agents? I know we can assume everybody's [10:10] writing code a lot with AI if not [10:14] exclusively at this point but more on [10:16] the observability side. Are you ex where [10:19] are you on that agentic journey which is [10:22] a journey it's not a light switch [10:24] >> it's a journey uh we're driving adoption [10:27] at the moment across the group uh but [10:29] particularly in our central teams um [10:32] both in terms of agentic engineering [10:34] toolings um such as aentic native idees [10:38] and so on but also in terms of uh [10:40] building an autonomous software delivery [10:43] agent or system that uh basically can [10:46] take a gyro ticket and that's um drive [10:49] it to conclusion alone. Right? So these [10:52] are the the things we're doing and in [10:54] addition we're also looking at um what [10:58] at least I call AI ops but I'm not sure [11:00] if that's the right uh branding here but [11:03] uh what I what I understand with that is [11:06] mainly using agents in operations for [11:09] incident response. That's something that [11:10] I think is hugely important to get right [11:14] early on because otherwise our engineers [11:16] will just burn out because they're uh [11:19] the time they spend will will be shifted [11:21] to incidents and that's everybody who's [11:24] been uh an on call engineer at some [11:27] point in their life understands that [11:29] this is not an awesome uh perspective [11:32] right you really want to keep that to a [11:34] minimum [11:35] >> and you want to be able to build that [11:37] trust with that I guess trust in a robot [11:40] maybe isn't the word. We we throw around [11:42] trust a lot. It doesn't sound great when [11:44] I say it that way. Like build trust with [11:45] that agent, but you need to [11:48] somehow prove that what the agent is [11:51] doing is what you want it to do, I [11:53] guess, and understand what it's trying [11:55] to do. [11:57] >> There needs to be definitely some [11:58] evidence there, right, before you give [12:01] uh agents the keys to your kingdom in [12:03] terms of your infrastructure. So I think [12:06] uh in the beginning it's more about [12:08] supporting with advice and supporting [12:11] with kind of looking at potential root [12:14] causes looking at potential actions that [12:16] engineers can do. But we leave uh the [12:18] decision to the on call engineer to to [12:21] actually um act right and uh to actually [12:26] make that infrastructure change if [12:28] that's needed. Um, I think the next step [12:32] would really be to give agents the [12:36] ability to uh execute runbooks in [12:40] well-defined scenarios, right? So let's [12:42] say you have an application that [12:45] regularly runs out of memory or whatever [12:47] you have that that particular scenario [12:49] that you know is happening from time to [12:50] time and you know exactly what you need [12:52] to do and it's the same that the on call [12:55] engineer typically does 100% of the time [12:58] then you can essentially allow the agent [13:00] to execute that like restarting the [13:02] application or whatnot right so I think [13:05] that would be the next step and in the [13:08] future couple years uh ahead um agents [13:12] will just become more competent, be able [13:15] to to execute more uh uh critical things [13:19] on their own. Um and we will allowed [13:23] based on the the evidence that we have [13:26] in terms of what they did before right [13:28] now. right now I think most people would [13:30] not be very uh willing to uh to trust an [13:36] agent to for example delete resources in [13:39] their infrastructure setup. Right? So [13:41] that's kind of dangerous. We hear these [13:43] stories from time to time in the news. [13:46] Um so you want to be careful with that. [13:49] You want to approach this slowly and in [13:52] wellont controlled um environments. [13:56] Okay, let's talk about some of the [13:58] different ways you can do that. So, we [13:59] have a poll [14:01] and just to read for those, you know, [14:04] working while they're listening in, [14:05] which is totally fine since we all do. [14:08] Has your organization adopted tracing [14:11] for your agents? Yes. No. Maybe. I'm not [14:14] sure the options. While that comes [14:16] through, Ryan, do you want to explain [14:17] what that is? What is tracing for agents [14:21] and which agents get tracing? [14:24] >> Yeah. I know I think it's interesting. [14:25] We talk a lot about how you can trust an [14:28] agent as you're implementing. Uh and as [14:30] you start on like step one, you put an [14:32] agent into place. Give it some tools. [14:35] What's it actually doing? Tracing is [14:37] going to be uh the mechanism to let you [14:39] see how it's reasoning, what tool calls [14:41] it's using, what it's not using that you [14:43] think it should, where there might be [14:45] latency or performance issues or even [14:48] security issues. You can do this through [14:50] a few different ways. Um, but for [14:53] example, if you use open telemetry, you [14:55] can trace every step, every request that [14:58] lets you play back exactly what happened [15:00] and you can derive lots of good metrics [15:02] and say, how often am I using a certain [15:04] MCP tool and does that correlate with [15:07] the auto remediation outcomes, for [15:09] example, you're looking for? [15:11] >> Do you not risk tracing burnout if [15:14] you're tracing everything? And I know [15:16] hotel is the standard, but doing it for [15:20] everything, do you not risk just too [15:22] much information for yourself or for the [15:25] agents? [15:26] >> Yeah, I think that's a really good [15:28] question. I think that's always the [15:29] nuance. Um, you know, from [15:32] Chronosphere's perspective, this is [15:33] something that we think is really [15:34] important. We have a control plane to [15:36] help you only keep the traces that are [15:38] valuable. Um, you can look at it [15:40] similarly to a quality control issue [15:43] where if you're just making children's [15:45] toys that have no risk and they're not [15:47] hazardous, uh, let's just say like a [15:48] wooden block, right? Nothing you can [15:50] choke on. Uh, you might not want to have [15:53] infinite nines of quality or [15:55] reliability. But if you're making, you [15:57] know, something like an airplane where [15:59] there's higher risk, you have to treat [16:01] it differently. I think tracing and [16:03] agent visibility is the same. If you [16:06] have a customer service agent, um, [16:08] again, just for your kind of side hobby [16:11] web app, you might want not want to [16:13] trace every request. You might want to [16:15] trace every tenth interaction or only [16:17] certain parts of an agent. Um, that [16:20] might be different in another use case [16:21] where you want to actually pay and and [16:23] have more visibility. [16:27] >> Uh, just to close out the poll, we had [16:30] great participation. Thank you to [16:31] everyone. So, has your organization [16:33] adopted tracing for your agents? Four% [16:37] or sorry, four I mean, sorry, 18% [16:41] because I can read 18% said yes. While [16:45] 30% weren't sure, that's totally fair. [16:47] This happens a lot. And then more than [16:49] half said no, they don't. And [16:54] that's okay. That's kind of part of what [16:56] we're doing here today. And we're [16:57] figuring out how each of those work. So [17:00] how do you go about setting up that [17:02] tracing? So I guess let's talk about [17:04] Google's setup for aentic AI because I [17:07] think we all kind of respectfully to [17:10] everyone in the room. We kind of assume [17:12] Google as itself is going to be a bit [17:14] further along than the majority of [17:16] companies in their experiments and [17:18] things or in their actual adopting at [17:22] scale. So yeah, please Jason tell us. [17:27] >> Yeah, it's a great question Jennifer. [17:29] you I'll touch a little bit on what Ryan [17:32] was talking about. If you think of, you [17:34] know, the trajectory of an agent being, [17:36] you know, what are the external things [17:37] that an agent does to get to an outcome [17:39] and then the reasoning chain being kind [17:41] of the internal thoughts, you know, it's [17:44] really about how do we capture all of [17:46] that information and then how do we [17:48] start to sift through that. So, you [17:51] know, for us, what does that tangibly [17:53] mean? You know, we do a lot of work with [17:54] evals. So for those um listening in eval [17:58] are just how do we you know determine [18:00] what an outcome should be for an agent [18:02] based on an intent or request and how do [18:05] we start to manage that because if you [18:08] think about you know agenda workflows an [18:10] end agent could do you know one step and [18:12] get to an outcome it could do five steps [18:15] and while we may care about some of the [18:17] you know the trajectory that the agent [18:18] has ultimately we want the requests and [18:20] the outcome to be you know colllocated [18:22] right in the same cohort of of of of [18:25] things. So that's really where we spend [18:28] you know a lot of time is actually on [18:30] you building the evals you know building [18:32] the data sets for that because it's not [18:34] easy you know as folks here you know [18:36] know well if you don't if you've never [18:39] had an agent before one of the hardest [18:40] things to do is to build trust in the [18:42] agent to deploy the agent and to do that [18:44] what you need is you need training data [18:46] or test data in order to make that [18:48] happen. Well, you one of the ways you [18:50] can do it is obviously with synthetic [18:51] data and you if you look at a lot of [18:53] large language models you know they both [18:56] pull on the corpus of you know public [18:58] information but a lot of them are [18:59] building their own synthetic data sets [19:01] now. So folks could do that you know [19:03] with tools obviously like big query and [19:05] even just Gemini out of the box you know [19:08] ask the question you know come up with [19:09] 500 ways to you know ask you why is my [19:13] bill overdue or something like that. Um [19:15] but the other way then is just you know [19:17] getting very verbose with you know [19:19] tracing and logging and you know I think [19:21] out of the box I would encourage [19:22] everyone you know do that and then start [19:25] to filter back um because you know all [19:28] that information is highly valuable and [19:30] if you think about your organization you [19:32] know the agents use cases like that is [19:35] actually kind of the what I think is [19:37] like the secret sauce going forward is [19:39] you know what is what is the agents [19:41] interaction with the customer you know [19:42] how is your data improving the [19:44] customer's interaction for it. So you [19:48] bring all those those things full [19:49] circle. Once we have those and we can [19:51] start to back off and say okay great now [19:52] we have enough training data where we [19:54] can continuously you know evaluate our [19:56] agent when it's running in production. [19:58] You know we know when trajectories you [20:00] know don't work out. So to Rodri's point [20:04] you know earlier like agents just tends [20:06] to make things 10 times better or worse. [20:09] You if you think of an agent that all of [20:10] a sudden decides it needs to call a [20:12] database but it calls the database 500 [20:14] times. Well, like you've put pressure on [20:15] downstream systems inadvertently with [20:18] that, but we need the reasoning chain to [20:20] understand why that happened. You know, [20:21] was it that the tool definition, you [20:23] know, needs improvement, you know, is [20:25] there something else that happened as a [20:27] part of that? But then we can start to [20:29] move in like actually build a gent [20:31] system on top of it. So that's where, [20:32] you know, candidly like a lot of our [20:33] time gets spent is, you know, how do we [20:35] actually, you know, build great evals [20:37] and then of course in in customer and [20:39] you compliance and regulatory safe ways [20:42] for that. So you know a lot of this [20:44] isn't actually technology right this is [20:46] business transformation it's thinking [20:48] about you know customer implications you [20:51] know um how we change the agreements [20:53] with customers and not for Google [20:55] specifically for for many organizations. [20:58] Um but that's it's the fun part but also [21:00] the part where we have to really think [21:01] about like what is valuable that we're [21:03] we're actually servicing the customer [21:04] with. When you talk about those 500 [21:07] calls, [21:09] are you setting up guardrails to prevent [21:11] that from happening or do you set a [21:14] limit on what an agent can do in a [21:16] certain time period? How do you prevent [21:18] that from happening while also wanting [21:21] to for the future understand the rails? [21:24] >> Yeah, you can. So there's obviously a [21:26] number of different ways like where you [21:27] can set you know limits on on different [21:29] things like this where I kind of come [21:30] back to like great software engineering [21:32] principles, right? If we're using APIs [21:34] like we can set you know limits and [21:36] backoffs on APIs such that we're [21:38] defending you know our services for that [21:40] like great software engineering does not [21:42] die in agentic engineering it actually [21:45] gets enhanced because we've already [21:46] defended the services that are around [21:48] the agent and you know sometimes like we [21:51] may want to have some back off or you [21:54] limit the retries within an agent [21:55] itself. Um some of those things you you [21:59] don't know what you don't know until you [22:00] deploy it and then you see what happens. [22:02] That's from you know for a lot of of [22:03] what I think about is do good software [22:06] engineering and then let the agents you [22:08] know in in smaller situations start to [22:11] pressure test you know different [22:12] environments for it because in many [22:14] cases you know you may have had 95 of [22:17] your services where you have you know [22:20] limits you have backoffs and all that [22:22] but what we want to find is actually [22:23] well what's the two services that like [22:25] didn't have that and sometimes you have [22:28] agents will find those things right [22:29] because they can execute work [22:31] significantly faster than we can. But [22:33] then what that means is for you [22:35] operations teams is you have to be on [22:37] anomaly detection like as the anomaly is [22:39] happening not after it happened you know [22:42] even 10 or 15 minutes down the line in [22:44] some cases. [22:47] >> Okay. [22:48] Absolutely. That is it feels like how [22:52] long did it take you to set that up and [22:54] at a scale where you have more you have [22:58] less limited employee resources. [23:02] You know, it's interesting like if you [23:04] go like take agent development code [23:06] which is one of our it's kind of our [23:08] flavor of how you build agents. Um [23:10] enabling telemetry is I think it's one [23:13] or two lines in Python. [23:15] >> Wow. [23:16] >> So you know a lot of these things and [23:17] and Ryan mentioned this you know even [23:19] with Langmith and and Langchain like [23:22] these frameworks have these things out [23:23] of the box. It's just not enabled by [23:25] default. [23:27] So I think for a lot of folks you know [23:28] it's just making sure that you actually [23:30] turn it on and then you start to look at [23:31] the results. And this is where like you [23:34] um Rodri kind of talked about this at [23:36] the beginning like agents like working [23:37] on agents I think is actually more [23:39] powerful in this model. [23:42] Um because what's happening with that [23:43] right is like you're going to get logs [23:44] and like tons of blogs of information [23:46] and we should budget that you know we're [23:49] going to actually get a lot of of [23:50] additional we're going to spend more on [23:53] things like logs and tracing and [23:54] telemetry going forward. And I think [23:56] that's just an expectation, you know, of [23:58] how we manage non-determinism. But then [24:00] once you use agents to sit through that [24:01] and you use agents to, you know, better [24:04] refine like our prompts and our [24:06] instructions and like the actual agent [24:08] design, like there is a virtuous cycle [24:10] in this, you know, that we can unlock [24:12] with one or two lines of code in many [24:14] cases for it. [24:16] >> Yeah. And I just want to bring my hot [24:17] tip. I love to mention if you've been [24:20] struggling to get observability budgets [24:22] or SRE budgets for years, say it's for [24:24] AI. You're more likely to get it. Same [24:27] with any migrations cloud. There's a lot [24:29] of things. Just say it's for AI. It's [24:32] fine. It'll make it stronger. So, how do [24:34] we actually reach that next stage of [24:37] agentic observability or agents [24:39] performing observability tasks like auto [24:42] remediation, root cause analysis? [24:45] uh what are the challenges to like [24:47] getting that adoption going? Like you [24:49] said, there's tracing or with OTL, it is [24:52] two magical lines of code. That's [24:54] amazing. But what are other things? And [24:56] I'm just going to open that up to the [24:58] floor of [25:01] that can help aid this process. How do [25:04] you decide which agents to deploy first [25:07] in your agentic observability? It's a [25:10] mouthful. [25:11] >> Yeah. Yeah. I'll I'll go very quickly [25:13] and then you know see the floor to um Ry [25:16] and Ryan. One of the things that I think [25:18] a lot about is like what are the [25:19] unwritten things that happen in an [25:22] incident response [25:24] and unwritten in this case right is like [25:26] what happens you in an incident response [25:27] room right you know what's the [25:29] conversation that happens you know [25:31] there's obviously the runbook that we [25:32] follow but what are the decisions and [25:34] what's the decision-m process that [25:36] people actually came to you know for how [25:38] to remediate a problem and with agents [25:42] you know we can't expect an agent to [25:44] outperform you know us or to you do the [25:47] same the same performance as this if we [25:48] don't give it the information and the [25:50] reasoning chains that people had in [25:52] order to you make something happen. So [25:55] you know I think a lot about what are [25:57] the undocumented things we need to [25:58] document into like the agents and and [26:01] give to the agent is you know additional [26:03] context or or context engineering so [26:05] managing the input window and then once [26:08] we have that you know then I think it is [26:10] you know hill climbing with the agents [26:12] to you know gameplay scenarios figure [26:14] out you know what worked what didn't and [26:16] then u moving for moving forward with [26:18] that so again similar themes around eval [26:22] and and tracing all that but you know [26:24] really you focusing on what is the [26:26] process, you know, what is the value and [26:28] then you keeping the human in the loop [26:30] like there is val there's a lot of value [26:32] in human creativity even in things like [26:34] instant response. Let's maximize that [26:37] and minimize the toil, you know, in [26:39] those things for it. But Rodrigue, Ryan [26:41] would love, you know, your perspectives [26:42] as well, [26:44] >> please. Where do you get started? What [26:46] do you need? [26:48] >> Ryan, [26:49] >> Ryan, you want to go first, Jay? [26:51] >> Yeah. Yeah, I I'll go quickly. I think [26:54] um you know, to Jason's point, you have [26:57] to understand the incident response [26:59] process because a lot of it is people in [27:01] process. Uh but then also kind of [27:04] mapping your agent into a harness so you [27:06] know what are the signals that it can [27:08] get the types of telemetry. Um but then [27:10] also what are the available mitigation [27:13] actions that can be taken. You don't [27:15] want to just necessarily let the agent [27:17] be so creative it can do anything. Um [27:21] put some guard rails in place around [27:23] that and then actually measure those [27:24] guard rails. This will help your [27:26] confidence. So, if you go a few [27:28] incidents in and you can see, hey, it's [27:30] it's trying to do things it shouldn't [27:32] and it's violating its own guard rails, [27:34] uh, you might need to make a lot of [27:36] adjustments. And if you get it to a [27:37] point where that's not happening as [27:39] much, you have some stable outcomes, you [27:42] kind of know you have some use cases [27:43] solved and then you can continue to um, [27:46] look for new use cases. You can you can [27:48] have a take on again looking at previous [27:50] incidents. Um, so I think that's to me [27:53] what's important. And then also if [27:55] you've been an S sur on an incident call [27:57] a lot of times you have one incident [27:59] commander um who's managing other [28:01] people. I think with agents a lot of [28:04] times everyone becomes a little bit of [28:06] an incident commander and you're [28:07] managing teams of agents uh for some [28:10] shared outcome. So I think there's also [28:11] just a mindset shift in how you're [28:13] approaching the work. So wait, are you [28:16] then nominating nominating selecting [28:18] maybe uh one or delegating one agent to [28:23] be an incident commander among agents or [28:27] is the incident commander [28:30] the human? [28:32] >> I think I think for now you still have [28:34] you still want human accountability for [28:36] sure. So I think the roles stay the [28:37] same, but previously an incident [28:39] commander might ask an S sur to go [28:41] analyze all these logs, come back with [28:44] their findings, plug that in, have a [28:46] developer working on a hot fix where now [28:48] you can have an incident commander who [28:50] might be able to accomplish that um with [28:52] agents directly or have those S surres [28:55] have a much wider footprint and have the [28:57] agents executing those individual tasks. [29:00] So a lot of the analysis work where you [29:02] have to like look through many logs [29:04] quickly um that's where you're thinking [29:06] how can I use these agents instead of [29:08] just my own capacity and brain power. [29:12] >> Right Rodri what are you all doing? What [29:16] are your first agents in observability [29:18] you may be deploying and how what are [29:20] you setting up for them? Yeah. So maybe [29:23] something we didn't really touch here, [29:25] but it's a very simple thing that [29:26] everybody can do quickly and is already [29:28] providing a lot of support is uh in most [29:31] incident response situations, you have [29:33] some sort of group chat or video [29:36] conference, right? Where the incident [29:38] commander is kind of trying to [29:40] understand the station, trying to [29:41] delegate tasks, trying to do something. [29:43] And you often have people kind of [29:46] jumping into that chat or into the call [29:48] at some later point in time, right? [29:50] because maybe they were not available [29:52] before and uh it's um usually a very [29:56] tedious task both for the people who [29:58] join as well as the instant commander to [30:00] keep everybody up to date and what's [30:02] going on. So a simple really simple [30:06] trick is using an agent to easily [30:09] summarize what has happened so far. Um [30:13] very we did that uh at deliver hero. [30:15] It's it's extremely helpful and uh [30:18] similar in um in video chats uh usually [30:22] you already have this kind of [30:24] functionality. So for example in in [30:26] Google meet uh you can easily just get a [30:30] recap of what was going on. So you don't [30:32] have to as an incident commander spend [30:35] explaining to people what has happened [30:37] every 5 minutes, right? You can [30:39] completely hand that off. Uh so that's [30:42] good. You could even go so far and have [30:44] executive communication done by agents [30:46] in a certain interval so that you have [30:50] um freed up uh your team to actually [30:53] look for the the root cause itself and [30:56] to fix the problem. Right? So that [30:57] that's the first step very very simple [30:59] step. Uh the next step in our case is [31:02] we're we've observed several different [31:06] players in the industry uh who want to [31:09] provide uh something like an SRE agent [31:13] or assistant in observability and so on. [31:16] So you see that from from observability [31:19] companies uh specifically I think they [31:22] have great training data for this and [31:25] can build awesome products. You see that [31:27] from hyperscalers um like AWS and Google [31:31] coming up with their own versions which [31:33] is great if you're on single cloud uh on [31:36] on their platform right uh you have [31:38] startups coming from other angles such [31:41] as uh training on postmortems and [31:44] runbooks and so on trying to to build [31:46] something that is uh knowledgeable about [31:51] what happened at your company with your [31:53] tax stack. So um it's it's still a phase [31:57] where I believe most industry players [31:59] are kind of experimenting and trying to [32:01] figure out what is the best solution [32:03] here. And if you're not one of those [32:05] companies, I think it's better to just [32:09] try out what is out there from those [32:12] players and see uh if [32:15] uh if it fits, right? I would not go and [32:18] try to reinvent the wheel here. Um, in [32:21] terms of uh agent observability or SRE [32:25] agents, [32:26] >> I really like that those use cases [32:30] don't take away the job security of [32:33] developers and especially observability [32:35] or S sur engineers. I like the idea that [32:38] they're focused on the boring, which I [32:40] think is always a great use case for AI [32:43] because it not that it it's important [32:46] the communication landscape and [32:47] updating, but it can be tedious and time [32:51] consuming and also [32:54] you can learn so much by capturing that [32:56] information for future and help improve [32:59] those processes and that communication [33:01] process, not just the technology of it. [33:04] So I think those are great examples. So [33:07] you can't improve what you don't [33:08] measure. [33:10] I guess Rodrigue if you want to start [33:12] what metrics are you running on your [33:14] again I I we haven't decided as a [33:17] company as a industry if we're going to [33:20] call agentic ops if we're going to call [33:21] observability agents or something in [33:24] between. So how do you measure that? [33:27] >> So currently we're measuring mostly from [33:30] the business perspective right. So we [33:32] are defining uh business KPIs that are [33:35] important for our agents and then we're [33:37] looking at that and see any deviation [33:39] there. We do not yet have uh let's say [33:43] specialized [33:45] uh agent observability in place. It's [33:47] something that we're still looking at [33:50] and uh we did not yet decide on on what [33:53] to use here. [33:55] >> Okay. Yeah, absolutely. What other [33:58] metrics do we have? either Ryan or [34:00] Jason, what are you measuring? [34:05] >> You know, a lot of what I look at in [34:06] agents kind of it still comes back to, [34:08] you know, I think eval primarily. [34:12] >> So, there's there's a number of other [34:13] things we can measure, right, in terms [34:15] of, you know, things that the agent is [34:17] trying to do. So, did the agent [34:18] accidentally, you know, jailbreak? So, [34:20] did it do something it shouldn't have, [34:22] you know, ride into a system where it [34:24] should have only had read? you know [34:26] those are metrics right that we can [34:28] control with things like gateways you [34:30] know and and other kind of security [34:32] measures. Um, I think I still come back [34:35] to, you know, what is the outcome that [34:37] we want? How many times did we actually, [34:39] you know, get to the outcome that that [34:41] the agent desire was supposed to get to? [34:44] And how do we think about, you know, how [34:45] many tokens did it take to get there? [34:47] Like I know token economics right now is [34:49] probably, you know, front and center for [34:52] almost anyone who's working with agents [34:54] or working to deploy agents. So, you [34:56] know, how many tokens did it take to get [34:58] to the destination that we wanted? you [35:00] know what's our token per second you [35:02] know rate or token per minute rate in [35:04] the system [35:05] you know the more we kind of track that [35:07] right I think the more we learn about [35:09] you knowic behavior so if it took 50,000 [35:12] tokens to get there you know could we [35:14] ask an agent to you know spend some [35:15] tokens to improve that um and you know [35:18] are we trying to drive you know what's [35:20] what's kind of the reason you know that [35:21] we think we can drive this down lower um [35:24] the other thing I think about a lot is [35:26] um context engineering so what's [35:28] actually in the agent's context window, [35:30] >> you know, and if we think about [35:32] measuring that, you know, there's [35:34] obviously like, you know, hey, what's [35:35] the percentage of of the spill, you [35:37] know, the time that the resulting window [35:39] takes, [35:41] but a lot of it is just, you know, I [35:42] think probably more what I'll say is [35:45] like traditional, you know, data [35:46] engineering metrics, although it is it [35:49] is application telemetry at the end of [35:51] the day. We just have to add I think a [35:54] few more pieces in terms of you know [35:55] rounding out with the things that we're [35:57] looking at with the agent and then how [35:58] we manage it. So like tokens and you [36:01] know how you manage back offs or like [36:03] model substitutions as a result of of [36:05] the outcomes that you're trying to [36:06] drive. [36:08] >> Very good. Ryan, what else do you have? [36:10] What other metrics? There's always more. [36:13] >> Yeah, this one's interesting. Um I I [36:15] think again it's going to vary a lot [36:16] based on the use case, individual [36:18] customer, and what you're looking for, [36:20] but I think there's a few layers. Uh the [36:22] others touched on heavily what's the [36:24] business outcome. So you need some way [36:26] of measuring what is this agent existing [36:28] to do and how often is it actually [36:31] accomplishing that independently versus [36:33] how often is it escalating for human [36:35] approval versus totally failing. U and [36:38] so I think that's the main thing that [36:39] you're always trying to focus on. And [36:41] then secondary to that, you're looking [36:43] at like what's inside of the agent. What [36:46] model am I using for a use case versus [36:49] another model? And how do I know uh for [36:52] example if I'm using too powerful of a [36:54] model and it's spending too many tokens [36:56] and I can actually use a weaker model [36:59] that's going to be more affordable and [37:00] get the same results or is there a [37:02] threshold where I use a different model [37:05] um and you know suddenly my business [37:08] outcomes are harmed and so yes I might [37:10] save money but that's not a worthwhile [37:12] trade-off. uh then I think there's one [37:14] level deeper that's very technical [37:16] performance-based this is where uh for [37:19] observability folks it's very [37:20] comfortable in the past we had like red [37:23] metrics golden signals request errors [37:26] duration uh this is more on the [37:28] inference performance but now it's not [37:30] just latency or number of requests it [37:33] might be latency plus time to first [37:35] token then you might also see uh not [37:38] just the number of requests but the [37:40] total amount of tokens in that time [37:42] frame [37:43] uh because each request isn't the same [37:45] anymore. Uh there might be vastly [37:47] different amounts of tokens that come [37:48] through. And then lastly, the eval to [37:50] say, hey, we don't just have technical [37:52] errors with 500s anymore. We might have [37:55] a totally fine uh performance from a [37:58] service perspective, but it might be [37:59] hallucinating or giving a biased answer. [38:02] And we might want to treat that as an [38:04] error scenario as well. [38:07] >> Absolutely. And then just not to forget [38:12] that we need to follow like the basic [38:16] DORA meanantime to recovery. There's [38:18] also like meantime to innocent. What [38:20] isn't at fault when something goes wrong [38:23] is kind of good to weed things out. And [38:26] then I think there's always value in [38:28] qualitative metrics as well. This is a [38:30] community that traditionally has a lot [38:33] of burnout because traditionally we [38:35] haven't we never have enough staff for [38:39] operations and observability and SR etc. [38:42] So understanding if they feel like [38:47] things are getting better if they feel [38:50] even if it there's the metrics but [38:52] there's also the qualitative of how they [38:54] are feeling with these new tools. [38:57] >> Yeah. And yeah, so let's jump into that [39:00] a bit. Like let's just briefly as we [39:02] wind down talk about like with the work [39:05] of the SRE changes like what is that job [39:07] f profile look like where this human [39:11] being feels useful still and purposeful. [39:17] >> Uh I'll jump in on this one. I I think [39:19] that one thing we do with all of our um [39:23] AI guided troubleshooting and Agentic [39:25] products is when you go through the [39:27] workflow, every response and [39:29] interaction, you have the button to say [39:31] like this is helpful or it's not [39:32] helpful. So as you're using the agents [39:35] and using the products, you're [39:36] continually giving feedback. that lets [39:38] the agents obviously get better and [39:40] improve, but it also gives you some idea [39:42] of how staff is responding to that if [39:45] it's actually helping their life or if [39:47] you're trying to put a new workflow when [39:49] they're already trying to learn 10 [39:50] others. I think that's always a critical [39:52] step. Um, from a technology standpoint, [39:55] but then also from an adoption [39:56] standpoint, you need to be doing those [39:58] same type of checks. Um, and then also [40:01] making sure that there might be other [40:03] similar workflows that aren't connected [40:04] yet. The more you can connect to these, [40:07] the more an individual can worry less [40:09] about managing all these different [40:10] technology stacks and more about just [40:12] getting, you know, more things [40:13] accomplished. [40:16] >> What else? How is it changing at the [40:18] place that invented the term SR to my [40:20] knowledge, Jason? [40:23] >> Yeah. One of the things that I'll I'll [40:25] piggy back off Ryan for, and I think [40:27] this is true of all roles, is it's [40:29] forcing engineers to become better [40:33] experts in the business. [40:36] And I think that's important you know [40:37] even in S sur because as as we kind of [40:40] talk about the layer cake of of problems [40:43] that sres are facing well [40:46] >> take like the ground truth problem you [40:48] know is the agent you know does [40:50] everything look fine but like the [40:52] customer is mad at the agent [40:54] >> you if an S sur has to go troubleshoot [40:56] that the S sur kind of like needs to [40:59] have a a pretty good foundational [41:01] understanding of the business that the [41:02] agent is is operating. [41:04] in the context of it. So, you know, one [41:07] of the examples I I actually just wrote [41:09] about this on Tuesday. Um, [41:12] you have an agent, but one agent [41:14] delivers, you know, the customer's [41:15] information about their last month's [41:17] bill, and the other one delivers, you [41:19] know, information about a bill that's [41:20] two months old. Well, you know, one a [41:23] one customer is now unhappy because they [41:25] asked for last month and they didn't get [41:27] it. You have to know what is the [41:29] customer support flow for things like [41:31] bills and like what does the customer [41:33] expect and then start to work back [41:34] because you know is it that the agent [41:37] was actually doing something that it [41:38] shouldn't have is it that it didn't have [41:40] access to data upstream you know did was [41:43] the bill never processed. So in that [41:46] case like that's not an agent like [41:47] that's not a problem of the actual agent [41:49] that is an upstream system problem but [41:52] the lights were all green. M [41:54] >> so you have to be I I think with all [41:57] rules now you really have to be [42:00] not embedded in the business but you [42:02] have be more consultative with the [42:03] business to understand okay what are the [42:05] problem scenarios that happen and then [42:07] how do you actually identify and and [42:10] root cause the thing that made the agent [42:12] do something that you didn't expect and [42:14] that comes with understanding you both [42:16] the application but also you know the [42:18] business processes on top of it [42:21] >> okay I'm wondering how do you empower [42:23] your SRUs and other enablement teams. [42:27] >> Yeah. So, first of all, I agree with uh [42:31] Jason about the topic that SRRES will or [42:34] in general on call engineers will need [42:36] to have solid knowledge about how the [42:38] business works. Uh that's also because [42:41] um the scope of the things that single [42:44] engineers are producing will increase by [42:47] a lot. Right? So the on call uh [42:50] responders will very likely not know the [42:54] code uh that produces the problem, [42:56] right? They're a little bit detached [42:58] from the code, which means they need to [43:00] to work on a higher level. They need to [43:02] know how systems behave, dependencies [43:05] between systems. They need to know how [43:07] the flows, customer flows and user flows [43:09] behave, right? To figure out what's uh [43:11] what's going on. So I think it's a um [43:14] still very likely a very stressful [43:17] situation because you have this [43:18] incident, you need to stop it. Um but [43:20] you have to think differently than [43:21] before. You're you're certainly not [43:24] going into some code repository to [43:26] figure out where the bug is. I think [43:28] that is something that uh we will leave [43:31] behind over time. Um [43:35] I also believe that uh we will need a [43:39] different um organizational structure [43:42] than what we had before right so in most [43:44] companies like delivery hero and others [43:47] typically you have teams centered around [43:49] certain product capabilities right let's [43:52] say you have a team that is centered [43:54] around the checkout process [43:57] um and in on call you will have some [44:00] sort of shift and there will be one [44:01] engineer who knows the checkout right? [44:04] Because um they will be able to handle [44:06] the incident. Now this kind of structure [44:10] will likely not be something that we'll [44:13] see in the future because we will not [44:15] have a single team just looking after a [44:18] checkout. This single team will look [44:20] after five different uh capabilities [44:23] maybe or maybe 10. So they're scope [44:25] broadens which means that the likelihood [44:28] of an expert of this particular team [44:31] being in an incident call is very small [44:34] right so uh that means the on call setup [44:39] needs to differ. We can uh make it a [44:42] little bit more generic, right? We can [44:44] have a wider scope for the on call team, [44:47] but we can only do that because agents [44:50] will be able to provide the necessary [44:52] technical context uh to the responder, [44:55] right? So the responders will look at [44:57] the um the business flows. They will [45:00] understand the big picture of how things [45:02] uh work together and with the help of an [45:04] agent will be able to resolve incidents [45:07] in code bases that they do not have uh [45:10] deep knowledge of. [45:12] >> That's very interesting getting away [45:13] from that idea. You built it, you run [45:15] it, you get it to production. That's [45:18] which [45:19] I guess it's another level abstraction [45:21] just like platform engineering was [45:23] before. That's very interesting. We [45:25] could talk about I'm very fascinated on [45:27] how team teams will change size but we [45:30] are alas out of time. Uh if people want [45:33] to continue the conversation there's [45:35] always the lead dev slack community. [45:37] It's a wonderful community of [45:38] engineering managers. [45:40] And if you want to get a copy of cloud [45:45] observability and action uh Manning book [45:48] uh just go ahead and email events@ [45:51] chronosphere.io [45:53] io and just say in the title something [45:56] like cloud observability and action [45:58] manning book and get a free hard copy of [46:01] that book. I am so grateful to each of [46:04] you. Thank you Jason. Thank you [46:06] Rodrigue. Thank you Ryan and thank you [46:09] listeners. Good luck on your journey. As [46:11] you can see it is a journey. It is not a [46:14] light switch. We will not be a company [46:16] that is agentically observable or [46:20] observably agentic in the near future. [46:24] But it's a process. So collaborate, make [46:28] it safe to experiment, [46:31] and have fun with it. It's exciting [46:33] times. Good luck. Y'all have a great [46:35] day. Thank you.  📄 **Full text:** ![[Full Document Contents/Articles/Do We Need AI Agents to Fix Our AI Agents.md|Do We Need AI Agents to Fix Our AI Agents? by YouTube]] ## Highlights - >> It's a It's a good question. To me, um, observability hasn't really changed other than the fact that we now have to manage non-determinism in addition to the rest of the stack. And you know what I mean by that? all the things that we've learned about and we know, you know, from software engineering and and S sur and and all those stacks. It's not that it's different, right? It's not that we're trying to do all these like new things and we have like new platforms and all this, but what does mean is that we have ([View Highlight](https://read.readwise.io/read/01kv3ty6ess9s52k5f7f8xd8bx)) - Note: Agent observability isn't new engineering — "observability hasn't really changed other than the fact that we now have to manage non-determinism in addition to the rest of the stack." Apply known SRE principles to non-deterministic systems. - Tags: atom/literature/h1, operations, observability - the same quality level as before it would just linearly scale the number of incidents. So we have to have something that helps us there and uh that's where I'm very optimistic about uh agents that can help with triaging root cause analyzis even executing runbooks in uh predefined scenarios for example. So that's one aspect. Then there is the aspect of observing agents that are embedded in consumer or ([View Highlight](https://read.readwise.io/read/01kv3tpffk3y31kzx6ctyr25bd)) - Tags: 👻 ai highlighted - uh agents the keys to your kingdom in terms of your infrastructure. So I think uh in the beginning it's more about supporting with advice and supporting with kind of looking at potential root causes looking at potential actions that engineers can do. But we leave uh the decision to the on call engineer to to actually um act right and uh to actually make that infrastructure change if that's needed. Um, I think the next step ([View Highlight](https://read.readwise.io/read/01kv3tyfxymeg4pghdzmq2a50m)) - Note: Build agent trust in phases: don't give agents "the keys to your kingdom" at first — start with advice + root-cause/action suggestions, leave the decision to the on-call engineer, then expand autonomy. - Tags: implementation-strategy, atom/literature/h2, agents - you start on like step one, you put an agent into place. Give it some tools. What's it actually doing? Tracing is going to be uh the mechanism to let you see how it's reasoning, what tool calls it's using, what it's not using that you think it should, where there might be latency or performance issues or even security issues. You can do this through a few different ways. Um, but for example, if you use open telemetry, you can trace every step, every request that lets you play back exactly what happened ([View Highlight](https://read.readwise.io/read/01kv3tysshxe27rz7fwrgsr3x4)) - Note: Tracing is the mechanism to debug non-deterministic agents — see how it reasons, which tool calls it makes/skips, latency, and security issues. OpenTelemetry lets you replay every step. - Tags: atom/literature/h2, observability, debugging - if you think about your organization you know the agents use cases like that is actually kind of the what I think is like the secret sauce going forward is you know what is what is the agents interaction with the customer you know how is your data improving the customer's interaction for it. So you bring all those those things full circle. Once we have those and we can start to back off and say okay great now we have enough training data where we can continuously you know evaluate our agent when it's running in production. You know we know when trajectories you ([View Highlight](https://read.readwise.io/read/01kv3tpfk1hgrr8mwfq2x24je3)) - Tags: 👻 ai highlighted - once you use agents to sit through that and you use agents to, you know, better refine like our prompts and our instructions and like the actual agent design, like there is a virtuous cycle in this, you know, that we can unlock with one or two lines of code in many cases for it. >> Yeah. And I just want to bring my hot tip. I love to mention if you've been struggling to get observability budgets or SRE budgets for years, say it's for AI. You're more likely to get it. Same with any migrations cloud. There's a lot of things. Just say it's for AI. It's ([View Highlight](https://read.readwise.io/read/01kv3tpfrcz2d74mt6yjnza05v)) - Tags: 👻 ai highlighted - fine. It'll make it stronger. So, how do we actually reach that next stage of agentic observability or agents performing observability tasks like auto remediation, root cause analysis? uh what are the challenges to like getting that adoption going? Like you said, there's tracing or with OTL, it is two magical lines of code. That's amazing. But what are other things? And I'm just going to open that up to the floor of ([View Highlight](https://read.readwise.io/read/01kv3tpftwy4d507d1syhn0pzx)) - Tags: 👻 ai highlighted - that can help aid this process. How do you decide which agents to deploy first in your agentic observability? It's a mouthful. >> Yeah. Yeah. I'll I'll go very quickly and then you know see the floor to um Ry and Ryan. One of the things that I think a lot about is like what are the unwritten things that happen in an incident response and unwritten in this case right is like what happens you in an incident response room right you know what's the conversation that happens you know ([View Highlight](https://read.readwise.io/read/01kv3tpg0951rrjs946fq2a00k)) - Tags: 👻 ai highlighted - So a lot of the analysis work where you have to like look through many logs quickly um that's where you're thinking how can I use these agents instead of just my own capacity and brain power. >> Right Rodri what are you all doing? What are your first agents in observability you may be deploying and how what are you setting up for them? Yeah. So maybe something we didn't really touch here, but it's a very simple thing that everybody can do quickly and is already providing a lot of support is uh in most ([View Highlight](https://read.readwise.io/read/01kv3tpg2s0qtfs6wgq0txyz4z)) - Tags: 👻 ai highlighted - process, not just the technology of it. So I think those are great examples. So you can't improve what you don't measure. I guess Rodrigue if you want to start what metrics are you running on your again I I we haven't decided as a company as a industry if we're going to call agentic ops if we're going to call observability agents or something in between. So how do you measure that? >> So currently we're measuring mostly from ([View Highlight](https://read.readwise.io/read/01kv3tpg6kkhv19j0hwrheh7jt)) - Tags: 👻 ai highlighted