CLI and MCP Basics for AI Agents-2 - CourtneyR.dev's 2nd Brain

CLI and MCP Basics for AI Agents ![rw-book-cover](https://readwise-assets.s3.amazonaws.com/static/images/article1.be68295a7e40.png) 🔗 **Source:** [CLI and MCP Basics for AI Agents](https://www.youtube.com/watch?v=ZR6U4TN8-Cw#transcript) > [!note]+ Document Notes > tags: [ai-agents, cli-tools, model-context-protocol, observability, developer-tools, ai-observability] > author: [MISSING] > source: YouTube > reference: https://www.youtube.com/watch?v=ZR6U4TN8-Cw > date created: {{date:YYYY-MM-DD}} {{time:HH:mm}} > > # 🏷️ Title: CLI and MCP Basics for AI Agents (The Context Window #4) >  ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ## 📝 Summary - Grafana team discusses two approaches for AI agents to interact with observability tools: Model Context Protocol (MCP) servers vs CLI tools - MCP provides standardized tool definitions with authentication, while CLIs offer progressive discovery and token efficiency - Both approaches achieve similar outcomes but with different trade-offs in token usage, discoverability, and implementation complexity - Grafana offers both GCX (CLI) and MCP servers for cloud integration, with evaluations showing comparable performance - The choice between approaches depends on use case, environment constraints, and personal preferences ## 📖 Main Content The Grafana team explores two competing approaches for enabling AI agents to interact with observability platforms: Model Context Protocol (MCP) servers and Command Line Interface (CLI) tools optimized for agents. ### Model Context Protocol (MCP) MCP is a standardization protocol introduced by Anthropic about 18 months ago that defines how AI agents interact with external tools. It solves the problem of every application needing to define tools separately for each agent platform. Key benefits: - **Standardized tool definitions**: Self-describing tools with parameters and authentication built into the protocol - **Cross-platform compatibility**: Works across different transports (local stdio or remote HTTP) - **Explicit tool discovery**: All available tools are defined upfront with clear descriptions Main limitations: - **Token consumption**: Tool definitions consume context window space (though mitigated by deferred tool search) - **Fixed tool set**: Less flexible for esoteric or rarely-used functionality ### CLI Approach with GCX Grafana's GCX (Grafana Cloud CLI) takes a different approach, optimizing specifically for agentic usage rather than human interaction. Key features: - **Progressive discovery**: Agents can explore available commands dynamically - **Agent-aware output**: Detects agent mode and provides JSON output vs. human-readable tables - **Efficient data handling**: Spills large payloads to disk rather than flooding context - **Semantic exit codes**: Provides actionable error information for agent decision-making - **Field selection**: Allows agents to request specific fields to minimize context usage ### Skills as Training Wheels Both approaches use "skills" - markdown files defining workflows that combine multiple commands or tools to accomplish higher-level tasks. These serve as training data until AI models have sufficient knowledge about the tools built into their training sets. ## 🗒️ Notes #### ⏱️ [00:00:00](https://www.youtube.com/watch?v=ZR6U4TN8-Cw&t=0) Introduction and Team Locations - Team across 4 countries: Portugal, USA (Seattle), Netherlands, and another US city - Nicole (Dev Advocate), Tiffany (AI/observability), Ben (ML engineer), Ward (Product Manager), David (GCX builder) #### ⏱️ [00:03:12](https://www.youtube.com/watch?v=ZR6U4TN8-Cw&t=192) Recent AI News - GPT 4.8 release with improvements over 4.7, especially cost efficiency and tool calling - Anthropic IPO filing and $980B valuation - Dreaming feature for managed agents (memory consolidation) - OpenTelemetry CNCF graduation - Google IO: Gemini Spark, Anti-gravity 2 IDE - PineCone pivot from RAG to "pre-compiled knowledge" #### ⏱️ [00:08:30](https://www.youtube.com/watch?v=ZR6U4TN8-Cw&t=510) What is Grafana Assistant - Chat-based agent for observability questions - Supports PromQL writing, dashboard creation, root cause analysis, SLO building - Available through multiple interfaces: chat, GCX, MCP - Now supports OSS and Enterprise, not just Cloud #### ⏱️ [00:12:15](https://www.youtube.com/watch?v=ZR6U4TN8-Cw&t=735) MCP Explained - Standardizes tool definitions between applications and agents - Prevents N×M problem where every app defines tools for every agent - Provides authentication, cross-transport support - Examples: GitHub MCP server with create issue, pull request tools #### ⏱️ [00:18:45](https://www.youtube.com/watch?v=ZR6U4TN8-Cw&t=1125) CLI in AI Context - Optimized for agentic usage, not human interaction - Progressive disclosure reduces token usage - GCX can call Grafana Assistant for specialized observability insights - Skills provide workflow templates combining multiple commands #### ⏱️ [00:25:30](https://www.youtube.com/watch?v=ZR6U4TN8-Cw&t=1530) Skills Explanation - Markdown files defining workflows - Bridge between low-level commands and high-level actions - Training wheels until LLMs have sufficient knowledge in training data - Similar to Claude Code skills, installed to standard directories #### ⏱️ [00:32:00](https://www.youtube.com/watch?v=ZR6U4TN8-Cw&t=1920) Ben's MCP Advocacy - Self-describing tools and parameters - Built-in authentication - Works across environments (local/remote) - Good tool discoverability upfront - Deferred tool search mitigates token consumption issues #### ⏱️ [00:38:15](https://www.youtube.com/watch?v=ZR6U4TN8-Cw&t=2295) GCX Features - Agent-native design with mode detection - Hints about command complexity and output size - File spilling for large payloads - Field selection for efficient data retrieval - Semantic exit codes and JSON errors - Covers entire Grafana Cloud API surface #### ⏱️ [00:45:20](https://www.youtube.com/watch?v=ZR6U4TN8-Cw&t=2720) Live Demo Comparison - Side-by-side Claude Code instances: MCP vs GCX - Task: Create SLO for assistant metrics grouped by cluster - GCX completed faster with $0.04 cost - MCP required API discovery, cost $0.07 - Both achieved the goal successfully #### ⏱️ [00:52:30](https://www.youtube.com/watch?v=ZR6U4TN8-Cw&t=3150) Decision Framework - Remote environments (Claude.ai, ChatGPT) require MCP - Local IDEs can use either approach - Consider binary installation preferences - Esoteric tasks may favor CLI flexibility - Performance evaluation through ObsBench benchmark ## 📚 References - [Grafana MCP Server GitHub](https://github.com/grafana/grafana-mcp-server) - 3000+ stars - [GCX CLI Documentation](https://grafana.com/docs/grafana-cloud/cli/) - [Architecture of Forgetting blog post](https://contexthorizon.com) - AI memory consolidation - [ObsBench](https://github.com/grafana/obsbench) - Observability benchmark for AI agents ## 🗣️ Paraphrase The Grafana team debates two architectural approaches for AI agent integration. MCP provides a standardized protocol where tools are explicitly defined upfront, offering consistency and built-in authentication but consuming context window space. CLI approaches like GCX use progressive discovery, allowing agents to explore capabilities dynamically while optimizing for token efficiency through techniques like file spilling and field selection. Both approaches achieve similar outcomes, with the choice depending on environmental constraints (local vs remote), preference for explicit vs discovered tooling, and specific use case requirements. The team emphasizes that the interface matters less than delivering high-quality observability insights efficiently. ## 📰 Newsletter The AI tooling landscape continues evolving rapidly! Grafana's latest Context Window episode dives deep into two fascinating approaches for connecting AI agents to observability platforms. Whether you're team MCP (standardized, explicit tool definitions) or team CLI (progressive discovery, token-efficient), both paths lead to powerful AI-assisted observability workflows. The live demo showing identical SLO creation tasks with minimal cost differences proves that good engineering execution matters more than architectural philosophy. Plus, their new GCX CLI and MCP server are both open source - check out the GitHub repos and give them a star! The ObsBench evaluation framework is particularly interesting for anyone building AI tooling. ## ✅ Action Items #### DevRel - Create comparison guide: MCP vs CLI decision tree for different environments - Develop workshops showcasing both GCX and MCP integration patterns - Build demo scripts for common observability tasks using both approaches - Measure and publish performance benchmarks across different AI platforms #### Community - Contribute to open source GCX and MCP server repositories - Share skills/workflow templates for common observability patterns - Create evaluation frameworks for AI tool effectiveness - Document best practices for agent-optimized CLI design #### WordPress - Apply progressive disclosure patterns to WP-CLI for AI agent optimization - Explore MCP server for WordPress management tasks - Consider skills-based training materials for AI-assisted WordPress workflows ## 🧪 Hypotheses - **Token efficiency hypothesis**: CLI with progressive discovery uses fewer tokens than upfront MCP tool loading for complex workflows - **Performance convergence hypothesis**: As models improve and context windows expand, performance differences between MCP and CLI approaches will diminish - **Training data effect hypothesis**: CLIs for established tools (GitHub, AWS) perform better than new tools due to training data inclusion - **Discovery vs definition hypothesis**: Explicit tool definition (MCP) provides better guidance for smaller/cheaper models than progressive discovery - **Skills obsolescence hypothesis**: As tools gain training data presence, workflow skills become less necessary for agent effectiveness ### 📊 Methodology - **A/B testing**: Run identical tasks through MCP and CLI approaches, measuring tokens, latency, and success rates - **ObsBench evaluation**: Use Grafana's observability benchmark across different models and tool configurations - **Longitudinal analysis**: Track performance changes as models are retrained with more CLI documentation - **Multi-environment testing**: Compare approaches across local IDEs, cloud platforms, and edge cases ### ❓ Questions - How do token costs scale with workflow complexity for each approach? - Which method provides better error recovery and debugging experiences? - What's the learning curve difference for developers adopting each approach? - How do authentication and security models compare between MCP and CLI patterns? ### 🔮 Future Research - **Hybrid approaches**: Combining MCP tool definitions with CLI progressive discovery - **Dynamic tool generation**: Auto-generating MCP tools from CLI help documentation - **Context-aware tool selection**: Choosing optimal approach based on task complexity and environment - **Agent specialization**: Training models specifically for observability tasks ### 💡 Implications - **Development strategy**: Organizations need multi-modal AI integration strategies rather than single approaches - **Standardization efforts**: Industry may converge on hybrid standards combining benefits of both approaches - **Training focus**: AI model training should emphasize tool discovery patterns alongside explicit definitions - **Tooling evolution**: Traditional CLIs will increasingly optimize for machine consumption alongside human usage ### 🧠 Learning Styles Assessment Reading level: Technical (Grade 14-16) - contains domain-specific terminology and complex architectural concepts. #### 👀 Visual learners - Present: Architecture diagrams showing MCP vs CLI data flows, token usage comparisons - Improve: Add sequence diagrams for tool discovery processes, visual decision trees #### 🔊 Aural learners - Present: Video format with detailed verbal explanations and live demonstrations - Improve: Audio-only version highlighting key decision points and trade-offs #### 🗣️ Verbal learners - Present: Clear definitions of MCP, CLI, skills, and progressive discovery - Improve: Add glossary of technical terms, analogies to familiar concepts #### 🧩 Kinesthetic learners - Present: Live coding demo, hands-on comparison of both approaches - Improve: Step-by-step tutorials, interactive decision flowcharts #### 🧠 Logical learners - Present: Decision frameworks, performance benchmarks, systematic comparisons - Improve: Formal evaluation criteria, quantitative analysis of trade-offs #### 🧑‍🤝‍🧑 Social learners - Present: Team discussion format, multiple perspectives represented - Improve: Community forums for sharing experiences, peer review of approaches #### 🧘‍♀️ Solitary learners - Present: Detailed technical documentation, individual exploration opportunities - Improve: Self-assessment checklists, personal experimentation guides #### 🌿 Naturalistic learners - Present: Evolutionary perspective on tool standardization, ecosystem thinking - Improve: Analogies to natural selection in technology adoption ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Clear headings, descriptive timestamps for video content - Improve: Add ARIA labels for complex diagrams, detailed alt text for code examples #### Keyboard navigation - Present: Standard video controls, accessible transcript navigation - Improve: Keyboard shortcuts for jumping between topics, skip links for repetitive content #### Contrast and readability - Present: Standard video player with good contrast - Improve: High contrast mode for code examples, larger text options for technical diagrams #### Text resizing and reflow - Present: Responsive transcript layout - Improve: Ensure technical diagrams remain readable at 200% zoom #### Media and captions - Present: Full transcript provided - Improve: Add closed captions with technical term spelling, audio descriptions for demos ### 🌍 DEIB Assessment #### Cultural breadth - Present: Global team representation (4 countries) - Improve: Include examples from diverse organizational contexts, non-Western technical patterns #### Inclusive language - Present: Professional, jargon-defined technical discussion - Improve: Provide plain language summaries, expand acronym definitions #### Neurodiversity support - Present: Multiple information formats (audio, text, visual demos) - Improve: Add structured summaries, consistent navigation patterns #### Representation - Present: Mixed gender technical team, various experience levels - Improve: Include diverse use cases, accessibility considerations in tool design ### ✅ Logical Validity #### Strong reasoning - **Comparative analysis**: Well-structured comparison of approaches with concrete examples - **Evidence-based conclusions**: Uses live demonstrations and benchmark data to support claims - **Systems thinking**: Considers broader ecosystem implications and trade-offs #### Fallacies - **False dichotomy**: Framing as "MCP vs CLI" when hybrid approaches exist - **Recency bias**: Overweighting latest model improvements without long-term analysis - **Selection bias**: Demo task may favor one approach over another #### Tone of voice Professional and collaborative, with good-natured friendly competition. Technical depth balanced with accessibility. ### 🛡️ DARVO Analysis No DARVO patterns detected. Discussion maintains professional focus on technical trade-offs without personal attacks or victim-blaming dynamics. ## 🎯 Conclusion Both MCP and CLI approaches offer viable paths for AI agent integration with observability tools. MCP excels in standardization and explicit tool definition, while CLIs provide flexibility and token efficiency through progressive discovery. The choice depends on environmental constraints, use case complexity, and developer preferences. As the ecosystem matures, hybrid approaches combining benefits from both paradigms will likely emerge. Organizations should focus on delivering high-quality insights efficiently rather than betting exclusively on a single architectural pattern. --- ## Full Transcript [00:04] And we're live. [00:06] I think this is the most number of [00:08] people that we've had at the same time [00:09] on this episode before. [00:11] >> I think so. And are we not in five [00:14] different countries right now? [00:17] >> Oh, me and Ben are in the same [00:22] >> You messed it up. [00:23] >> Sorry. American cities if that helps. [00:26] I'm in Portugal, [00:29] >> USA, Seattle, [00:32] >> and the Netherlands. [00:36] >> Well, maybe we should get started. Let's [00:38] introduce ourselves. I'm Nicole [00:41] Vanderhovven. I'm a dev advocate at [00:43] Graphana and I'm here with Tiffany. [00:48] >> I was like, am I in the same direction [00:49] as you like on my screen? But I guess [00:51] so. Uh [00:52] >> I think so. Yeah. [00:54] Um yeah, so I'm a developer advocate at [00:57] Graphana for uh tempo slow tracing, [01:01] pyroscope for profiling and AI things. [01:05] Um I'm going to go downwards. Hello Ben, [01:08] >> which I think is me. Yeah. Uh hello [01:10] everyone. I'm uh I'm a a software [01:13] engineer at Far. I've been here for five [01:16] years now, so practically forever. Um, [01:19] I've been on the machine learning team [01:21] and now work on lots of different AI [01:24] based products including uh the MCP [01:27] server which we'll talk about a little [01:29] bit. Yeah. Uh, and several other bits [01:31] and pieces. Um, so yeah, let's go that [01:35] way toward [01:37] >> I [laughter] think the other way around. [01:39] Doesn't matter. You called me out. So [01:41] I'll I'll do my introduction. [01:43] >> Yeah. So uh so I've been with Grafana [01:46] Labs for almost seven years now and I [01:49] had various functions but uh currently [01:52] I'm a product manager focused on Grafana [01:55] cloud and specifically nowadays on GCX. [01:59] >> Yeah, I guess I'll go last. So I'm I'm [02:01] David. I've been here about the same [02:03] time as Ben. So just over five years [02:05] I've been building GCX with Ward. Um and [02:09] most recently been focusing on trying to [02:11] understand how GCX performs. um with his [02:14] skills against other other tools. [02:20] >> Awesome. So, I I guess today we're going [02:22] to be talking about MCP versus CLI, what [02:27] all those letters mean and which one Ben [02:30] thinks is appropriate for all [02:32] situations. [02:34] >> The [laughter] big the big versus word [02:36] came out. I thought we were thought it [02:38] were all [02:39] >> Yeah, we're picking sides. [laughter] [02:40] Except only Ben ended up doing that. [02:46] >> I wonder which one. H [02:49] >> I know. [02:49] >> I guess some announcements first. Um did [02:52] you all know if you're here then you're [02:54] probably interested in what we're doing [02:55] in AI? We're actually doing this maybe [02:59] brave, maybe silly thing where we're um [03:02] taking I'm taking a bit of an excerpt [03:05] from every team weekly meeting and I'm [03:08] putting it up on YouTube and it's mostly [03:11] the demos part, but if you haven't seen [03:14] it, it is here. It's called Inside the [03:17] AI weekly and uh I'm trying to get one [03:20] up every Monday. Uh, but let's go [03:23] through some like some AI news from the [03:27] last month. I think one of the biggest [03:29] things is that 4.8 dropped. I think it [03:31] was just like was that last week or [03:33] something? [03:34] >> It was very recent. [03:36] >> I think it was last week indeed. [03:38] >> Yeah, it's much better than the previous [03:41] one because it's a higher number. [03:44] [laughter] [03:44] >> Yeah, it's May [03:47] >> the world changing revolutionary. [03:49] >> But I do think it's better. Uh, I think [03:51] it seems more like an evolution than a [03:54] revolution. I think they they fixed some [03:57] um cost inefficiencies with 4.7. At [04:01] least that's that's what I read reading [04:03] between the lines. It seemed just a [04:05] little bit better on on everything, but [04:07] especially like a bit more efficient in [04:10] in calling tools and stuff. [04:13] >> Yeah. Yeah. And I also also think they [04:14] wanted to get get a rid of the 4.7 bet [04:17] wrap like uh there were around it. Yeah. [04:21] >> Yeah. I think there's an association [04:22] with like the new tokenizer in 4.7 that [04:25] they were like, "Okay, we'll quickly [04:26] quickly get paper over that and get a [04:28] new one. I hope that doesn't have that [04:29] association." That's the cynic in me. I [04:32] don't know. [04:34] >> And another uh there's a couple things [04:36] from Anthropic. Anthropic is going to [04:38] IPO now. [04:41] Yeah, they did a not so secret filing I [04:44] think with the sack. [04:47] >> That's always scary when something when [04:51] something that big like one of the [04:52] frontier models is potentially up for [04:55] grabs and might you know it all of these [04:58] quality issues like who's who's going to [05:00] control one of the best frontier models [05:03] in the world. That's that's kind of [05:05] scary [laughter] [05:07] but we'll see what happens there. And [05:09] also if you see their latest funding [05:10] rounds, oh sorry I didn't want to talk [05:12] over you but if you also saw their [05:14] latest funding rounds like those are big [05:16] numbers. It's amazing like uh 980 [05:20] billion or something valuation. It's [05:22] incredible. [05:22] >> Oh wow. [05:24] >> Yeah. [05:25] [laughter and snorts] [05:26] >> That's a lot of cappuccinos. [05:28] >> Yeah. I'm just like thinking of [snorts] [05:30] how much money that the employees there [05:33] are potentially getting. [05:34] >> Wow. [05:36] >> So many tokens. [laughter] [05:37] tokens as well. [05:38] >> Yeah, drinks are definitely on them if [05:40] we [05:40] >> they get paid in tokens. [05:42] >> Yes. [laughter] [05:45] >> Uh they also introduced dreaming for [05:47] managed agents. So it's it's interesting [05:50] because I think so open claw was [05:53] actually the first to come out with this [05:54] dreaming feature. It's a memory [05:55] consolidation feature. But I think and [05:58] they didn't say this directly but I [06:00] think they got it from when the clawed [06:02] source code was leaked. There were [06:04] already some hints to dreaming there. [06:06] Open Claw actually implemented it first [06:08] and now this is Anthropic's actual [06:11] implementation. I've been playing around [06:13] heavily with with Open Claw um mostly [06:17] using Opus. So I'm interested in seeing [06:20] what Anthropic had in mind. [06:23] >> Wow. And what does it do when it dreams? [06:26] I didn't follow that story. [06:28] >> Yeah, I it's actually a really tidy [06:30] thing. Oh, I I wrote a blog post just on [06:34] this. [06:34] >> Shameless plug. There we go. [06:36] >> Yeah. Uh it's called the architecture of [06:39] forgetting. It's on context horizon, our [06:42] AI based um kind of blog. And it is [06:46] basically it's a three-stage memory [06:49] process. It's basically how uh how AI [06:52] agents remember [06:54] remember things and then also forget [06:57] things on purpose. [07:00] Interesting. That is interesting. [07:03] >> It's got there more and more agents or [07:06] at least AI systems are operating as a [07:10] human brain in that sense like we're [07:12] we're dreaming. They are now also [07:14] dreaming. What's next? Yeah, open clause [07:18] implementation is particularly [07:19] interesting because you can look and see [07:22] what it dreamt in the light and deep and [07:26] REM phases and you can see what it chose [07:30] to carry over into long-term memory. [07:34] >> I wish I had that choice. [07:37] [laughter] [07:39] Um, also open telemetry graduated from [07:43] CNCF only officially on May 21 [07:45] apparently. And although this isn't [07:48] strictly this isn't just an AI thing, [07:50] the whole project really is leading hard [07:53] into Gen AI and um we are our our own AI [07:57] observability app follows Genai semantic [08:00] conventions. So I thought that it was [08:02] relevant. Uh, and then Google IO also [08:06] happened and dropped a bunch of models. [08:09] The most interesting one was Gemini [08:11] Spark, which is, it sounds to me like [08:14] their open claw, like a continuously [08:16] running um, a personal agent that's only [08:20] available on their new, brand new Google [08:23] AI Ultra plans, which I can't get access [08:26] to because I'm on a workspace. It's not, [08:28] my Google account is not personal, which [08:30] kind of sucks. I would like to access [08:32] that. [08:34] Yeah. And then I was also saying that um [08:36] it was like anti-gravity 2 or something [08:38] like that. Now they have some like IDE [08:39] and whatnot. [08:42] Yeah, it it's like it's like VS Code. [08:44] It's very cursor like I think. I haven't [08:48] played around so much with it, but [08:49] that's also included in the AI ultra [08:51] plans. So it's like use of anti-gravity [08:54] and Gemini and YouTube Premium, which [08:57] isn't AI, and Google Drive. [09:01] That's how Google's gonna going to try [09:03] to win this by bundling in all of the [09:05] Google things. [09:09] Uh, another thing is that Pine Cone, [09:12] who's the one that came, well, not maybe [09:15] not came up with, but like popularized [09:18] the idea of rag retrieval augmented [09:21] generation [09:23] just released this blog post saying [09:25] basically we don't think that rag is the [09:27] way to go now. instead pre-ompiled [09:30] knowledge is and they're calling it pine [09:33] cone nexus. I'm not entirely sure how [09:36] that works yet, but I just thought it [09:38] was interesting to see whenever you see [09:40] like a flip-flop like this. Um, they [09:43] popularized it, so they, you know, they [09:45] own this space and they're kind of [09:47] saying, well, these are the areas where [09:49] it's not so good. [09:51] >> Interesting. It looks like from the [09:53] article that I just scanned, it looks [09:55] very similar to what we do in Grafana [09:58] cloud around uh the knowledge graph and [10:01] AI memories or the infrastructure [10:04] memories that we have in the graphana [10:06] assistant. So I guess we're on the right [10:09] track there. [10:11] >> Sounds like it. Well, since you [10:13] mentioned it, Ward, why don't you why [10:14] don't we get started and you can tell us [10:16] what Graphana Assistant is. [10:18] >> What is Graphana Assistant? Uh so Cfana [10:21] assistant is our u kind of chatbased [10:24] agent for all your observability [10:27] questions and uh the assistant is kind [10:30] of an engine and you can basically [10:32] access it through all kinds of different [10:34] means different doors if you will and [10:37] there's two doors that we're going to [10:38] talk about today I think the GCX door [10:40] and the MCP door but um most of our [10:43] customers are probably now familiar with [10:46] using the assistant through kind of a [10:48] chat interface like little bit similar [10:50] to chat GPT and uh they cannot ask it [10:53] all kinds of questions. So think about [10:56] simple questions around um like help me [10:59] write [11:00] PromQL and the query language for our [11:03] metrics database. uh it can ask you can [11:06] ask to create a dashboard but you can [11:08] also do more advanced things like do [11:10] full root cause analysis or maybe help [11:13] building SLOs's and much much more but [11:15] that's essentially what the agent of the [11:17] what the assistant is and uh what we're [11:20] uh yeah making available through all [11:22] kinds of various ways like MCP and GCX [11:25] and APIs and the web. [11:29] >> Yeah. And then also like if you're you [11:31] if you've used assistant like I don't [11:34] know before fonicon basically you were [11:36] doing everything related to stuff you're [11:37] running in cloud for like if you're [11:39] directly using assistant um but as of [11:42] graphonicon you can also use graphon [11:44] assistant with graphon oss and also [11:46] enterprise and I'm putting a blog in [11:48] here that came out with that if you want [11:50] to see more about that. [11:55] Yeah, exactly. Big news of our Graphana [11:59] conference that we just had. [12:03] >> Yeah. And also at Graphfana, I noticed [12:05] that we we just announced so many things [12:07] all at the same time. And um you know [12:10] now assistant has tools and there's a [12:14] lot of people who are confused like if [12:16] they want to use their AI tools, their [12:20] AI agents uh to to do things in [12:23] Graphana. There are now many different [12:25] ways like they can use assistant [12:27] everywhere it seems and and then um they [12:30] can use those tools they can use the [12:33] graphana MCP there's now graphana cloud [12:36] MCP there's the CLI there's GCX so like [12:40] let's let's figure all of this out [12:43] because otherwise it's super confusing [12:46] um what does it mean for assistant to [12:48] use tools and what can you do with it [12:52] >> yeah good question I think like tools [12:55] are are the thing that makes AI into [12:58] agents. That's the way I think of it. [12:59] And it gives them the power to interact [13:02] with the real world in in some way [13:04] rather than just being like text in and [13:05] text out. Uh so it allows the assistant [13:08] to do all of the clever things it can do [13:10] such as looking for dashboards, creating [13:13] dashboards. Basically, it allows them to [13:16] allows the agents to yeah perform useful [13:20] tasks and to do so more accurately as [13:22] well by by having real knowledge bases [13:24] they can query or uh being able to hit [13:26] APIs in like predefined ways [13:33] and I think I think that there's two [13:36] about a year and a half ago there was [13:38] this this kind of big thing released by [13:41] anthropic which was the model context [13:43] protocol CP. Um, and this was like a [13:46] standardization of the way that uh the [13:49] tools are defined and and the idea was [13:53] to solve the problem of you have lots of [13:56] different applications that want to talk [13:58] to lots of different agents and and have [14:01] want have have lots of different tools [14:03] defined. And how do you do that without [14:07] each like application such as cursor or [14:10] VS code or whatever your IDE is defining [14:13] all of those tools separately like each [14:15] one of them would have to go and define [14:17] their own tools to interact with GitHub [14:19] and to interact with Graphfana and and [14:22] likewise [14:23] GitHub would have to if they wanted [14:25] their their tools to be available and [14:26] everything they would have to contribute [14:27] them towards each of those tools. Uh so [14:30] MCP is is kind of a a standardized [14:32] protocol defining those tools amongst [14:34] other things. Uh it's not just tools. Um [14:37] but it basically allows you to tell a [14:40] model uh an an LLM an agent. Um these [14:44] are some tools I have and and every [14:46] client application knows how to talk to [14:48] them. Um so some examples are the GitHub [14:51] MCP server which has tools such as like [14:54] create issue or create pull request uh [14:57] fetch some code um and any one of the [15:01] various IDE can connect that up um and [15:05] the user just logs in quite simply [15:07] ideally quite simply and gets all of [15:09] those tools for free. Um there are a [15:12] myriad trade-offs with this like it's [15:14] it's great for lots of reasons. It's [15:17] terrible for some others like you have. [15:20] It's very partisan. Um, but we'll talk [15:22] about some of that hopefully soon. [15:25] >> Yeah. I guess for people that maybe are [15:27] on like the observability side less so [15:30] than AI side, you can kind of think [15:31] about it in a way of like how open [15:33] telemetry is where instead of having to [15:34] figure out something for like every [15:36] single uh backend that you have, whether [15:38] it's different whether it's for like [15:39] logging or metrics or even if it's [15:42] different um types of like logging [15:44] systems. So whether you're just [15:47] different ones that you might be using, [15:48] if you're using tracing, if you're like, [15:49] hey, I have Jäger or I have Tempo and [15:51] trying to figure all of that out instead [15:53] of having to set up everything [15:54] separately for every single one of [15:55] those. It's kind of similar, different [15:57] but similar. [15:59] >> Right. [16:02] >> Okay. So it's kind of like a MCP is kind [16:05] of like an API schema of sorts. It just [16:08] kind of defines a standard format for [16:11] what how to how to define these tools. [16:15] >> Yeah. Yeah. And and that's that in [16:17] itself is really powerful. Like even if [16:18] you just had that plus the like [16:21] authentication side. So for now at last [16:25] we finally have a way of saying this is [16:27] how a client should authenticate with a [16:30] server and how what these are the RPCs [16:33] or the functions of the tools that you [16:35] can call and they have standardized [16:36] descriptions like that alone is really [16:38] valuable. I think that's not really been [16:40] something that the industry has had uh [16:42] for a long time. like there's been [16:44] various different proposed things on top [16:46] of JSON APIs like rest and those sort of [16:50] things but this is yeah it's it's quite [16:53] good for that from that point of view [16:56] >> okay so what is the CLI then well as the [17:00] CLI is not actually an an AI thing it's [17:02] a command line interface right but uh to [17:05] anyone who wants to answer what is the [17:07] CLI in an AI context [17:10] >> right right so the CLI by indeed [17:14] everybody that is kind of familiar with [17:15] the command line on Unix, Linux etc. But [17:19] it's one of the oldest concepts in [17:21] computing in that sense. Um like people [17:25] are very uh well aware of that. But uh [17:28] the thing in the AI age is that it's [17:31] actually optimized not so much for human [17:33] knowledge even though like in the case [17:35] of GCX which which is the graphana cloud [17:37] CLI uh like it's fine if humans use it [17:40] but it's actually optimized actually for [17:42] agentic usage. So we're assuming that [17:46] you're not calling the CLI directly but [17:48] you're actually calling it through cloud [17:50] code cursor Gemini uh codeex all those [17:53] types of agentic uh frameworks and um [17:57] that u we need to make sure that like [18:01] the the the agent [18:05] like cursor cloud code can actually [18:07] figure out uh based on the flags and the [18:11] commands that are within in uh GCX or [18:14] within the CLI uh they're able to figure [18:16] out which ones to call and uh then the [18:19] CLI will do actually do the heavy [18:21] lifting and uh and the idea is uh [18:24] because it's agent optimized that we u [18:26] need to make sure that it's doing that [18:28] very efficient and when you're talking [18:30] about efficiency you're talking about [18:32] low token usage because context and the [18:35] context window uh we need to make sure [18:38] that there's only relevant data in the [18:40] context window and that we're not [18:41] filling that up with like a big dump. Uh [18:44] let's say you're going to go to Grafana [18:46] Cloud and you're use GCX, you're going [18:48] to ask it, hey, give me all the [18:49] dashboards. That can be maybe thousands [18:51] of dashboards. You don't want to put [18:53] that in the context. So, it's really [18:54] important that the CLI is indeed giving [18:58] a small result back. And then um one of [19:01] the things that we are doing is that uh [19:03] GCX this CLI can actually also call the [19:05] assistant. So it has uh even access to [19:09] that um the specialized agents that we [19:12] have on Carfana cloud for observability. [19:16] [snorts] [19:17] And then like when I hear people talking [19:18] about these things like with MCP, you [19:20] just hear people talking about MCP when [19:22] people are talking about CLIs and the AI [19:25] sense. I hear people talking about [19:27] skills. Could someone kind of explain [19:29] what that is? [19:32] Well, I guess you know if you consider [19:34] the kind of commands that the CLI [19:36] provide as the kind of building blocks [19:38] of functionality, right? You can decide [19:40] to manually call a command to do a [19:42] particular action. So, you know, get get [19:44] a dashboard or list your dashboards or [19:47] push an alert rule or something. The [19:49] skills uh kind of are a way of defining [19:52] uh a workflow to maybe weave a couple of [19:55] those skills together uh those command [19:57] uh calls together to perform a [19:59] particular action. Right? So it's kind [20:02] of like bridging bridging a gap to from [20:04] from like a set of kind of unopinionated [20:07] command line uh calls uh towards [20:10] something more like what MCP tool to [20:12] tools do which is to provide like an [20:14] opinionated action for a given use case. [20:17] Um so they're really they're really [20:20] flexible things that they're essentially [20:21] just a set of markdown markdown files [20:23] with some references to maybe some other [20:25] markdown files or bash bash scripts or [20:27] something like that. Uh but for our use [20:28] case, we're trying to use them to kind [20:30] of build build CLI commands, build GCX [20:33] commands into into actionable workflows. [20:37] >> Yeah. And you can you can maybe also see [20:38] that in the case of GCX that skills are [20:41] a little bit also training wheels [20:43] because our bet is or our hypothesis is [20:47] that the more information there is [20:50] around GCX the usage use cases that [20:53] actually LLM will be trained on GCX. So [20:57] if you ask an LLM to use the GitHub CLI [21:00] or the AWS CLI, it knows exactly what to [21:03] do because there's so much information, [21:04] so much blog post around it. So [21:07] hopefully and of course we need to prove [21:09] that hopefully at some point in time [21:11] there will be so much information around [21:13] GCX that we don't use skills that we we [21:16] don't need skills that much because it's [21:17] able it's basically part of the training [21:19] set of the LLMs. But uh that will [21:22] probably take a little bit of time uh [21:24] because like the there's always kind of [21:26] a lag between the information that LLMs [21:30] are trained on and the information [21:31] that's out there on the interwebs [21:34] >> and [clears throat] is the way that uh [21:37] skills are described with like GCX is [21:39] that the same way that like because I'm [21:41] just like looking at an article about [21:43] like cloud skills like is is skills used [21:45] in the same way or is it different [21:46] because like say if someone talks about [21:48] like services whether you're talking [21:49] about like Docker or Kubernetes or a [21:51] bunch of other stuff. Services doesn't [21:52] all mean the same thing. [21:55] >> Yeah, in our use case, uh skills means [21:58] the same for as if you were kind of [22:00] installing one of the kind of claw code [22:02] skills or something like this, uh the [22:04] method of delivery is slightly [22:05] different. Um but the result is that you [22:07] get the same files in the same place. So [22:10] what it you know if you're downloading [22:12] some skills from what say one of the [22:14] claude code skills or some skills from [22:15] skills.sh SH or some other kind of [22:17] registry or you know place where skills [22:19] are published. Um you are essentially [22:22] you're downloading a set of markdown [22:24] files to a specific directory in your on [22:26] your computer that uh the kind of the [22:28] agents know to look in and I think each [22:30] vendor has a has a different one. So [22:32] there's like you know Claude looks at a [22:33] predefined path and maybe Codex looks at [22:36] a different one. Um but normally these [22:38] are like sim linked from some kind of [22:41] standard place or something. So there [22:44] are some skills in bundled within GCX [22:46] and there's a specific command that you [22:47] can run that essentially writes those [22:50] like uh skills files in in the standard [22:52] kind of format to the expected place on [22:56] your on your computer so that they are [22:58] available for for ages to use. Yeah. [23:01] >> Yeah. Maybe good to mention is that [23:04] we're talking about CLI MCP and maybe [23:07] you never know right in a few weeks [23:09] times there there is something that is [23:11] even more hip in the AI world you know [23:13] how it goes right it's kind of like [23:15] fashion nowadays [23:17] um and from our perspective from Grafana [23:21] cloud's perspective from Cfana Lab's [23:23] perspective like the interface is kind [23:25] of the fashion even though we try to [23:27] make it very high quality very useful uh [23:31] very valuable for our customers. But I [23:33] think the real value is the insight that [23:35] it provides the observability insights. [23:37] So one of the things that we're focusing [23:39] on when we're building graphana cloud is [23:41] not only like u providing just raw data [23:46] like results of promql queries to their [23:48] customers but actually giving high [23:50] quality observability insights. So the [23:52] these are like fetted insights. They're [23:54] prioritized. They're actionable and we [23:57] try to deliver them in a fast and cheap [23:59] way. So how can we give an answer with [24:01] the least amount of tokens in a way that [24:04] is really valuable for the agent or the [24:06] human that is uh accessing those [24:09] insights. So that's really where we put [24:12] a lot of time and effort in that we [24:14] design products like the assistant and [24:16] our APIs around that so that um [24:20] basically the doors if you will the the [24:23] CLI MCP API doors are able to leverage [24:26] that and then yeah if there is something [24:28] next if there's a new fashion in town [24:31] then we can adopt that and we can make [24:33] our insights accessible through that [24:35] interface. [24:37] We do have a question here actually from [24:40] one of our graphana champions. [24:42] He he says, "Hi all. Can we use a custom [24:45] MCP server from Graphana AI assistant? [24:48] Do we have any sample configs or docs or [24:51] blog post?" So I'm posting a link to [24:55] docs here for that. Um but maybe maybe [24:58] Ben would you like to take this? [25:01] >> Yeah. Yeah, sure. Uh yes is answer uh [25:04] with a couple of caveats. I think the [25:05] main caveat being it has to be a hosted [25:08] custom MCP server. So it has to have a [25:12] use the streamable HTTP MCP transport. [25:15] Um Oh, nice. We've got a demo. [25:17] >> Yeah, I had it up so I figured Yeah, [25:19] here's one that I [laughter] added and [25:21] then you can go and click add here and [25:23] >> Yeah. Yeah, exactly. So as long as you [25:25] support uh as long as it supports remote [25:27] MCP which means like the streamable HTTP [25:31] uh transport has to be hosted somewhere [25:33] it can be hosted uh ideally publicly is [25:36] the easiest way but there are ways that [25:38] you can make it work with the private um [25:41] private network connection um and you [25:44] can it supports authentication via [25:46] headers or oorthth. So as long as you [25:49] you kind of use pretty standard or [25:52] mechanisms that should work fine. [25:55] So, Ben, while you're at it, maybe maybe [25:57] you could explain to us why you're on [25:59] team MCP. [laughter] [26:02] >> Um, okay. Yeah. Well, [26:06] I I so the re the reason I'm into the [26:08] MCP side is because, um, I initially [26:11] wrote the MCP server, the Grafana MCP [26:14] server, uh, in December 2024. [26:18] Yeah. So, a little while ago now. And um [26:21] that was kind of just as MCP was [26:23] becoming a thing and [clears throat] [26:25] people weren't really as aware of tools [26:28] and how tools could work with agents and [26:30] MCP was like a nice way of getting that. [26:33] Um and you get a lot you used to get a [26:35] lot for free back then because you [26:37] didn't really have as many frameworks as [26:38] you do now. Um, so yeah, wrote the wrote [26:41] the initial MTP server as a hackathon [26:43] project and um, it did quite well and [26:46] then we decided to open source it and [26:48] since then it's it's kind of taken off [26:49] quite nicely. We have I think more than [26:52] 3,000 GitHub stars now which is really [26:55] exciting and and we we recently released [26:58] a cloud version of the MTP server which [27:02] um, allows you to connect seamlessly to [27:05] like a Grafana cloud instance. Um, and [27:09] things I really like about MCP, [27:12] um, despite despite all the hate is is [27:14] kind of the, um, [27:17] uh, is the explicitness, I guess, like [27:20] you, you you do have to define a lot of [27:22] things like tools up front and you you [27:24] you [27:26] have to be a little bit more explicit [27:28] there, but you get what you get in [27:29] exchange for that is self-describing [27:32] like tools and parameters. You get [27:35] authentication built into the protocol [27:37] which is really nice. Um you can use it [27:40] across lots of different transports. So [27:42] you use it either locally using the [27:44] standard IO1 or remotely with streamable [27:48] HTTP. And that means that it kind of [27:50] works [27:52] everywhere roughly as long as you have [27:54] the right MTP servers. Um you can use it [27:58] whether you're in for example the [27:59] Grafana assistant that can connect to [28:01] any remote MCP server. Um, it doesn't [28:04] matter what language it was developed in [28:06] or where it's hosted. As long as it has [28:09] like a publicly accessible URL, you can [28:11] connect to it and use it. [28:14] Um, not that I don't like CLIs at all. [28:17] I'm a big fan of [laughter] [28:19] >> That's what I heard. [28:21] Yeah, I did see like like uh like what [28:24] people had against MCP uh in the past [28:28] was that MCPS define the tools and the [28:31] tool definition gets loaded in the [28:33] context window and when you only have [28:35] like 200k uh context window that fills [28:38] up pretty quickly and so uh because I [28:41] think like I did some calculations [28:43] around 76 tools that's probably like 20k [28:45] token uh tokens uh that will cost you. [28:49] So that's about like 10% of your token [28:51] window um is already being used before [28:54] you even start the conversation. [28:55] However, what you see nowadays that a [28:57] lot of platforms they're implementing [28:59] tool search. So that means that they're [29:01] not loading it. They're only loading [29:02] those tools on demand. So that um you're [29:05] not filling up the token window or the [29:08] context window that much. So that's one [29:10] of the the main pet peeves of a lot of [29:12] folks that were using MCP in the past [29:14] and that's getting less and less true [29:16] depending on like the agent that you're [29:19] using to access the MCP. [29:21] >> Yeah, exactly. I think I think the [29:23] downside there [29:25] that that is that is the main great [29:27] mitigation that's appeared in the last [29:29] year is the deferred tool search kind of [29:31] side of things. Um the the main [29:33] limitation is that the MCP host uh [29:37] called code cursor has to implement [29:38] deferred tool search itself but that is [29:40] increasingly common. So yeah uh agree [29:43] what like that was that was my main uh [29:46] complaint until quite recently as well. [29:49] Yeah, a lot of people could be like, [29:50] "Oh, I'm already use I'm using cursor [29:52] cloud or like other insert other um tool [29:56] and then or not I guess not tool but [29:58] like insert another um I guess what's [30:01] even the correct word that describes all [30:03] of those things. My brain is not working [30:05] at the moment but um [30:07] >> harnesses or Yeah. [30:09] >> Yeah. But yeah, basically people are [30:12] already using those for like their own [30:14] uh stuff that they're doing. They're [30:16] like, "Oh, I know how to use this thing. [30:17] um let me just add the MCP server that I [30:20] need for whether it's Graphana or GitHub [30:22] or something else and they didn't have [30:24] to like really in start with some other [30:28] entirely different thing. [30:29] >> Yeah. Yeah. Well, some some a lot of [30:32] times especially in technology like good [30:34] enough is sometimes just fine, right? [30:36] Like it doesn't need to be perfect. [30:37] There might be some downsides but for a [30:40] lot of folks that's perfectly [30:41] acceptable. Maybe from kind of a pure [30:43] purist point of view, right? people are [30:45] like me like it uses a lot of tokens. uh [30:48] but yeah nowadays maybe depending on the [30:51] way that you're interacting with the [30:52] agent that doesn't really matter and and [30:55] also the agents for example I use cloud [30:57] code we now have 1 million token windows [31:00] uh context windows and uh so yeah it's [31:04] it's less of a yeah a big problem I [31:06] think but uh yeah like if you look at [31:10] CLI CLI was the like one of the reasons [31:12] why a lot of folks like CLI is that uh [31:15] it with a CLI you're aiming for kind of [31:18] a near zero baseline where you have like [31:20] progressive disclosure. [31:22] it has some commands where you can then [31:24] find indeed okay what are the commands [31:26] that I need and that is basically [31:29] discovered by the agent and so you could [31:33] say that a CLI has way less token usage [31:37] but again that like those those [31:39] differences become less and less uh [31:41] relevant I think um or well not relevant [31:46] but they're they're like closing in on [31:48] each other [31:50] >> I think one area Where I I like MCP [31:54] better is that um I feel like the tool [31:56] discoverability is better on MCP. Um on [32:00] CLI, you know, you kind of have to know [32:03] those commands and yeah, you can go [32:05] through like d-help or whatever. Uh but [32:08] you still have to go through each one [32:10] and then each command has different um [32:13] parameters and stuff. Um sometimes you [32:15] just want to know what's possible before [32:18] you can figure out what you want to do [32:20] next. [32:22] Yeah. And each one of those like cycles [32:23] to discover [32:25] uh the arguments and the subcomands and [32:27] everything takes latency and tokens and [32:30] you know that's going to be done again [32:32] every conversation that you have. It's [32:33] not something that's up front. So that [32:35] that can that can be slightly that can [32:38] wear in after a while. I think it's [32:40] quite I do wonder if it's quite fun to [32:41] see it the first time you use a new CLI. [32:44] You're like oh my god it's discovering [32:45] everything so good. How impressive. So, [32:49] and that gives you that feeling of like [32:50] things happening and you know it's [32:52] making progress. Um, which is nice, I [32:55] have to admit. [32:57] >> I think it really [32:58] >> What do you think, Dave? [32:59] >> Yeah, I think you see a lot of [33:00] variability as well with um with the [33:03] models and things like this. So for [33:05] example, you know, very anecdotally, but [33:07] we've we've run kind of uh agents trying [33:10] to use GCX to interact with with Grafana [33:12] for various uh kind of use cases and uh [33:15] you know, with them with and without the [33:17] GCX skills installed to kind of give it [33:19] any hints or not, right? And um you we [33:22] can see a lot of variability in whether [33:24] the models know whether to whe that GCX [33:28] exists or whether to bother trying to [33:30] look for um the command structure before [33:33] calling anything or just to kind of give [33:34] it a crack and see see if it works. Um [33:37] but you'll see it quite often kind of do [33:39] this kind of stuttering thing where it [33:41] gives something a go, fails, uh then [33:43] decides to read the help docs for the [33:45] given command uh to try and look for the [33:48] guidance on how to do it. It's not so [33:50] well some of the models aren't so great [33:51] at like trying to discover exactly what [33:53] to do before they try. They just try [33:56] first and then fail and then discover. [33:58] So you know um media skills are trying [34:00] to bridge that gap but you can see MCP [34:03] provides a lot more kind of guidance [34:04] from the start for these for these [34:06] models maybe especially the kind of the [34:08] cheaper smaller ones. [34:10] Yeah, I actually had something similar, [34:11] not with the CLI, but like the other day [34:13] I was I gave um Claude a blog and was [34:16] asking questions and then after and I [34:17] was reading I was like this is weird. [34:18] I'm like did you actually read the blog? [34:20] They're like no let me go pull that now. [34:22] And I'm like oh [laughter] [34:26] >> yeah, exactly. Yeah. [34:28] >> Yeah. And um what I think is also [34:30] important for us as like creators of [34:33] MCPS and CLI is that we want to make [34:36] sure that we're knowing when a change is [34:39] being made that there's no regressions. [34:41] It's actually an improvement and like [34:44] Dev was already talking about a lot of [34:46] the complexities because like it's a [34:47] very diverse landscape. Uh there's like [34:50] new versions of LLMs every day. There's [34:53] like different coding tools that people [34:55] use. A coding tool that's popular today [34:57] might not be popular tomorrow. It can [35:00] change overnight. So yeah, we do need to [35:02] have a way to figure out okay, how [35:05] effective are our tools for our users [35:07] and that's I think uh something that uh [35:10] dev has been working on on as well. [35:13] >> Yeah. So well we can go into evaluations [35:16] and benchmarks if we want to but just [35:19] initially and Ben I think alluded to [35:20] this previously around where you know [35:22] models models have training data, right? [35:25] So without without any external data or [35:27] tools they they have a certain amount of [35:29] knowledge baked into them and if they've [35:31] been trained on things if they've been [35:32] trained on CLI before they'll have [35:33] they'll have more information about or [35:35] knowledge about how they um how they [35:36] work and stuff. So you'll you'll see um [35:39] agents use like the GitHub CLI very [35:42] effectively today because it's been [35:43] around for a long time. It's very stable [35:44] and and things like that. But um what [35:47] we're seeing especially when GCX as GCX [35:50] is so new it's um sometimes the training [35:52] data is kind of fighting the way that we [35:54] want to to present the commands and [35:56] stuff. Um a specific example of this is [35:59] um it loves well cloud code loves to um [36:03] or Sonnet and Opus love to kind of [36:05] redirect stud error to to stud out and [36:08] kind of slurp all of that up into it [36:10] into its input and do things like with [36:13] that. But um it tries to do that and [36:16] then it tries to you know compose the [36:18] output of that command command line tool [36:20] call into a Python script or into jq or [36:24] something like this. Right? So what we [36:26] were doing is kind of adding hints and [36:28] other kind of usage kind of ideas to the [36:32] stud error output, right? But when it [36:34] kind of redirects that out to stud out [36:36] as well and then tries to pipe that into [36:38] some some other thing that assumes, you [36:40] know, pure JSON output or something like [36:42] this, it was falling over getting an [36:43] error. So there's a few little quirks [36:45] like that that we're learning about [36:46] along the way. Um, and all of those [36:48] kinds of things uh change constantly [36:50] over time as these models are trained on [36:52] new data and things like that. So [36:54] regardless of what we try and pin down [36:57] to kind of you know learn about what we [36:59] need to change to make our kind of [37:01] command line interface and skills better [37:04] the the landscape around those things is [37:05] changing too [37:09] >> better. Are there any other ways that [37:12] you think uh any other use cases or [37:15] things that MCP servers are are better [37:18] at than CLI? [37:20] Um, so my I guess is an extension of the [37:25] previous thing like I think that if an [37:27] MCP server is designed correctly then it [37:31] it should kind of serve the needs of [37:34] like 80% of the use cases that people [37:36] have really well and that means that [37:39] like there will be some slightly higher [37:42] level tools rather than just raw API [37:45] call rappers to do the most common tasks [37:47] that users ask for. um to kind of allow [37:52] users to do most of the things they need [37:54] to do maybe with an escape hatch to do [37:57] things that you know we don't want a [37:58] direct tool call tool call for to avoid [38:01] having to define hundreds of tools for [38:04] every like esoteric use case that you [38:06] might have. So one example of this in [38:08] the graphana world would be to like [38:11] query uh your data sources query [38:14] Prometheus or to create a dashboard. [38:16] Those are some really common use cases [38:18] that we can provide really decent tools [38:21] for with good descriptions like decent [38:23] test parameters and and that allows us [38:25] to serve those common use cases really [38:27] well. Um less so the case for things [38:31] like I don't know interacting with some [38:34] of the less commonly used cloud APIs [38:36] that we may have uh things that you set [38:38] up once or twice and then never really [38:41] touch again. Is it worth defining a tool [38:44] and eating up the context window of then [38:46] every single user who uses that tool? [38:49] That's like some that's dollars on my [38:51] conscience that I have to justify every [38:54] single time someone uses it just just to [38:56] kind of occasionally serve those those [38:59] uses which and often that is not the [39:01] case. Um, and I think like that is one [39:04] of the things that I would recommend GCX [39:06] for without a shadow of a doubt is like [39:09] if you need to do everything like you [39:12] need to be able to do all of these [39:13] different things that for example [39:14] Graphana allows you to do or Grafana [39:16] cloud allows you to do GGX is is a [39:19] really really good uh avenue for that. [39:25] Okay, maybe let's um go to the CLI side [39:29] of things and uh Ward, do you want to [39:32] talk about GCX and what it is and why [39:35] why you thought that MCP was not enough? [39:40] [snorts] [39:40] >> Yeah, exactly. Um [laughter] [39:43] um I I think MCP was enough like again [39:46] right [laughter] [39:47] like I'm not um I'm not changing minds. [39:50] I think it's and and and not or or or [39:52] definitely. [laughter] [39:54] Um [39:54] >> well, if he's team MCP, then you must be [39:58] team CLI. I don't know what that [39:59] >> I don't like that framing. I I reject [40:01] the framing [laughter] at all. But like [40:02] you can try, but I I reject it. No. Uh [40:06] so let let me just talk like what what [40:08] we try to do because like CLIs are very [40:11] effective like that that is uh [40:13] definitely something that you see when [40:14] you use cloud code and use CLI. So for [40:16] example, I use the GitHub CLI a lot and [40:18] I love it. It's great. it works very [40:20] well. Um could be that there's like a [40:22] GitHub MCP server that also worked [40:24] great. Haven't used this but um uh what [40:27] what we noticed is that um like [40:30] designing well let's first start with [40:32] what GCX is because we named uh we [40:34] referred to it uh quite a few times [40:36] already. So GCX is the new Graphana [40:39] cloud CLI. So there have been uh maybe [40:43] quite a few already kind of CLI attempts [40:45] for Cfana in the past and these are [40:48] still great but we never had a CLI that [40:51] was focused on all the endpoints all the [40:53] solutions all the products that we have [40:54] in Kfana cloud. So this is our well I [40:58] don't want to say attempt because it's [41:00] actually already in public preview. It's [41:01] going to go G. It's actually uh already [41:04] quite successful but this is our [41:06] approach uh to a CLI optimized for [41:09] aentic usage for Grafana cloud and uh [41:12] yeah we're like I said we're now on the [41:14] road to GA. Uh there's uh quite a few uh [41:18] Grafana users uh Grafana cloud users [41:21] that are adopting GCX. So we now seeing [41:24] like hundreds of organizations are [41:26] actually using it. We don't have the [41:28] amount of GitHub stars yet as uh the MCP [41:31] server has. So we're uh we're slightly [41:34] behind. We need to do better. So if [41:36] people are listening, go to GCX on [41:38] GitHub and give it a star. Just see if [41:40] we can maybe maybe if there is a battle [41:43] MCP versus GCX, let's see it in the [41:45] GitHub stars. Maybe that's the nice way [41:47] to go about it. Yeah, that is [41:50] I see Ben sweating there. Let's see. [41:52] Let's see. [laughter] [41:53] >> I'm sweating. Yeah, I've got a year [41:55] advantage, but I don't know if I can [41:56] maintain that [clears throat] [41:58] [laughter] [41:59] >> and uh yeah, we see also like uh like in [42:01] our API usage like people are heavily [42:04] using uh GCX already. We see like [42:06] millions of API calls. Uh so that's [42:08] really exciting and we're going to go um [42:10] yeah, we want to go GA as fast as [42:12] possible, but there's just some a few [42:14] things that we want to get in order [42:16] before we we officially stamp at G. So [42:20] uh there's a few things that we focused [42:21] on when we were building uh GCX. So of [42:24] course it needed to support the entire [42:26] Grafana cloud API. Uh but we also wanted [42:29] to make it agent native and I already [42:32] spoke a little bit about that. But what [42:33] it uh concretely means is that we're [42:36] detecting whether uh like you're running [42:39] in agent mode. So cloud code cursor [42:42] co-pilot whatever and then it flips the [42:44] output. But we always make sure that uh [42:46] when an agent is asking for stuff uh we [42:50] give an agent friendly output typically [42:52] JSON. Uh if you for example as a user [42:55] call that API without any specific flags [42:58] we typically give you uh CLI text that [43:01] renders nicely like tabular information [43:04] uh renders very nicely. So uh we also [43:07] make sure that we add hints to all the [43:09] commands. So there's like an entire [43:11] command tree. There's for every piece of [43:14] graphana there's basically a command and [43:16] subcomands. And we also make sure that [43:18] we add hints. So the hints about hey if [43:21] you call this command as an agent you [43:24] get um probably um medium sized output [43:28] or large output. So uh so that maybe the [43:31] agent can look at okay this is maybe a [43:34] command that I should not directly [43:36] trigger because I can maybe get a huge [43:38] result back. Um, so the agent uh is [43:42] hopefully based on that information know [43:45] what the costs are of calling those APIs [43:47] and maybe to narrow down the API or the [43:50] commands that it needs to run before [43:52] running or uh like the plan to make the [43:56] like the plan to interact with uh GCX in [43:59] that sense. Another thing that we did is [44:01] also uh we spilled a file. So let's say [44:04] you have a big payload, maybe a very big [44:06] dashboard or like a long list of alerts. [44:09] You don't want to put that in the [44:12] context window like hey agent here you [44:14] have like one megabyte of JSON that's [44:17] not very that makes the agent not very [44:19] happy. So we spill to disk in a JSON [44:21] format and then we tell it what the [44:23] structure is so that the agent can then [44:26] decide how to read that data in [44:28] effective way so the context stays clean [44:30] there. Uh two other things that we did. [44:34] Um so uh we with the hints we tell it [44:38] how it can select some fields. So if you [44:41] are for example returning alerts what [44:43] what is important for alerts it's maybe [44:45] like the uh the name uh maybe some [44:48] labels maybe like the query that's being [44:50] executed maybe the thresholds. So uh [44:53] these are all fields within JSON and we [44:56] uh allow the agent to specify which [44:59] fields it wants to get back and that's [45:01] all again again to make sure that it's [45:04] as efficient as possible and then the [45:06] last thing that we do is we have [45:08] semantic exit codes and JSON errors. So [45:11] in place in in place of retry and abort [45:15] um we we can actually u make sure that [45:18] the agent understand what goes wrong and [45:21] based on that take the next action uh [45:24] plan the next action and make that a [45:26] little bit better and yeah one of the [45:28] things that we're I'm focusing on right [45:30] now and thinking a lot about is that uh [45:32] yeah we want the feedback loop so we [45:34] want to understand how people are using [45:35] it and where agents that people are [45:38] using are using GCX wrong or that GCX is [45:42] wrong and uh basically use that as kind [45:45] of a feedback loop to learn how we can [45:47] improve GCX and make it more effective. [45:50] So that could be like maybe anonymized [45:53] usage uh reports or something like that [45:55] or maybe that people can ship uh like um [45:59] debugging or troubleshooting payload to [46:01] us. But we we want to get that feedback [46:03] loop running to make sure that GCX is as [46:06] efficient as possible. [46:08] Anyway, so that's a little bit about [46:10] GCX. [46:13] >> Great. I I kind of feel like I want I [46:15] want to see the it what it looks like in [46:17] practice, like to to really see how you [46:20] do something in GCX or Yeah. And then [46:23] how you do something with Graphana MCP. [46:26] Uh Ben, do you want to go through the [46:28] demo? [46:30] >> Yeah. Yeah, sure. Um okay, let's try it. [46:34] I'll I'll explain what's going on once [46:36] uh you see Oh, we got it up. Amazing. [46:38] Okay. So, um this quick demo is just uh [46:41] two side by side clawed codes and it's [46:46] intentional that they look very similar [46:47] right now. Uh the one on the left has an [46:49] MCP server configured for this specific [46:52] clawed code instance and the one on the [46:54] right does not but it has some skills [46:56] explaining how to use GCX in the like [46:59] local directory. Um so [clears throat] [47:01] what I plan on doing is basically asking [47:03] it to do the same thing. Um, and the sec [47:07] in the second one I'm going to say use [47:08] GCX because I haven't told it more about [47:11] GCX yet. Um, and this task is actually [47:16] one that the MCP server doesn't have a [47:18] direct tool for. So, it's a little bit [47:20] disadvantaged, I suppose. Um, we'll see [47:22] how it goes and we'll see what it gets [47:23] up to because we have plenty of other [47:26] examples of things that the MCP server [47:28] does have a tool for. So, let's go. So, [47:32] all I'm going to do is send these and we [47:34] can I mean it's going to be impossible [47:36] to keep track of all of the things it's [47:37] doing. Um, but what this should do is is [47:42] sort of this one is now calling MCP [47:44] Graphana, which is the MCP server that [47:46] we have uh on the left. And um the one [47:52] on the right is using GCX here. Um, it's [47:56] discovered that GCX is a CLI and now [47:58] it's going through all of the different [48:00] help that it can find to find how to [48:03] create an SLO. Um, I don't know if I [48:06] even talked about the question here. [48:07] It's actually create a graphana SLO for [48:09] my assistant metrics group by cluster. [48:12] Um, so what this should do is use [48:14] they're both talking to the exact same [48:15] graphana instance and we're going to [48:17] look to see like which one how each one [48:21] achieves the goal. Uh [48:24] so actually this is very quick. Uh the [48:26] one on the right here, the GCX one, uh [48:29] almost immediately it has uh looked [48:32] through all of the different GCX [48:33] commands. Uh it's written out uh what it [48:36] knows a G an SLO should look like to a [48:39] YAML file that we can then check and do. [48:41] And then it's done a dry run using some [48:43] GCX commands here. Um and then it's gone [48:47] gone ahead and uh actually created it [48:50] using the SLA command there. does very [48:52] quick. So, this one cost um I'm just [48:56] going to zoom in slightly. This one [48:57] cost4 [49:00] uh to do this. Uh the one on the left is [49:03] asking me some follow-up questions. [49:05] Let's talk about this one. What what go [49:07] and have a look at what it's done. So, [49:09] it's looked for some data sources using [49:10] some NCP tools. Uh it's searched for [49:13] labels using the MCP tools. It searchs [49:15] for metrics and it's querying to figure [49:19] out how the SLO API should look because [49:22] it doesn't have a tool for that. Um, and [49:25] then it's going to go ahead and it's [49:27] going to ask me a couple of questions [49:28] about how to do this. Um, hopefully it [49:32] will now call the Graphana API to create [49:35] that SLO and we'll see exactly what what [49:39] we wanted. So, both of those have [49:40] achieved this goal. It's managed to [49:42] create the SLO there. um they've just [49:45] done it in very different ways. One of [49:47] them has just hit the API directly [49:48] because it didn't have a tool for it. Um [49:51] so I mean this doesn't necessarily [49:54] answer the question of which one you [49:56] should use. So I'm going to trying to [49:57] find out how much this one has has cost [49:59] on the left. Um but I'm zoomed in really [50:03] far so I can't actually see the the bar [50:05] at the bottom. Uh [50:08] >> Sven, another engineer, Sven Gson, [50:11] another engineer on the AI team says AI [50:14] observability could help keep track of [50:16] what they're doing. [laughter] [50:19] >> Very [snorts] true. Yes. And I need to [50:21] get both of those configured in the [50:22] right way. Um [50:24] >> you're still on Opus 4.6, man. Didn't [50:26] you hear? [50:26] >> Yeah. I didn't like 4.7 and then I was I [50:29] was actually on PTO last week, so I [50:31] haven't got around to switching [50:33] everything to 4.8. [laughter] [50:35] So, this is immediately invalid. [50:37] Everything that I've done here means [50:38] nothing. Um, but this one cost 3 cents [50:41] more. So, I guess it's an obvious win [50:44] for GGX. And I need to just retire [50:48] >> or add a tool. [laughter] [50:49] >> Or add a tool or add a tool. Yeah. Um, [50:52] so so yeah, that's that's kind of a [50:56] little bit of a demo of how they how [50:57] each works. Uh, hopefully that's [51:00] instructive at least in some way. [51:04] Zoom back in a bit now. [51:05] >> Yeah. Okay. Well, they both they both [51:07] got there and the price difference is [51:10] not enough that I would be like, "Yes, [51:12] definitely use this." So, it's still [51:15] kind of like, "Oh, which one do you use [51:17] for which situation?" [51:19] >> Yeah. Um and and one of the ways that we [51:22] can help to figure this out is I like to [51:25] write some kind of decision tree and and [51:28] there's some some [51:30] uh aspects of that decision where it's [51:32] really easy to make the the choice [51:35] because if you are using a remote maybe [51:39] you're using claw.ai AI or chat DBT [51:42] remote and you know that doesn't have uh [51:45] a shell so it can't just use it can't [51:49] just use GCX because it can't run GCX [51:51] commands um in that case you have to use [51:53] a remote MCP server like that is the [51:56] solution um and in others it's less [52:00] obvious like if you're using an IDE [52:01] locally or you're using CL code um it [52:04] could be as much as a personal [52:06] preference it could be uh whether you [52:09] are happy running [52:12] binaries, arbitrary binaries on your [52:14] machine. No shade on GCX because they're [52:16] all very they're very good binaries I'm [52:18] sure. Um and or it could be whether you [52:22] need to perform more like esoteric tasks [52:25] as we talked about. Um another another [52:28] way we can try to figure this out is by [52:31] running evaluations and actually looking [52:34] at which of these is better for an [52:37] individual task. uh whether that's uh [52:40] for cost or for token usage or for speed [52:45] as uh as we just got a comment I saw. Um [52:48] yes, it could be that speed is what you [52:49] want to optimize in which case you know [52:51] you choose that. Um so Dave, I'm going [52:54] to hand over to you because I know [52:55] you've got things to talk about here. [52:58] >> Yeah, thanks. So, you know, for for our [53:00] own education as well as uh as well as [53:02] for curiosity, [53:04] what we've been doing is um running [53:06] running uh the the Oben suite, which [53:09] I'll explain what that is in a minute. [53:10] Um both, you know, with uh with the MCP [53:13] tool and with GCX as the as the door to [53:16] Grafana, the analogy that we've been [53:18] using, isn't it? So very briefly, [53:20] Olebench is a is a is a bench is an [53:23] observability benchmark that Grafana [53:25] Labs have defined and it uh contains [53:28] this is this is publicly available open [53:30] source. You can go and run it yourself [53:31] with your own agents um if you want to [53:33] which is always quite interesting. So by [53:36] default um well and this was originally [53:39] defined to kind of try and test [53:40] different models against uh or to see [53:42] how the different models perform uh for [53:44] interacting you know against graphana to [53:47] perform a variety of kind of [53:49] observability related tasks right so you [53:52] know originally it was for kind of you [53:54] fix everything down and you compare you [53:55] compare the models but you know for for [53:58] our use case for developing GCX and and [54:00] the skills that we bundle with it we [54:03] kind of flip that a little bit right so [54:04] we fix the model as well as the harness. [54:06] And what we what we we adjust slightly [54:08] in the harness is we adjust the the way [54:10] that it interacts with Graphfana, right? [54:12] So instead of we can run it with with [54:14] MTP, which is the default, and we can [54:16] also run it then with GCX um and install [54:19] the skills that GCX has um and run it [54:21] like so. And so you know, this will [54:24] provide a bit bit more balance and a bit [54:26] more data maybe to the to the anecdotal [54:28] run that Ben had. Um and it's worth [54:30] mentioning as well that all of the [54:32] category categories of tests here. So [54:34] you'll see all of the um I think all of [54:37] the tasks that are kind of defined in [54:38] this benchmark there are um MCP tools [54:41] for the majority of them right um so [54:45] yeah there are 60 63 tasks across these [54:47] five categories this particular run here [54:49] is we're running um opus 46 on both of [54:51] these uh benchmarks this one's running [54:53] MTP and then I have another tab up here [54:55] where we can see we're running DCX but [54:58] everything else in theory is the same so [55:01] you know we can you can spend hours kind [55:03] of digging through this um but there are [55:04] some headline numbers at the top. Um so [55:07] for MCP like about twothirds of the uh [55:10] tasks completed or like passed three [55:12] times. We run this with a kind of n of [55:14] three. Um and it costs about $50 in [55:17] tokens essentially to to run all of [55:19] this. Um and you you have some you know [55:22] each test has a grading mechanism [55:24] defined grading mechanism to be scored. [55:26] There's some score associated with each [55:28] task too. [55:30] And so the same, you know, compare those [55:31] headlines to running against GPX, GCX, [55:34] sorry, with skills. So it's more more [55:35] expensive for the runs at the minute. Uh [55:37] the mean score is slightly lower. [55:39] There's a slightly lower kind of uh uh [55:42] incidence of all three kind of uh runs [55:45] all passing. Um so we we see like a [55:48] slightly degraded performance when using [55:50] GCX. [55:52] Now this is really useful for us [55:53] internally for uh what what we could do [55:56] is compare these scores. we can go and [55:58] look inside a particular task, right? Uh [56:00] and you can see you'll see the prompt [56:02] for each task at the top. You'll tell it [56:04] kind of it will show what it tried to [56:05] do. Um and it has kind of then uh the [56:08] kind of trajectory or the path that the [56:10] agent took through all of these various [56:11] things, all of the tool calls it made. [56:13] And you can see the responses and I go [56:15] through that. And so this is really [56:16] useful for us developing GCX. And you [56:18] can see sorry what skills it loaded and [56:20] things like that too. So we can see, you [56:23] know, compare maybe the categories where [56:25] it didn't perform so well compared to [56:27] MCP or maybe, you know, just compared [56:29] like performed badly on its own as well. [56:31] Um, and we can go and inspect to see [56:34] where some of those tools uh, you know, [56:36] where the agent went wrong or maybe [56:38] where the agent kind of misused the the [56:40] tool or maybe we've designed the tool in [56:42] a in a way that's easily misused or [56:45] doesn't fit the scenario and things like [56:47] that. Um and then we can decide to [56:49] either you know fix the make changes to [56:51] GCX, we can make changes to the skills. [56:54] Um and you know for full transparency as [56:57] these are all quite new things like [56:58] sometimes there are kind of little [57:00] quirks within the benchmarking suite as [57:02] well that we need to adjust right so [57:05] there's like each one of these tests has [57:07] a a grading rubric. Um, and maybe they [57:11] were, you know, when the whole [57:12] benchmarking suite was designed with the [57:14] assumption that it would use MCP as the [57:16] front door to Grafana. Um, so we've kind [57:18] of broken that assumption, you know, by [57:20] by sticking GCX in there instead. And [57:22] maybe, you know, one of the grading [57:24] criteria, for example, um, was [57:27] originally checking that a specific uh, [57:30] tool was called, right? So the agent's [57:32] never going to a specific MCP tool was [57:35] called. So that's never going to be true [57:36] in the GCX version. Um so you know we [57:40] have to go and adjust that to be less [57:41] kind of or to be more agnostic of uh [57:44] what you know how it got to the end [57:45] result that kind of thing. So yeah the [57:48] so these kind of benchmarking runs are [57:50] really useful for the headline scores [57:52] and then for for digging into each of [57:54] those uh kind of paths that the agent [57:58] took to to perform each of those tasks [58:00] so that we can go and learn from that [58:02] and iterate and improve the GCX uh tool [58:05] and the skills that we provide with it. [58:07] Awesome. [58:11] This is a great way to to dog food, too. [58:13] Like now we we needed this for ourselves [58:16] to be able to to see more uh to test to [58:20] evaluate our own our own AI tools more. [58:24] Um, and then now Olebench is something [58:26] that's available to everyone and is not [58:28] just for our stuff, but it is pretty [58:30] handy to to pit GCX and and um MCP like [58:35] this and see like what what's one good [58:37] at, what's the other one better at and [58:38] stuff. Um, but I want to also kind of [58:42] talk about other Graphfana things like [58:45] that that might be mentioned in the same [58:46] breath like assistant that is sometimes [58:49] conflated with one or both of MCP and [58:53] CLI. How would you say they're related [58:56] to assistant? [59:02] That's a good question. Um I think that [59:05] the [59:07] GCX and MCP are almost like the tool [59:10] similar to analogist to the tools that [59:12] the assistant has available. And [59:13] actually assistant is more similar to [59:15] clawed code or you know the actual agent [59:18] harness like it has the whole harness [59:20] from running the agent loop to calling [59:24] different tools um different ways of [59:27] managing context and doing tool search [59:29] and all of those things that that the [59:31] MCP host or the you know the the agent [59:34] application normally has. Um the big [59:36] advantage that Grafana assistant has is [59:38] that it's really well optimized for [59:41] Grafana specific tasks especially things [59:43] like creating dashboards where you know [59:46] it needs it can it can actually [59:48] visualize the dashboards as it goes and [59:50] it can it can interact with a DOM and it [59:52] can update things live in front of you [59:55] as you're as you're like visual as [59:57] you're working through creating a [59:59] dashboard from example. you can you can [60:01] change these things which is really [60:02] really good and and it also has some [60:04] some quite bespoke tools there for for [60:07] doing Grafana specific things across the [60:09] whole suite between like not not just [60:12] like doing those raw API calls but it [60:15] also knows all the way the full workflow [60:17] all the way through. Um, so yeah, I I I [60:22] always reach for Grafana Assistant if [60:23] I'm doing anything to do with like [60:25] creating dashboards or interacting, you [60:28] know, debugging issues that I know [60:30] involve several different data sources [60:32] or, you know, I've started on one page [60:35] and it can navigate me to others. I find [60:37] it really really powerful for that, [60:39] especially for like the investigation [60:41] side of things. [60:42] >> Yeah. [60:43] >> Yeah. I'd say for anything visual [60:45] especially, right? Like I would go to [60:47] assistant first if I want to see it or [60:49] if I want to learn something like teach [60:51] me how to do this thing like not just [60:54] create an SLO for me but like where is [60:56] that on the interface then assistant [60:58] would be better for that. [61:00] Yeah, exactly. And and like uh have the [61:03] assistant also has access to the infra [61:06] memories to the knowledge graph. Uh for [61:08] the folks that are familiar with Grafana [61:10] cloud schema context um like the we we [61:14] put a lot of effort in the [61:17] investigations. Assistant investigations [61:19] is part of the assistant where we have a [61:21] validated workflow working doing root [61:24] cause analysis based on hypothesis. And [61:27] uh there's a lot of value there because [61:29] the thing is like with GCX with MCP you [61:33] you definitely have the hands to operate [61:36] Grafana cloud to work with Grafana cloud [61:38] but um like it still needs to improvise [61:42] uh based on the knowledge that the LLM [61:45] has and the benefit of the assistant is [61:47] that we we basically do you say that the [61:50] full banana was that the the way that uh [61:52] that was always mentioned I don't know [61:54] what the term is but we we basically do [61:57] um it knows Grafana cloud. It has like [62:00] great insights about what is actually [62:02] happening with your services and not so [62:04] much on a like raw telemetry view or [62:07] level but actually uh like um uh real [62:11] time outlier detection and that kind of [62:14] stuff. So like I think that assistance [62:17] is still like a very important part of [62:21] getting kind of valuable insights and [62:24] then like for the more deterministic [62:26] things like GCX and MCP are great but [62:29] you always need the assistance for the [62:31] brains if you will [62:34] >> and and I also wanted to mention because [62:36] people might get confused uh that [62:39] there's there's Grafana MCP but then [62:41] there's also K6's MCP P and that's a [62:46] very different thing because they're [62:47] they're targeting they have different [62:50] goals. So the graphana MCP is about uh [62:52] your observability stack and the Ksix [62:55] MCP is about writing Ksix writing and [62:58] running your own Ksix tests on on your [63:00] machine. So different things but just [63:03] because um there might be a confusion [63:05] there if you look for MCP and they both [63:08] come up. Uh just to wrap things up [63:11] because we are out of time. I just [63:13] wanted to do a quick like when to use [63:16] which for Graphfana. Luckily, we we had [63:18] this beforehand. So, I'm just going to [63:20] read it out. If you have a non-terminal [63:23] client, so something like clawed desktop [63:26] or cursor, then MCP is better. If you're [63:29] building automated pipelines or scripts, [63:32] CLI or GCX [63:34] um token constraint or if you need a [63:37] full feature coverage, then the CLI is [63:39] better because you have you get the full [63:42] banana, which is what Ward said. [63:45] [laughter] [63:45] Um if you want zero setup authentication [63:48] for cloud users, then MCP is better uh [63:52] for hosted. And then if you have complex [63:55] multi-step workflows with data [63:57] transformation then you would use the [63:59] CLI plus jq and pipes. Um and then if [64:04] both are available then start with MCP [64:06] for discovery or actually I would say [64:08] start with assistant first honestly and [64:11] then and then go to MCP and get familiar [64:15] with all the tools available and then [64:17] when you're ready for the heavy lifting [64:19] switch to GCX and then someone said if [64:22] you're called Ben then just MCP all the [64:24] way. [64:25] That wasn't me by the way. [64:29] >> I was having a question. Um, so like if [64:32] you're using assistant where you can add [64:34] your own MCP servers, like is there a [64:37] benefit to specifically in assistant [64:39] adding like graphana like the open [64:41] source graphana MCP for instance? Like [64:43] does it do anything differently there [64:45] versus if you just had assistant by [64:46] itself or whatnot? [64:49] >> Uh, that's a good question. Um, not [64:51] something we've ever really thought [64:52] about before, but I can definitely think [64:54] of some use cases where the assistant [64:56] itself doesn't have tools for these [64:58] things. And one of them is just that it [65:00] can't make arbitrary calls to graphana [65:02] as a API unless I've missed something [65:05] over the last week. And the MCP server [65:07] does allow that. So it, you know, it [65:10] almost opens up limitless possibilities [65:12] for the assistant because actually these [65:15] agents do understand the API quite well. [65:18] I think all of the docs are in the [65:20] training set and you know we just saw uh [65:23] claude code managed to make the right [65:24] API call to create an SLO even. Um so [65:28] yeah you could use it for that for [65:30] example there are some other tools there [65:31] that maybe the assistant doesn't quite [65:33] cover although I I can't think there [65:34] would be many at this point um I think [65:36] the assistant is generally got a pretty [65:38] decent coverage of those things. Yeah, [65:42] and it's also a little bit turtles all [65:43] the way down because we can also think [65:46] about maybe running GCX as part of the [65:48] agent environment of the assistant [65:51] environment so that you like the [65:52] assistant now can also execute Python [65:54] scripts. So it could also call the GCX [65:58] CLI as part of it. So you get all the [66:00] functionality of GCX also available [66:02] within the assistant again. [66:04] >> Yeah. [66:04] >> So and then you can actually call [66:06] [laughter] with the CLI the assistant [66:09] again. So turtles all the way down. It's [66:12] infinite recursion. [66:13] >> It's the Spider-Man meme just pointing [66:16] to [laughter] Yeah. So if people are uh [66:19] like probably people will leave from [66:21] this podcast more confused than be they [66:24] were before probably. That's a [66:27] [laughter] [66:28] >> Yeah. Well, you know, join the club. [66:30] Like, uh, this is an attempt to to try [66:32] and talk things out um in the current [66:35] state of everything. But the reality is [66:38] like it's not as simple as like you [66:41] should always use this tool. Like we [66:43] don't want to we don't want to do that. [66:44] Um, we don't want to have to choose as a [66:47] company which one is the better bet or [66:50] anything. So, we're just kind of saying [66:52] yes to everything and we'll see what [66:53] sticks. [66:57] Yeah, it's a game of trade-offs as with [66:58] so many other things in in this field, [67:01] right? [67:02] >> Yeah. And things keep advancing so much [67:04] and changing so much that like it could [67:07] be completely different in a couple [67:08] months for instance. [67:10] >> Yep. It's you all back again. [67:14] >> Yeah. Yeah. Indeed. Yeah. We can talk [67:16] about our next pet next project that [67:18] we're going to launch. Indeed. [67:19] >> It has to be a threeletter acronym [67:22] though. Like that's we can't accept [67:23] anything else. [67:25] >> [laughter] [67:25] >> That's real acronym. Okay, we can [67:27] already start brainstorming the name. [67:29] Then we can use the name and then [67:30] [laughter] [67:31] we can figure out the technology that is [67:32] actually powering it later. [67:34] >> Okay, sounds like a good plan. Thank you [67:37] everyone for joining us. Thank you to to [67:40] the people that are on here now watching [67:43] us. And then also if you're watching [67:45] this after the fact and if anyone has [67:47] any questions, uh we might not know the [67:50] answers, but we'll get you someone who [67:52] does. [laughter] [67:54] So just leave them in the comments [67:55] below. [67:56] >> Yeah. And then there's also uh [67:58] slack.garfana.com. [67:59] Um there's like a graphana assistant [68:01] channel in there for instance. But yeah, [68:02] basically it's the community Slack. So [68:05] if you want to ask questions in there, [68:06] talk to people, etc. [68:09] And the next episode is actually going [68:11] to be with it's a recorded one. Not [68:13] because not because u we didn't want to [68:15] do it live, but because the person that [68:18] I interviewed, Yas, is is going away for [68:21] a bit and I wanted to catch him before [68:22] he did. But it's going to be all about [68:25] eval focus on offline eval since we did [68:28] online with AI observability a couple [68:31] episodes ago. But watch out for that. [68:34] But thank you everyone for watching and [68:37] we'll see you next  📄 **Full text:** ![[Full Document Contents/Articles/CLI and MCP Basics for AI Agents-2.md|CLI and MCP Basics for AI Agents by Grafana / The Context Window]] ## Highlights - So uh so I've been with Grafana Labs for almost seven years now and I had various functions but uh currently I'm a product manager focused on Grafana cloud and specifically nowadays on GCX. >> Yeah, I guess I'll go last. So I'm I'm David. I've been here about the same time as Ben. So just over five years I've been building GCX with Ward. Um and most recently been focusing on trying to understand how GCX performs. um with his skills against other other tools. >> Awesome. So, I I guess today we're going to be talking about MCP versus CLI, what all those letters mean and which one Ben thinks is appropriate for all situations. >> The big the big versus word came out. ([View Highlight](https://read.readwise.io/read/01ktskrnhr1a8nc6f7pcy4pr6n)) - Note: Grafana offers both GCX (CLI) and MCP servers for cloud integration, with evaluations showing comparable performance - Tags: atom/literature/h2, mcp-vs-cli - And it gives them the power to interact with the real world in in some way rather than just being like text in and text out. Uh so it allows the assistant to do all of the clever things it can do such as looking for dashboards, creating dashboards. Basically, it allows them to allows the agents to yeah perform useful tasks and to do so more accurately as well by by having real knowledge bases they can query or uh being able to hit APIs in like predefined ways and I think I think that there's two about a year and a half ago there was this this kind of big thing released by anthropic which was the model context protocol CP. ([View Highlight](https://read.readwise.io/read/01ktskrf5pa355x763vrgj3h1y)) - Note: Grafana team discusses two approaches for AI agents to interact with observability tools: MCP servers vs CLI tools - Tags: mcp-vs-cli, atom/literature/h1 - it has some commands where you can then find indeed okay what are the commands that I need and that is basically discovered by the agent and so you could say that a CLI has way less token usage but again that like those those differences become less and less uh relevant I think um or well not relevant but they're they're like closing in on each other >> I think one area Where I I like MCP better is that um I feel like the tool discoverability is better on MCP. Um on CLI, you know, you kind of have to know those commands and yeah, you can go through like d-help or whatever. Uh but you still have to go through each one and then each command has different um parameters and stuff. ([View Highlight](https://read.readwise.io/read/01ktskrjc2x4he4k810ktbw838)) - Note: Both achieve similar outcomes with different trade-offs in token usage, discoverability, and implementation complexity - Tags: mcp-vs-cli, atom/literature/h2 - Um and and one of the ways that we can help to figure this out is I like to write some kind of decision tree and and there's some some uh aspects of that decision where it's really easy to make the the choice because if you are using a remote maybe you're using claw.ai AI or chat DBT remote and you know that doesn't have uh a shell so it can't just use it can't just use GCX because it can't run GCX commands um in that case you have to use a remote MCP server like that is the solution um and in others it's less obvious like if you're using an IDE locally or you're using CL code um it could be as much as a personal preference it could be uh whether you are happy running binaries, arbitrary binaries on your machine. ([View Highlight](https://read.readwise.io/read/01ktskrrmmjmcdy1b4v15swzr5)) - Note: Choice depends on use case, environment constraints (remote vs local shell), and personal preference - Tags: mcp-vs-cli, atom/literature/h3