Overview of AI Evaluation  🔗 **Source:** [Overview of AI Evaluation](https://www.youtube.com/watch?v=_XGLBqVpL-Y#transcript) > [!note]+ Document Notes > --- > tags: [ai-evaluation, llm-evals, observability, grafana, ai-engineering, devrel] > author: Nicole Vanderhoven > source: Grafana Labs — The Context Window (podcast) > reference: https://www.youtube.com/watch?v=_XGLBqVpL-Y > date created: 2026-06-28 09:00 > --- > # 🏷️ Title: Overview of AI Evaluation (The Context Window #05) > ## 🧭 Table of Contents > - [📝 Summary](#-summary) > - [📖 Main Content](#-main-content) > - [🗒️ Notes](#️-notes) > - [📚 References](#-references) > - [🗣️ Paraphrase](#️-paraphrase) > - [📰 Newsletter](#-newsletter) > - [✅ Action Items](#-action-items) > - [🧪 Hypotheses](#-hypotheses) > - [📊 Methodology](#-methodology) > - [❓ Questions](#-questions) > - [🔮 Future Research](#-future-research) > - [💡 Implications](#-implications) > - [🧠 Learning Styles Assessment](#-learning-styles-assessment) > - [📊 Accessibility Assessment](#-accessibility-assessment) > - [🌍 DEIB Assessment](#-deib-assessment) > - [✅ Logical Validity](#-logical-validity) > - [🛡️ DARVO Analysis](#-darvo-analysis) > - [🎯 Conclusion](#-conclusion) > ### 🔗 References > - [The Context Window #05 (video)](https://www.youtube.com/watch?v=_XGLBqVpL-Y) > - [Anthropic — "Demystifying Evals"](https://www.anthropic.com) > - [Ollie Bench (Grafana AI benchmark)](https://github.com/grafana) > - [LMArena.ai](https://lmarena.ai) > - [Matt Schumer — "Something Big Is Happening"](https://www.mattschumer.com) > - [Unprompted (Grafana Labs Medium blog)](https://medium.com) > ## 📝 Summary > - **What happened.** Grafana developer advocate Nicole Vanderhoven interviews AI engineer Yas (pre-recorded ahead of his two-month break) for a full episode on AI evaluation — "evals." They define what an eval is, split the practice into offline vs online evals, walk through graders/verifiers, and demo Grafana's public **Ollie Bench** benchmark plus the internal **LLM Spec** CLI tool and its self-improving "eval loop." > - **Why it matters.** As coding agents write most of the code (Yas says manual code he writes is "very close to zero"), the bottleneck shifts from writing to *verifying*. Evals are how teams turn fuzzy "is the agent good?" questions into a tracked number so they can improve reliably instead of vibe-shipping. > - **Key stakeholders.** AI/observability engineers, DevRel teams, product managers (being recruited to author evals), open-source model maintainers (Qwen, Kimi appear on the leaderboard), and labs that might tune against an open benchmark. > - **Risks and tradeoffs.** LLM-as-a-judge is itself unreliable; benchmarks can be gamed/tuned-against; agents produce *plausible but wrong* answers that pass shallow checks; reproducibility vs realism is a live tension; AI-generated tests get silently edited to pass. > - **Bottom line.** Evals are the durable differentiator. Measure → change → measure, ideally with a coding agent in the loop and verifiable graders, and keep a human reviewing the trajectory, not just the final answer. > ## 📖 Main Content > The episode reframes the testing discipline Courtney's audience already knows (pre-prod vs production, integration tests, synthetic monitoring) into AI terms. The core move: an AI agent is "programmed" in natural language, so there is no clean mapping from instructions to behavior. The only way to probe that is to run many tasks and grade outcomes. > Two families: > - **Online evals** — score real production conversations (a sample or all of them), typically via LLM-as-a-judge. Analogous to RUM / synthetic monitoring. Catches live regressions (e.g., a vendor silently swapping a model and quality dropping) but only tests the deployed variant and is hard to verify. > - **Offline evals** — a controlled "golden set" run in CI/CD against a real-ish environment. Analogous to integration tests. Reproducible, lets you compare *all* changes (prompts, code, tools), and supports drill-down (PromQL vs LogQL, dashboarding, etc.). > Terms defined: a **harness** = the agent plus its tools (and now skills). A **grader/verifier** grades the full transcript of steps against the task. **Fact-based rubrics** run known-good queries against the controlled environment so the judge can catch nuanced wrong answers (e.g., a fabricated P95 latency that "looks plausible"). **pass@k vs pass^k** distinguishes "got it right at least once in three tries" from "got it right all three times" — capability vs reliability, where Yas notes 10–20% gaps. > The **eval loop**: measure current state, learn from transcripts, make targeted changes (new tool, tool instructions, system prompt, bug fixes), measure again — and crucially, hand that clean feedback signal to a coding agent so it iterates toward 100% on its own. Contested/uncertain claims: whether security-tuned models (Claude "Mythos" preview) are meaningfully better or marketing; whether benchmark rankings (the new "deep software engineering" benchmark flipping Opus vs GPT-5.5) are reliable when labs tune against them. > ### 🗒️ Notes > #### ⏱️ [00:00:00](https://www.youtube.com/watch?v=_XGLBqVpL-Y&t=0) Intro and framing > - Pre-recorded because Yas is taking a two-month break (not leaving Grafana). Hosts riff on Dutch/Turkish names and Grafana's 3-year/5-year tenure perks. > - Yas: AI engineer ~5 years; the old ML team (forecasting, anomaly detection) got absorbed into a larger AI department. > #### ⏱️ [00:04:00](https://www.youtube.com/watch?v=_XGLBqVpL-Y&t=240) AI is useful but needs guidance > - AI "definitely has use," not crypto/NFT vaporware. Guidance per session has dropped from ~50–70 steerings to ~5–10, but isn't zero. > - "If you just blindly v code your way out of this you're not going to end up in a good place." > #### ⏱️ [00:07:00](https://www.youtube.com/watch?v=_XGLBqVpL-Y&t=420) Announcements > - Grafana Assistant now available for OSS/self-hosted; new "Unprompted" community blog; Yas's first engineering post on building an evaluation loop; weekly AI team demo videos ("warts and all"). > - News: Warp open-sourced; ChatGPT's "goblin" personality bug; GPT-5.5; Gemini 3.1 Ultra 2M-token window; Anthropic Claude "Mythos" security preview; Matt Schumer's "Something Big Is Happening." > #### ⏱️ [00:14:00](https://www.youtube.com/watch?v=_XGLBqVpL-Y&t=840) Why trusting AI is hard > - Models are nondeterministic; cost of writing code → near zero, so PRs are bigger and more numerous; review burden is higher than writing burden. > - "Shepherds of intent" framing. Using AI to review AI erodes your own taste for good/bad code. > - Process vs outcome: a plausible-looking outcome with a wrong process is not trustworthy ("final answer problem"). > #### ⏱️ [00:22:00](https://www.youtube.com/watch?v=_XGLBqVpL-Y&t=1320) Human eval and defining evals > - Thumbs up/down and LMArena-style A/B human prefs are useful but don't scale into CI/CD. > - Eval defined: does the AI agent behave the way you want? Simple example — ask "who are you," expect "I am Grafana Assistant." > #### ⏱️ [00:27:00](https://www.youtube.com/watch?v=_XGLBqVpL-Y&t=1620) Offline vs online > - Online = production observability with an AI flavor; turn text into a trackable number, alert on it. Catches vendor model swaps. > - Offline = golden set in CI/CD; reproducible; tests every change and supports per-capability drill-down. > #### ⏱️ [00:34:00](https://www.youtube.com/watch?v=_XGLBqVpL-Y&t=2040) Anatomy of an eval (Anthropic "Demystifying Evals") > - Single-turn (e.g., incident summary): prompt → LLM → graded for readability, completeness, length. Easy. > - Agentic: variable trajectories (5 vs 7 steps), runs against a real environment with tools, then grade the whole transcript. Harder and slower. > #### ⏱️ [00:42:00](https://www.youtube.com/watch?v=_XGLBqVpL-Y&t=2520) Building evals for Assistant > - Tension: as real as possible vs reproducible/controllable. Started with manual test scripts (4–5 scenarios), doesn't scale to 500. > - Front-end assistant tested via Playwright (runs as the user in the browser); host considers K6's browser module reaching parity. > #### ⏱️ [00:52:00](https://www.youtube.com/watch?v=_XGLBqVpL-Y&t=3120) Ollie Bench > - Public benchmark, ~six task categories (dashboarding, investigation, logs, metrics, traces). Built on Harbor (same framework as Terminal Bench); separates harness / environment / tasks. > - Synthetic Prometheus/Loki/Tempo Hey everyone, welcome to another episode of the context window. This is a a podcast that is semi-regular. We try to make it every month and it's all about the things that are going on with AI at Graphana. Usually this is live and you might think looking at this now that this is live, but it's actually pre-recorded and that's because Yas has a busy schedule ahead and I for sure wanted to get him on here before he left. So, hi, I'm Nicole Vanderhovven. I'm a senior developer advocate at Graphfana Labs and I'm assigned to the AI team these days. And one of the people on the AI team is Whoops, I always get the directions wrong. This person over here. Yes. How do you say your last name actually? >> Oh, it's Akeni. >> Akeni. Oh, it's the C with a little thing. >> Yeah, it's Turkish last names. C is a G. Something like that. Yeah, I was totally confused because I just found out that you speak Dutch. [laughter] >> Yeah. >> Did not know that. >> I'm based in Belgium. And I was like, "Oh, Nicole Vanderhuven." I was like, "Okay, that's that sounds familiar enough to me." >> Yeah. It's the na my ex. That's the name of my ex-husband. But it's okay. We still like him. [laughter] >> Also, you before I left, but uh typically I'm not leaving Grafana. I'm just going on a two-month break. >> Sorry, I probably should. Yeah, I'm like starting rumors now that this is his last ever one. >> Got to be careful. You know how how they spread around the virtual office. >> Yeah. Yeah. No, I am Dutch. I've lived there for like nine years now. I have Dutch nationality. So, I do speak it. Not natively, but I do speak it. >> Can we do this in Dutch, Felina? No. >> Yeah. >> Specifically [laughter] for the Dutch audience. That's super niche. But how about we do it in English and tell us like what what you what you do on the AI team. >> Um yeah, so I'm Yes. I am an engineer on the AI team. I've been at Grafana now for um I think like >> at least three years. >> I mean like I'm getting to five years in next week. Uh, >> I was saying at least three years because we both have our our >> this is like this is a subtle flex for for people outside of Graphana. You only get that when you've hit three years in Grafana, right? >> The only problem is due to cost savings, we didn't get the fancy remote that the initial people got. So, we can't like dim it uh as they do. We have to actually go and turn it off, but that's about it. Um, >> animals. [laughter] >> No, what do I do? time and uh so I've kind of like when I joined we had uh this is like preAI uh we had the machine learning team and we were building like predictive models things to kind of like make sense of the signals and do forecasting out detection that kind of stuff um and as the AI wave came we kind of like all got absorbed into this like newer larger AI department um so I would say it's been quite varied so I'm working on things like obviously the assistant uh or like AI agent inside of Graphfana. Um, but it's been uh quite a lot. I mean, like it's hard to say like I'm just working on this one thing. Uh, I can say the last uh I think like the the latter half of especially like this this since this year um I've been more and more focused on specifically eval. So specifically how do we figure out what is the quality, what's the reliability of these things that we're deploying out. Um AI is like a very useful tool. Um and it works quite nicely. It it has utility, right? It's it's not like uh I was going to mention crypto or or NF no what was it? Oh my god, I even forgot the name. I was going to say NFC, but like NFTTS, but it it's definitely not like that. It definitely has um use like we use it dayto-day. I use it day-to-day. my job has changed drastically over the last two years. Um, but uh you still notice that it does sometimes need guidance like it does have sometimes have issues with reliability. It doesn't quite get it always all the time. Uh, and even in my day-to-day work like working with these coding agents, you kind of notice like you do they still even the top ones even the GPT 5.5s and the oops, they still need a lot of guidance, right? They and the thing is the trend is that the guidance has come down like if I compare with like a year ago or like two years ago uh where maybe during a session I would have to like steer it maybe 50 times 60 times 70 times that has kind of come down to like now I just need to do that five or 10 times but you still need to do it. I think if you just blindly v code your way out of this uh you're not going to end up in a good place. Yeah, I it really moves so quickly. Like whenever I hear from someone that like they tried AI and it didn't do what what they were wanting it to do, I'm like, well, when was that? Because if it's like 6 months ago, I'm like, try it again because maybe it does now, you know? And that's not that's without getting into prompt engineering and like maybe people don't don't know yet. This is a new thing. We're all learning how to prompt better. Um, but yeah, things are are moving all the time and um, in this in this episode, we're going to be talking all about eval. But first, I wanted to go through just some announcements because uh, there's there's a lot that's going on. Everything that I mentioned though is going to be linked in the description below. So, uh, I I won't be putting up the the links here, but I will put it in the description. The first is that I have a video for Graphana Assistant for Gra Graphana OSS. That's a mouthful. Assistant is available for for OSS and self-hosted Graphana now. And I just made a video showing you how to how to do that. Um, another thing is that Yas So, oh yeah, so there's this new medium blog for Graphfana Labs now, which is called Unprompted. It's a cool name, actually. And I really um I I really love the idea of it being a community focused blog for all things Graphana. And Yas just created one. Um he wrote an article about building an evaluation loop. That's kind of topical. >> Yeah, we can talk about that. >> Yeah, great. >> Um to work working on this uh and being I think the first engineering post on unprompted which is quite nice. >> Yeah, that's great. And then um there's also this thing that we're starting which is the AI weekly videos. So every week the AI team so both Yas and I are are in it have this weekly team meeting and I don't know we just Matt actually Matt Ryer had this crazy idea of like what if we just took an excerpt from that and actually published it. So it's usually around the demo part. Um, and I I edit those and I put it out and the first two are already out. So, if you ever wanted to know what it's like, but you know, just be warned that we demo things that may not ever make it into production and almost certainly even if they do will look completely different. So, like you're getting you're getting it raw. Warts and all. [laughter] >> Yeah. Yeah. I just want to make sure I need to make sure now that I don't wear my PJs anymore for the AI weekend. >> Yeah, >> it was nice meeting >> and [laughter] also need to keep a PG-13. >> Yeah. Okay. Well, also uh last month in AI some some quick AI news. Warp is now open source. Do you use warp? Yes, >> I used to, but I don't like my tooling. It's quite weird. I'm very bare bones actually. So, I'm like cloud code, codeex, and open code. I think those are my three go-tos. >> And I don't do a lot of I think in open code I just have one plugin to help me with like terminal shells. I don't do a lot of these crazy plugins and like multiple like agents that kind of like you know or like memory systems and everything. Um, everybody's making some wild claims about like, oh, use this or like use this approach. Uh, and like that's going to make your development flow like 100x better. I'm pretty bare bones. Uh, there's a good blog post about this that's just that's about like just talk to it. And it's really about like you just need to keep talking, keep guiding. And I think that works quite well. The only recent I think like shift is like I've been trying out OpenSpec and I quite like it. >> Ah, yeah. Me too. >> Yeah, it's because like you do especially the explore like the ping ponging of an ID. Uh I do feel like it works better than like just the regular plan mode. Um and then it is encoded and again like we we're going to talk about like reliability. Uh openspec can help with that as well because like now you first ping the ID and then you get like a nice little plan and you have a way to verify it if if the agent actually did did everything according to plan. Um, so yeah, I've been uh doing more and more of that. >> Yeah, I've been using that as well. And I actually have on on my list of things to do is to to get Lucas on on here to talk about OpenSpec. So that's for sure on the cards. Um, another thing, oh this is a really cool one. ChachiPD has a goblin problem. Have you heard about this? I love this story. [laughter] So what happened was that um Open AI said they released this blog post link below um saying that some versions of chat GPT they realized had this weird bug um that seemed harmless enough where it just mentioned goblins a lot and it it apparently had something to do with a particular personality that could you can you can select for chat GPT and um it's a it's an interesting read. I I recommend it. It's just so of all things goblins like actual goblins apparently had learned that you know people who are nerdy because it was the nerdy personality that was selected talk about goblins disproportion a disproportionate amount of times. [laughter] >> You know that's quite stereotypical like oh yeah nerdy it has to be like magic the gathering and all that kind of stuff. Yeah. >> I mean, >> I did play Magic, [laughter] >> you know. >> Yeah. I mean I mean, yeah, I play D and D like it's it's a stereotype for a reason. Um, there are also a few new models. GPT 5.5. Gemini 3.1 Ultra now has a 2 million token context window. That's huge. I think that's the biggest one of of like the Frontier models. Uh oh, another big thing was Anthropic talked about the Claude Mythos preview, which is their their model that is specifically targeting security vulnerabilities. >> Also quite topical [laughter] is interesting. Like I do feel like it's hard to distinguish how much of this is like marketing and hype and how much of it is real. Like I obviously it's going to be better than what we have but is it that much better or is it just optimized for security? I don't know. But they they are I think they announced it or at least I don't know if they announced it or that there are rumors that it is going to go public soon in a few weeks. So we >> Yeah, I think they they gave a few people private preview a few companies a private preview to allow them I guess to kind of fix some of those issues before they release it. So yeah, we'll see. We'll see what's happening there. Um, another another thing that I wanted to talk about was an an awesome article by Matt Schumer and uh he it's it's entitled something big is happening and and it's exactly what you were saying at the beginning that your job has changed and very drastically and very recently and there's an interesting quote in it where he says I am no longer needed for the actual technical work of my job which was like that hits hard. [laughter] >> And his advice was >> he his advice was think about where you stand and lean into what's hardest to replace. >> I thought was pretty pretty solid, >> right? I think it's still that that guidance thing, right? So there's still some piece that where right now you still need the the human in the loop, let's say, to kind of guide it. So in the but in a way it's also inevitable like um if you ask me like how much code have I manually written. So like the actual technical piece >> very close to zero in the lab. So uh that is quite interesting. We we're'll talk about this at evals as well because evals doesn't work that well as it does for coding but we'll get to that. >> H okay. Well, let's talk about um difficulties in in trusting AI to begin with. Like you talked about just basically yoloing it and and just accepting commits and you know things that we're we're exploring to soften that like open spec and stuff. Why why is it so difficult to trust AI? Why don't we just yolo all the time and just accept everything? It's it's a mix of um so I think it's a mix of I mean a there I mean everybody knows that these models they're nondeterministic and they are not perfect. These are not new problems to be fair like u in the past uh there was a thing called machine learning which is still AI. AI is just a nice little rebranding of it. Uh but at the end day all of these models they are just based on like they're all machine learning at when you get down to it it's just all numbers and math and stats. Um, and they're not perfect. They're not deterministic. So every time you talk to it, it might give you like slightly different response. Um, so I think that's one part of the problem. I think the second part is like it writes a lot of code like suddenly because it's a new tool, it it can like you can 10x your output essentially like I can create because it takes much less effort for me to actually write the code. um I can just like you know send 10 agents to do like you know 10 different features or bug fix or whatever like 10 different types of work launch all those PRs uh but then because the pace is moving so fast and because you have like tens hundreds of PRs that are open somebody needs to review them um it also just gets hard to review because a the PRs are typically quite big this is also why like you know when I talk about the guidance piece one of the one of the things I always do at the end when I'm done with like a piece of work I go okay now now try to find like a simpler array like simplify this because >> it does tend to create a lot of code like we talk about AI slop it it just creates like you know a very big overengineered solution to the problem and then you need to kind of guide it to get to uh like a much smaller version of that or you need to like break up that PR into like you know that whole branch of changes into like smaller bits uh that you can merge more easily. But yeah, so it's like both bigger PRs I think generally probably if we did a like if we actually to track this the amount of changes in a PR are bigger on average compared to human PRs. Um and because so many changes are happening it's just hard to keep track have a mental model of like the entire thing. Um like even in the assistant uh repo uh we've got like a monor repo set up. So there's like 10 12 working groups that are all working on different parts of assistant. Um and it's really hard to keep track of what every working group is doing all the time. In the beginning like I can give the example of like I was quite involved in the beginning when we were working on the slack assistant and over time like I focused on something else and like now when I try to review some of those slack uh like you know changes to slack to the slack assistant it's uh it does take me a while and interestingly I do use then AI to help me review that PR and again you get back to the same problem of like well if you >> use AI to review the AI well how much can you trust said and like you also lose a bit like your opinion of like this is good code or this is bad code. >> Yeah, I like the I forget where I got it from but I like the the phrase that now developers are becoming shepherds of intent. So we're not really writing the code anymore but we are making sure that the intent is clear and that it is implemented. >> Yeah. [clears throat] But on the review the I think the burden is Yeah. almost higher on the review side. >> On the review. Yeah. >> Yeah. Yeah. Like because like writing stuff is now so like the cost of writing code has become like so low that it's just cheap to just write a bunch of like features and changes and let's make like you know let's add like a whole new like section to the UI and like just creating that doesn't take up that much effort. But then somebody needs to actually look at that and say actually this is good or this is not good and like these are changes you need to make. Um so >> yeah it's subjective >> especially as these PRs get too big. Yeah. And and uh this is one of my problems with openspec that that it generates a lot of documentation, you know, like it is it is it is good in that it um there's like a a clear process for describing your intent and then the proposal and the design. But yeah, AI can just generate so much stuff and what are we supposed to like somehow read all of that whether it's code or just text? And then also um there's the difference between the process and the outcome. And I call this like the final answer problem because often cuz I I'm I'm a tester. I I come from a performance testing background and I'm used to looking at the outcome and saying did this test pass or fail. But now even if the outcome is looks like it might pass and it is what you expected, if the process is wrong is wrong, I I don't trust that outcome. Like did you just magically like where did that come from then? And and then the reverse is also true. If the process is right but then the outcome is wrong then that's like okay then something else is wrong here and I'm not sure what >> I mean it it will change tests are interesting like now that I mention it because most of those tests are also AI generated and AI generated tests are you know it's like on average quite implementation specific and also the AI doesn't shy away from just changing the test to make their branch pass like to make their changes pass. Um, >> yeah. >> So, it's uh I don't think we found like the gold there this there's like if you go on X or like Twitter or whatever um you'll see a lot of like different claims and like hype people like oh use this approach use this like memory la and like everything is going to work like perfectly fine and like this how you write very high quality code with these agents. But uh I think like you still need that human touch somewhere. Um like I still for any branch that I try review I will look at all the changes like I'm not going to like just blindly you know rubber sand things like I I look at the changes I want I want to like test it out it's like you know if you make change to like whatever the assistant UI I want to like test it out test try out some use cases um and uh yeah I think that that is still like the that's currently probably the the biggest bottleneck of ing. >> Yeah. And you're talking about still wanting to be involved. That's the whole human in the loop thing. One approach to testing AI for quality or trusting it is human evaluation. So this can be like, you know, sometimes you get a prompt, you get a response from a from an assistant and it's like you thumbs up or thumbs down. That's like the most basic one, right? It is a human that clicks which of those thumbs matches their experience. But then there's also things like LMA arena.ai where um you a user a real human sends a prompt and then you get two answers back and from different agents and you don't know which one it is. It's not identified and you just say like is A better than B or are they both bad or are they both good? And that's useful but it's also not really scalable like you can't run that in a CI/CD pipeline you know. So useful but limited. >> Yeah, we can like on the eval side I I think they are all complimentary, right? So you got broadly speaking like two big sides of evals. Like in our case we talk about offline evals and online evals. Um I mean also we keep talking about evals but what is eval? Evals is essentially your testing to see if the AI agent is behaving like you want. Right? So a simple eval could be let's say I want to make sure like that my uh agent identifies itself correctly. So then your eval is you ask it as a user who are you and you expect uh the LM to answer I am graphana assistant that's a simple one it's much more complicated ones uh around like you know uh querying dashboarding uh like or like you know software engineering benchmarks are like very different as well but that's at at its core it's really about an eval is how do you figure out if this AI agent is any good and the reason we do that is because when we make changes to our agent, how do we know if it's better? And I think if you can answer those questions with confidence, then that gives you like a path in the longer run to actually improve your agent over time consistently. Um, and I think that's probably also like going to be more differentiating factor between the agents that will be actually useful in the long run versus the ones that are just like they look nice and they they can have like really good marketing, but then when it gets down to it, like you know, they will have reliability issues and if you just vibe change it, you're not going to get end up in a good place. But yeah, so that's kind of like the eval thing. And like we talked about offline and online and offline the so the difference I think is like online is your and we've we've like you know recently launched AI observability for agents um and online is really about that part right so online is about uh looking at how are real people interacting with your agent and then the online aspect is that it's online so it's like it's we're measuring how our agent that is deployed in production is doing against the conversations that it's having in in production. Um, so with these real users and that's quite valuable because it I mean a it you're get you get a score based on what real behavior looks like. Um, but these are a bit very hard to verify. Uh like typically we use LMS as a judge to judge like you know okay the user asked for this this is the whole transcript figure out if was that like a good answer or not. Um but again it is still you're relying on an LLM so it is not as robust so it's not quite verifiable as something you would do like offline and I think the offline counterpart of that uh is that offline is used more during development like it's during development it's kind of like in your CI/CD it's much more controlled and whereas like online is just you're scoring every conversation or like a sample of the conversations you're having in the real world offline is about you have this like golden set of evals uh that try to cover all of the behavior uh that it would happen when you were to deploy this. So you still need to have an idea of like what can my agent do and how will users interact with it. Um but that's kind of like what you're testing here. And the thing is like you're always testing the same set. So if you make changes you can actually see is it getting better at this set but then also dive deeper and see is it getting better at promql versus loql is it getting better or worse at dashboarding. Um, so I think that's that's the big difference. Uh, online really useful to have like a almost like a pulse check uh of is my agent behaving. It could also be that you know sometimes like entropic swap their models and like you see quality go down. That's something you can't really capture with offline. Um, so it's really useful for that. But online only tests the variant that is deployed, right? Even if you do AB testing, it would only test those two variants. Whereas offline, you can test all of your changes, your implementation, your entire code. uh and all your promps. >> Yeah, I think it it took me some time to translate these terms because the testing industry already has this and we just call it pre-pro and production testing. And so in prepro, it's good because it's all the way from like unit testing to system testing, system integration, which means you're testing and staging and dev and and pre-prod environments. Um, and it's good because you can really hammer the thing like you can you can change your scope, you can exercise absolutely everything. It's not going to go down, right? Or if it goes down, it doesn't matter. You have to be a bit more careful when it's in a production environment. But we still have things like synthetic monitoring or rum, real user monitoring. And that's sort of that's the same idea just with an AI flavor. >> Uh, yeah. Yeah, definitely. like on the on the online side, it's definitely much more almost like traditional observability in a sense. You're trying to observe a system. It just happens to be powered by AI. Uh there is this natural language and text component to it. Um and but at the end of the day, you still try to turn that into a number you can track over time, >> right? And then it's basically number goes up, number goes down, and then alert on it. Uh the offline aspect is uh I would say the closest analogy is probably integration test because you're trying to test the whole system. So it's like you give it you give it a task like a user prompt. you say who are you or I'm trying to figure out what my CPU usage is for service X then it runs the whole thing like the whole agent against like a a real environment and then you get something out and then separately from that you need to then uh grade that outcome essentially u so it's it's a bit closer to an integration test I would say >> okay yeah so we did an AI observability um episode actually and This was unfortunately without you because you were a graphonicon and we wanted to time it for for that time. Well, not just so that you were away, but also to to talk about it from from our side. So, I'll also leave a link to that if you want to go more into details because we actually like demo it and stuff. Um, and I I showed how I I used it as well. But maybe we could talk a bit more about offline evals because that's not something we've talked about at all. Um yes. Um so offline it's it's a bit of a beast. Um I would recommend like there's a good post by Entropic called Demystifying Evals. Um really uh it really goes into detail. Um I can share my screen up. I mean this is the post right. [clears throat] Um it's quite a lot. It talks about like the structure of an eval, but just to give you like I might make this a bit bigger. Um um when we talk about evals, this is not really like the split between offline or online in this case like this could also be online except you don't really have your environment stuff. You just have an agent that runs and your grading is just grading that that conversation. Um, but previously, so all the way back when we had like simple single turn LM calls, uh, I'm thinking I think one of the first ones that we launched was incident summary. So it would take all of your your prompt would essentially be here is all of the context of the incident. Can you summarize this? That would go through an LM and you would get a summary out. And then our grading could be things like make sure it's readable, make sure it captures all the details, make sure the length is uh appropriate, like it's not too long, it's not too short. Um, and that h that's how we would evaluate a single turn one. The good thing here is like you just have a prompt, like you just have text, you have an LLM, which is an API call. And again, this grading could either both be a mix of like hard code like testing checks uh and it could be LM as a judge. Um but it's just a single term. So this is quite relatively easy to um to to both test and set up. Uh whereas with agents they get a bit more complicated. Uh and the biggest complication is that you don't know like every time you run a task against an agent, it can perform the task in very different ways, right? It can take very different trajectories. Sometimes it will finish the task in five steps, sometimes it will finish it in seven steps. Um but because the agent is in this loop where it keeps executing until it's done. Um you have to I mean a it takes longer to test, right? So you write a task like in this case this is I think a coding one. Um your agent has a certain set of tools and I think like the tools and the agent this is roughly speaking the harness like now we also have things like skills as well but you can kind of put that in the tools bucket for now. Um and then the agent runs against some type of environment, right? So the environment in our case would be like it runs against a real graphana with data sources with a certain like you know state alert setup dashboarding and all that. Uh and then essentially the agent will just keep interacting with this environment using the tools to complete this task. Uh and then once it's done it will say I've done this or like if you ask like a data question like you know how much CPU is my service using it will answer with it will use the Prometheus tools and answer with this is how much it's using. Then we need to take the output of all of this all the steps it took uh the whole transcript and then grade that to verify given this task did it actually get it right right that's the core of it. The other thing that's quite nice in this one is like just kind of like an overview of all the different components. Uh but we're going to talk about graders or sometimes also called like verifiers. Um so you got like a suit suite of evaluations that are like a bunch of different tasks that run against this harness that run against the environment. Um and then you have an evaluation that comes out. >> Okay. So how do we do it like for for assistant? Uh you talked a little bit about the environment being everything to do with graphana which is like a super it's a very broad environment. There's a there's a lot of scope there. Um how do we do offline evals for assistant? >> Yeah. So we have um we have both I think like we have two things right. We have uh the public side of benchmarks that we released recently and then we've got our internal evals. Um they generally follow the same approach uh just the number of tasks are different and the some of like you know for internal eval we might be using uh different data sources. Um, roughly speaking, what we want to do, and this is still like a big challenge, is like we want to have an environment that is real, as real as possible, but the tension is like we want to have something that's very real, but we also want something that we can control. Like if we want to run a benchmark, we want to be able make sure that this benchmark is reproducible. If we can't reproduce it, it kind of means like you can't really fully trust the results because if I run it today and then somebody else from the team runs it tomorrow and they get a completely different score, then it's like which one was right? And so this is definitely has been uh one of the big challenges and like maybe if I like go back uh to like how we all started. So as with everybody like you know when we first started developing we just started with some manual tests right? So every time somebody would make a change, uh me, somebody else from team would go in and actually just chat with the assistant to see how it would do on like there were there were a few like was like I remember like listing the data sources, making sure that the tool actually works was like a very simple one just to see that everything is wired up correctly. Uh some CPU questions like showing like your logs and all that. Um so you would just do these types of tests manually. Um and that to be fair is not necessarily a bad thing. I think like you have to do something right. U maybe even without any type of setup. The easiest way to get started is just [clears throat] have like encode this like put this in a doc, put this in a script like here is our manual testing script and it's like bring up the app. This is the first prompt scenario. This is the second one. This is the third one. This is the fourth one. Um and then you just essentially you judge. you are the judge, the human judge that judges whether or not the assistant did the right thing. Um obviously you can kind of do this at a very small scale with like four or five scenarios. You can't do this as a human with like 500 scenarios. Um so from there we started looking into okay like how can we actually like run the whole thing and and see what we can do. Uh so I think the big the initial big challenge was really around the machinery um an agent and especially in our context uh the assistant especially is like it's a front-end assistant right some of the tools run in your browser uh that has some benefits security-wise off-wise it runs as you um but it makes testing a bit hard as well. So for example for the front end assistant we use playright because we want to make sure that uh we we are replicating the real behavior and the implementation fully as it is on the benchmarking side as well. So the benchmark just acts as like me in the browser and then it will like run and like go to the page and open up the sidebar and like put in the doc and then we do like some uh magic to get like all the telemetry out. Uh but yeah so it kind of does that and then so the machinery was about how do we test like a front- end plug-in that was like specific challenge the other challenge [clears throat] was environment what we do is we kind of have two uh approaches to it so we have uh both uh we're testing against like cloud instances um and this is mostly because for example for incidents it's just a lot of data so like um the machinery to like get that snapshot of that data out and like reload it in. Not that like you know we we have some things in place for that as well. Um but at least like to get started. >> Yeah. >> Sorry to interrupt but I uh before you you go on to the environment um I was just curious really did you consider using K6 instead of playright? Um interesting yes and no I would say because it like Ksix is also yeah Ksix is also like a tool that the assistant needs to use. Um, but we didn't our our goal is not to like load test. I think like it's really to hook into the plug-in. And the thing was we still needed to do some special things to get the messages, the the telemetry uh like things like the steps out uh out of the the front end plugin. So that's why we have like uh kind of like a playright harness that can run the front end. There's actually a browser module for Ksix that seeks to be is going towards feature parody with playright. >> I did not know that. So it might it might be interesting. Yeah. Yeah. >> Yeah. Just like dog food all the things. [laughter] >> Let's Yeah, I haven't considered that. No. So >> yeah, I'd love to work with you on that. >> Okay. Sorry. Go on with the environment. uh no what was I saying uh environment yeah so we have uh cloud environments that run against like cloud instance and then we can run this internal eval tool like we I haven't mentioned like you know but we call it lmspec it's kind of like a CLI tool uh but we can also run it against [clears throat] what we call like our local controlled environment and that one is quite similar to our open benchmark that we published and u maybe >> could we see that Actually, >> I can show you that one as well. Yeah. >> So, you've mentioned um Ollie Ollie bench is the benchmark that that Yas was talking about. >> Yeah. >> Uh and it was just announced at the last Graphana too. It was pretty fast. >> It was pretty fast. So, uh Oliven is our set like I I'll start with the task. We've got like a bunch of tasks around these like six different categories. Uh we got dashboarding task investigation some log task metrics and traces. Um and with Oliven we kind of a wanted to bring the approaches we're having internally um outwards and it was also like a learning opportunity because um for example we used the this framework called Harbor which is also used uh this one um this one is also used by the makers of like terminal bench and it was quite useful to see okay how do they approach benchmarking uh from their end so that's what we used for Olebench as well. And the nice thing about that is it separates the harness from the environment from the tasks which means you can run olebench against like the default harness that we have is a very simple small system prompt and the graphana mcp but uh I know like the folks that are working on the GCX or CLI for graphana uh they are using cloud code plus GCX and then testing it against Oben and then see how does GCX compare with MCP. Um so that's quite nice feature to have uh with this. Um and it's also useful to for example test how our MCP uh server is doing and like how we can improve our MCP uh graphon MCP as well. Um we've got the leaderboard. It shows you like a bunch of different models. We also have interesting like some open models like the quentry 6 plus uh and the Kim ones as well. So you get you get like an overview of how all these models are doing against each other. Yeah, they're getting good. >> Quen 3.6 is awesome. Not not, you know, still not in the top five, but still >> for an openw weight model, it's pretty good. >> And the interesting thing is like especially this, right? So, you know, we talked about like reliability. So, if you sort on like pass at tree, it looks a bit different than if you sort on pass hat tree. Uh, and the difference I kind of put it here as well. Um so the at tree it says every model it get it runs the same task three times right so it will be uh I think like if we take one of the tasks like let's say the metrics one uh it will be like you know check the service health and it will run that three times and if it gets it right one out of three times it gets a pass on that task right so that task will be a pass here um whereas pass hat three means to get one point you need to score like you need to get three out of three on every task. So, we run the same task three times and then we check, did it pass every time of those three times? Like you get three shots. Did you score all three shots or did you just happen to be lucky and you shoot three times and you only score once? And that's kind of like the difference between okay, the model has somewhere the rough capabilities to solve this task versus the model is actually reliably solving this task like every time that you ask it to solve this task. Um and that distinction is quite interesting because like you see like 10 20% differences between these two scores. Um yeah, just want to mention that. >> Yeah, that's awesome. And this is public, right? >> Yep. So, we have um I can go to the git repo here as well. So, we have a repo in git. Um we've got all of the tasks. You can see how they're defined in here as well. So, you can see like all the different things that we're using. And this I think is quite interesting. So, uh one thing that we're doing, you know, I talked about like LM as a judge. Um now we are running Olebench against this like synthetic environment. We are generating the data. So we have a Prometheus, a Loki, a tempo inside of this uh Docker setup >> that we also like generate the data uh so that every time you run the benchmark, it's the same data. But it also means like before we even run the benchmark, we we know what the answer should be because we control how the data is generated. Um and that means we can set up like certain like criteria to pass which is like it has to site this particular thing and we know it has to be that particular thing because we know what the uh data looks like. The other thing that we're doing is we're doing this thing called uh factbased uh rubrics. So in this case we are uh not just like giving the LM judge a piece of text but uh separately from what the agent does we the grading side also runs like certain checks on against the environment. So it can run queries get the result and say does the result of this known good query right this query that we know is the right answer does that match with what the agent said and this is quite important because sometimes these agents they fail in very nuanced subtle ways. So if you ask it things like like the CPU question, right, the simple one uh or like a latency question like what's the P95 latency for my API? And let's say it responds with 3 seconds or 3.1. How do you know if it's actually 3.1 or if it just made it up? Like how do you know if it's not if it's not 2.5 or 5.5 or something else? So these facts are trying to resolve that by uh running like a query that we know has the answer against this environment and then passing that through to our judge. And that way we can capture this like you know these small nuances and that this is kind of like where benchmark gets tricky because um most especially like the most of the good LLMs now they all look plausible enough right if you look at it from the outside >> most of the LLM output like the top tier models especially >> they all look plausible right but plausible does not mean >> that's the dangerous >> and that's the danger >> this is what I was saying with process versus outcome right like cuz it's they if you if they said if the result of that question was like -2 seconds then it's like okay it's clearly wrong or 500 seconds it's like probably not but it gives an answer that is plausible that's even more dangerous >> yeah and that is uh that happens a lot we had there was this um I remember to Italy we there was this one change that we made to I think the temper tool and it actually broke the spans So it the agent was not any any of the spans properly but it was still answering and interestingly it answer was not completely wrong because I think from the metadata it could kind of come up with oh the span should look like this. Uh but that was quite interesting where like even without the tool results being empty, it was still trying to answer that in a realistic enough fashion, >> but if you didn't test for it, you wouldn't even capture this at all. Like from just the answer from just the text, it looked realistic enough. >> I know an early version of assistant um I I said like, "Oh, I have this data source. Can you can you create this kind of a dashboard?" And it was like, "Okay, okay." and it created a dashboard. I was like, "Huh, this doesn't line up. I was just using this. There should be usage here and it it doesn't seem right." And then upon interrogation, it was like, "Oh, yeah. I didn't find that data source." So, I just used dummy data. Like, what? [laughter] >> Yeah, it u interestingly there was one like this was like when I was working on locally. I broke I broke all the tools. So none of the tools were going into the agent and it was still >> because we had some like rough instructions in the system prompt about which tools to use it would come up with some tool calls that it couldn't actually execute. >> Okay. So so this oy bench one um what is that that route because you said that there's a public one and there's an internal one. What is missing in Obench that you still needed like an internal? Um that's a good one. I think internal one I think this it's on two levels. Um I think one is that there are some scenarios that are kind of based on like some real conversations that we had for example with the assistant uh or that are based on uh some data we can do from our own internal testing like from our own internal environments let's say. So obviously anything that involves real data we can't put that put that in the public. Um so we can't have scenarios that are just based on public data. uh or sorry private data. So anything we do with olebench it is going to be limited to this like synthetic data generated type of uh setup and we can still do a lot with that and we still use that internally as well. The the way that I see it is like we got olive banks our public set but then our internal eval private set um is just a larger collection but the public set is still part of this as well. So internally, yeah, we want to test against Olly bench as well, >> right? Not just our internal emails. >> Um, so I think that's one of the differences and the other uh what was the other one? The other is probably because we have some very specific things we want to test about the assistant itself. Um, I'll give the example of Slack. So in Slack, we have certain things like we want the assistant to behave slightly differently in Slack. Um, there's also some formatting things that are different in Slack compared to the web. Like Slack has this has their own version of markdown that looks like markdown but it isn't quite markdown. And initially the initial versions of the Slack assistant had an issue properly like you know consistently writing Slack Markdown. U so we had to like write some tests around that. So some of these tests are very specific to Grafana assistant. So those would be in the private set but things that would be in the public set would be generally observability tasks like in the most gen generic general sense like it's what are people trying to do with observability ask questions about that data uh like you know do dashboarding like some basic graphana API type of checks uh that we need to make sure that the assistance uh the agents can call uh and the interesting thing is that I talked about bit about that harness so you know I talked about you can with GCX you can test MCP we can test our assistant we can test any other agent actually against Olly bench u and that kind of focuses our grading towards uh the outcome like we don't really care how the agent solved it every every one of these agents and harnesses have different tools they have different like ways of operating like cloud code is going to work differently than like open code and the MCP server for example um and We want to make sure that the benchmark is as independent from the how and the harness and it's really grading things based on okay what is the real answer. >> Yeah. So um I was also sorry I was playing around with the layout and [laughter] distracted. Um yeah I I also wanted to ask like what is you know you we talked about your your blog post already. What is the evaluation loop? Maybe we should actually define that. >> Um yes. So I think it starts with so at its core and we talked about this uh in in the graphonicon talk we also talk about I think it's out on YouTube now as well. Um so at its core it's really about we measure right we measure the current state and we look at how good is it now and where is it working where is it not working. Then we based on that we try to make some changes like we learn from all of these like you know hundreds thousands of transcripts and figure out okay what are some changes we want to make like maybe we want to like add a new tool change some tool instructions change the system prompt uh maybe fix some bugs with the implementation that are causing issues. We make the changes and then we measure again and then we make changes and then we measure again. We make changes we measure again. The thing is that this loop uh that is can be done by a coding agent. It's not us manually doing this. So the whole setup is that if your evals are defined well enough and robust enough so you got like really good ways to verify if the answer was correct and you can pass this signal like this clean feedback signal back to a coding agent. It can then make changes and then basically it can keep improving until it hits 100% on on the evals. Uh and then you get into okay now I need to like make harder tasks or increase the coverage. Um but that's the core idea of the the eval loop is that um at scale it's nearly impossible for one human to actually understand the whole system and make targeted improvements to it because it's not like you're not an agent is not programmed as code in the regular sense. It is coded like you know we code program with natural language and there is [clears throat] no clear relationship between the language and instructions you put in versus what the AI the LLM model is going to do and the only way to really test that and probe that is like you need to run like a bunch of these loops to figure out I made a change now I need to know what the impact of that change was and I can't do that with code you can write a test and actually do that with LMS you And is this evaluation loop um equally implemented and effective for both OLAB bench and and um LLM spec? >> Uh so it is implemented internally. OLSPEC is really about just the first part I would say like the Oben's goal is not to be in a while loop for agents. Yeah. Although >> we've indirectly have used it for GCX and MCP in that manner. uh but more manually I would say >> manually yeah >> uh but for example in lmspec we have this skill called lmspec improve and when you call it it will ask okay which agent do you want to improve you you select your agent and then it will just go uh and run measure improve measure improve um as well >> okay so maybe let's zoom out a little bit and and think like where what have you been working on now on on this whether it's Obench or LLM spec and then also where are you going like just directions no promises. >> Yeah. Um so I've had like a lot of work done on especially like the harnesses uh like you know I talked about this like playwright thing that's that's been quite painful. There was also there's also been like an interesting I think like historically if I would say there was like an LLM spec 1.0 then there was like a 2.0 revamp that was about how do you make it run faster and more robustly then we did Olly bench and then after Olly bench there was like quite a lot of learnings from ollybench that we put into what I would call like lmspec 3.0 So which is the the thing that I worked on after all bench and grafonicon. Um now I think like uh the the two big challenges I think one is I talked about like environment fidelity that is uh especially with all the different plugins that we have. So this is probably more of an issue for internal LLM spec like internal evals more than more so than olive bench. That's one. Uh the second big thing is like just the number of tasks and like how can we improve them both in quantity and quality right how can we cover more of graphana uh and in the same sense that Olly bench we weren't quite sure when we did Olly bench like is this going to be is this going to be a thing or is this a one-off thing that we're going to do or not uh but I think even like internally it's got like a lot of interest that uh I would say we're definitely doing like an version of that like we're gonna probably work on like a 2 ochen 2.0 or that type of stuff. Um so both for uh using that internally because like O punch obench has proven quite useful for the development of GCX and MCP uh the graphon MCP. So just having more harder tasks is like the big one going next. Um, and as part of that, one of the things I've been working on is like how can we get more people to contribute. So, and it just went like yesterday I was talking at the PM alliance like with the product managers on like how can we get product managers to write some of these evals like because evals kind of these tasks they also capture a bit like the desired behavior. So like for product managers is like you know when somebody asks about my product uh like how should these agents respond ideally? Yeah, it's almost like, you know, what I would love to see is some sort of uh kind of sub extension or I don't know what you want to call it. It's not really a plugin, but uh kind of like cuz I before before when Graphana acquired Ksix, I was actually on K6. So that's why I'm like Ks, how about Ksix? How did you try it? But um Ksix has also has extensions. So there's like there are graphana plugins but Ksix itself also has extensions and modules and and stuff and I would love to see that same sort of model applied to evaluators because you know we've gotten a lot of really cool community extensions and I think evaluators would be the same you know um just like I I did a talk in Madrid yesterday and I and I showed off a bit of like AI observability stuff and then the evaluator section and the questions were all like can we add this can we add that why is this not in the template it's like it could be you know that would be really awesome >> yeah and like closing it would be nice if we could do that across both oybench and like the internal ones like there's still some uh open questions on like how how can we make the those two work more nicely together uh there is a difference because like internally we're working in Golang whereas Oben It's based on Harbor, it's Python. Um, but the tasks themselves are kind of like language independent and and so are most of like the verifiers at least in design. Um, yeah, there was probably like a fourth thing and this is mostly more on the tasks like how do we kind of get to this like better golden set over time as well. Um, and that is really about how what we haven't like we've kind of done it as one-offs, but like we haven't quite had like automate the part where we can go from AI observability like real conversations and turn those into like verifiable tasks in our internal emails because like you know as the product moves on or as people's usage of the product moves on like your tasks that you have right now might not be relevant. anymore or they might not cover the whole spectrum. Like imagine you launch completely new product or if Ksix didn't exist and then now Ksix is in graphana and people are asking assistant questions about Ksix or interacting with Ksix uh true assistant that's going to look very different and if you don't cover that in your offline evals yet. So essentially that like bridging the gap between online and offline evals and doing that in a more automated way I think that would be like the fourth picking. Yeah, I also have some like a wish for Olly Bench isn't just really Olly Bench, but what it could be used for. Speaking totally with no internal knowledge. I'm not promising anything or whatever, but when I saw it, I was like, "Oh, this is a step. This could be a step towards um making assistant be b y bring your own model." Because one of the barriers to that is, you know, people will try assistant with their whatever crappy small tiny local LLM and and think, oh, assistant doesn't work or something and that's not true. It's just you need to use certain models for it. and having a standardized way to measure those and just presenting it to the user and being like, "Hey, you can do this, but here are where here's where your model falls short and here's where whatever Opus model is the latest is at on that leaderboard, you know, then they can just make an informed decision and we can say like you can still do this, but you know, this is what you're signing up for." Mhm. Um I I agree on the like so I I would go even beyond assistant. Um like we've got GCX which is quite like nice if you're in CL and use GCX CLI. Uh we've got MCP as well that people use. Um, and if you want to use your own models with GCX, with uh with MCP, we kind of want to make sure that this benchmark reflects, okay, if you're interacting with Grafana and on working on observability tasks, these are the best models that there are >> and these are the best open models that there are, right? So, good point. >> Kind of like gets to your point of like, you know, people can make a more informed decision. >> Yeah. Well, what about like not just for Aliben and LLM spec, but just evaluation in general? Where do you think we're still missing? We as an industry, not not just as a company. Um there is I mean it's was it yesterday or day before there was this new benchmark called deep su deep software engineering that was released and it is like a bit of controversy because it kind of flips the ranking quite a lot from the most popular software engineering benchmark so far uh where like you know Opus was always leading and in this benchmark it's GT 5.5 that is leading by quite a margin and even For example, I think 3.5 the new Gemini 35 flash is beating Opus 46 or something. >> Um, [snorts] and >> for what tasks? >> Uh, for like coding tasks, for engineering tasks. So there is still I think u there's still something to be said about the reliability of these benchmarks themselves. uh like how uh obviously models are like or the labs are using some of these benchmarks to both like test their models against sometimes even like tune their models against those benchmarks. >> Yeah. Uh so the better I think and this also to be fair also might be a good thing for olive bench because we I would want a lab to tune against olive bench for example so that we essentially we get free fine-tuning out of the box because now that olebench is in the open and if we keep working and improving obench and labs use olive bench to tune their models on observability tasks then like in the future these newer models like quen 3.7 or 3.8 eight or even the big labs, they're going to get better at using graphana and solving observability tasks. Um, so the end goal, I think, is that uh overall. >> Okay, thank you. We we just hit our time um here, but it it was super it was really informative and I'm so glad that we we found we both found time to do this. [laughter] It's been like rescheduled and stuff for various reasons. So, thank you so much. That that was a really great overview of all the evals that we're doing. >> Yeah. No, thanks for thanks for this as well. It's it's nice to sometimes like just stop uh getting out of our dayto-day and talk about stuff. >> Yeah. And if anyone is watching this, if you have any questions about evals, you can leave it in the comments below and I'm going to bug Yas to answer them. [laughter] Thanks [snorts] everyone for watching., Overview of AI Evaluation, Grafana — The Context Window, 01kw8162ev3tvcp7gqjw7k6r59.