What’s Your Telemetry Data Really Trying to Tell You

What’s Your Telemetry Data Really Trying to Tell You? ![rw-book-cover](https://img.youtube.com/vi/5IXDcywBy_0/maxresdefault.jpg) 🔗 **Source:** [What’s Your Telemetry Data Really Trying to Tell You?](https://www.youtube.com/watch?v=5IXDcywBy_0) > [!note]+ Document Notes > tags: #observability #telemetry #monitoring #automation #ai #microservices #tracing #metrics #logging > author: LeadDev > source: YouTube > reference: https://www.youtube.com/watch?v=5IXDcywBy_0 > date created: 2024-12-19 10:30 > > # 🏷️ Title: What's Your Telemetry Data Really Trying to Tell You? >  ## 🧰 Action Inbox - [H1] Implement top-down observability focusing on critical user journeys — start with what decisions the data will inform #observability - [H2] Separate telemetry signals by time horizon to avoid alert fatigue — 12:45 #monitoring - [H3] Embed observability into daily work rituals rather than treating as separate activity — 41:23 #culture ## 🧭 Table of Contents - [[#📝 Summary]] - [[#📖 Main Content]] - [[#🗒️ Notes]] - [[#📚 References]] - [[#🗣️ Paraphrase]] - [[#📰 Newsletter]] - [[#✅ Action Items]] - [[#🧪 Hypotheses]] - [[#📊 Methodology]] - [[#❓ Questions]] - [[#🔮 Future Research]] - [[#💡 Implications]] - [[#🧠 Learning Styles Assessment]] - [[#📊 Accessibility Assessment]] - [[#🌍 DEIB Assessment]] - [[#✅ Logical Validity]] - [[#🛡️ DARVO Analysis]] - [[#🎯 Conclusion]] ### 🔗 References - [Open Telemetry Project](https://opentelemetry.io) - [EBPF Auto Instrumentation](https://github.com/open-telemetry/opentelemetry-ebpf) ## 📝 Summary - Expert panel discussion on making telemetry data meaningful and actionable rather than overwhelming - Key insight: shift from "collect everything" to decision-driven observability focused on critical user journeys - Panelists advocate for top-down approach prioritizing what decisions data will inform - Major themes include avoiding alert fatigue, implementing traces strategically, and embedding observability in daily work - AI and automation should reduce cognitive load and provide structured insights, not just collect more data ## 📖 Main Content This panel discussion features Tani (Proper Science founder), Andre Sherroh (Cosley co-founder), and Anthony (BBC senior engineer) exploring how to make telemetry data actionable rather than overwhelming. The conversation challenges the common "instrument everything" approach, advocating instead for decision-driven observability. The panelists emphasize that good observability makes systems "legible" to those responsible for them. Rather than providing all possible answers, it should highlight imbalances and guide deeper investigation. This requires understanding critical user journeys and focusing instrumentation on those paths. A key framework emerges: metrics surface problems, traces reveal location and dependencies, logs provide specific debugging context. This hierarchy helps teams avoid drowning in data by following a structured investigation path from high-level signals to specific details. ## 🗒️ Notes #### ⏱️ [00:00](https://youtu.be/5IXDcywBy_0?t=0) — [H1] Panel introduction and observability fundamentals - Key idea: Telemetry encompasses logs, traces, and metrics as key observability components - Evidence: "Metrics are usually an indicator that there's a problem somewhere. Traces would do a similar job with an emphasis of what components or where exactly you're experiencing a problem" #### ⏱️ [03:15](https://youtu.be/5IXDcywBy_0?t=195) — [H2] Shifting from more data to better decision-making - Key idea: Focus on what you're trying to achieve rather than collecting everything - Evidence: "Shifting the emphasis from more telemetry to better learning by decision-making... what are we trying to achieve and then what can we reliably see decide and then do under real constraints" #### ⏱️ [05:23](https://youtu.be/5IXDcywBy_0?t=323) — [H2] Starting with proper instrumentation from day one - Key idea: Implement observability early rather than as an afterthought - Evidence: "I wanted to start with proper instrumentation... as opposed to having it as an afterthought as I often see people struggle like oh I wanted to have better observability but it's too late" #### ⏱️ [08:30](https://youtu.be/5IXDcywBy_0?t=510) — [H1] Top-down approach to avoid alert fatigue - Key idea: Focus on critical user journeys instead of instrumenting everything - Evidence: "Think about this problem top down. What are the critical user journeys? What are the critical paths critical data flows in your system?" #### ⏱️ [10:45](https://youtu.be/5IXDcywBy_0?t=645) — [H2] Separating signals by time horizon - Key idea: Different urgency levels prevent everything appearing urgent - Evidence: "Separating signals by time horizon... if you need something that's minutes versus things that need attention over and some triage... that prevents the everything is urgent type approach" #### ⏱️ [12:45](https://youtu.be/5IXDcywBy_0?t=765) — [H1] Decision-driven dashboard creation - Key idea: Only create dashboards that inform specific decisions - Evidence: "Every time we create a new dashboard he asked me and everybody else like what decision are you going to make based on this" #### ⏱️ [15:30](https://youtu.be/5IXDcywBy_0?t=930) — [H3] Embedding observability in work rituals - Key idea: Tools must integrate into daily work cadence to provide value - Evidence: "Whatever charts dashboards... have to live inside the rituals or the operating rhythms and the operating cadence the daily use" #### ⏱️ [18:15](https://youtu.be/5IXDcywBy_0?t=1095) — [H2] Context-appropriate data presentation - Key idea: Push information where people work rather than separate dashboards - Evidence: "As a developer I don't sit in front of dashboards... push information either into a slack channel or into my IDE" #### ⏱️ [22:30](https://youtu.be/5IXDcywBy_0?t=1350) — [H1] Value of distributed tracing - Key idea: Traces reveal service dependencies critical for microservices - Evidence: "Tracing as a signal gives me an insight into how these services communicate depend with each other... if you're running a monolith it's not important" #### ⏱️ [26:45](https://youtu.be/5IXDcywBy_0?t=1605) — [H3] Strategic approach to trace implementation - Key idea: Start with high-value user journeys rather than comprehensive tracing - Evidence: "Using some good engineering practice like thin slicing and find... the right user journeys to trace... highest value things where to start" #### ⏱️ [30:20](https://youtu.be/5IXDcywBy_0?t=1820) — [H2] Logging strategy: answer key questions quickly - Key idea: Focus on operational and audit questions rather than log volume - Evidence: "It's not about quantity but about whether we can answer key operational and audit questions quickly without drowning" #### ⏱️ [33:15](https://youtu.be/5IXDcywBy_0?t=1995) — [H2] Using traces to focus log analysis - Key idea: Traces help narrow down which logs to examine in context - Evidence: "Instead of looking at every single log all over the place I am in that context looking at the logs in the time window and in the context" #### ⏱️ [36:40](https://youtu.be/5IXDcywBy_0?t=2200) — [H3] Boundary conditions as logging focus areas - Key idea: Problems often occur at system boundaries and state changes - Evidence: "Places where you see failure or problems most happen... are in the boundaries boundary crossings" #### ⏱️ [39:30](https://youtu.be/5IXDcywBy_0?t=2370) — [H1] AI requires focused, structured data - Key idea: Large data volumes make AI ineffective; context and structure are crucial - Evidence: "Sending large volumes of data to any model is almost a non-starter... minimizing the amount of information that either a human or... an AI needs to consider" #### ⏱️ [43:45](https://youtu.be/5IXDcywBy_0?t=2625) — [H2] Good observability makes systems legible - Key idea: Observability should reveal imbalances and guide investigation - Evidence: "Good observability makes the system legible to the people responsible for it... see where things are out of balance and see where you can go dig deeper" ## 📚 References - [Open Telemetry Project](https://opentelemetry.io) - [EBPF Auto Instrumentation](https://github.com/open-telemetry/opentelemetry-ebpf) - [Proper Science](https://properscience.com) - [Cosley Platform](https://cosley.com) ## 🗣️ Paraphrase - Effective observability starts with understanding what decisions your data needs to inform, not collecting everything possible - A layered approach works best: metrics reveal problems exist, traces show where and how systems interact, logs provide debugging details - Automation and AI should reduce cognitive load by providing structured, contextual insights rather than just aggregating more data ## 📰 Newsletter **Making Telemetry Data Work for You: Key Insights from Industry Experts** Are your dashboards gathering digital dust? A recent panel of observability experts shared game-changing approaches to make telemetry truly valuable. **The Decision-First Approach**: Before creating any dashboard or alert, ask "What decision will this help me make?" This simple question eliminates noise and focuses on actionable insights. **The Three-Layer Strategy**: Use metrics to surface problems, traces to understand system dependencies, and logs for specific debugging context. This hierarchy prevents data overwhelm while ensuring thorough investigation. **Embed in Daily Work**: The most effective observability tools integrate into existing workflows—Slack notifications for developers, operational dashboards for on-call teams—rather than requiring separate interfaces. Ready to transform your monitoring from overwhelming to actionable? Start by identifying your three most critical user journeys and instrument those first. Resources: [Open Telemetry](https://opentelemetry.io), [Observability Best Practices](https://opentelemetry.io/docs/concepts/observability-primer/) ## ✅ Action Items #### DevRel - Developer education: Create workshop materials on decision-driven observability approach - Documentation: Develop guides for implementing traces in microservices architectures - Community engagement: Host office hours for observability implementation questions - Content strategy: Blog series on avoiding common telemetry pitfalls #### OSPO - Developer support: Provide consultation on open source observability tool selection - Community engagement: Contribute to Open Telemetry project documentation and examples - Strategic alignment: Define KPIs for effective observability program measurement #### Community - Contributors: Create first-issue programs for observability tool improvements - Maintainers: Develop templates for observability strategy documentation - Ecosystem: Coordinate cross-project observability standards adoption #### WordPress - Core: Implement performance monitoring for critical user registration/login flows - Core Performance: Add tracing to database query optimization efforts - Meta: Improve .org API observability and error reporting - CLI: Add instrumentation commands for common debugging scenarios - Hosting: Develop cross-host observability best practices checklist ## 🧪 Hypotheses - Teams using decision-driven observability spend 40% less time investigating false alarms - Implementing traces for top 3 user journeys provides 80% of distributed system debugging value - Embedding observability in daily workflows increases tool adoption by 60% - AI-powered incident analysis requires 90% less data when properly contextualized ### 📊 Methodology - Survey teams before/after implementing decision-first dashboard approach - Measure MTTR (Mean Time to Resolution) with and without strategic tracing - A/B test embedded vs. standalone observability interfaces - Compare AI model accuracy with full vs. contextualized telemetry datasets ### ❓ Questions - How do we balance comprehensive observability with cognitive load management? - What's the optimal ratio of automated vs. human-reviewed alerts for different system types? - How can semantic conventions improve AI-powered incident analysis effectiveness? ### 🔮 Future Research - Investigate correlation between observability strategy maturity and incident response effectiveness - Explore integration of business metrics with technical observability for better context - Study impact of observability-driven architecture decisions on system reliability ### 💡 Implications - Practice: Shifts observability from reactive monitoring to proactive system understanding - Policy: Organizations need governance around what gets instrumented and why - Security: Better tracing reveals attack vectors and system vulnerabilities more clearly - Education: Engineering curricula should include decision-driven observability principles - Funding: ROI calculations should factor in reduced cognitive load, not just tool costs ### 🧠 Learning Styles Assessment Reading level estimate: 12th grade (technical audience appropriate) #### 👀 Visual learners - Present: System architecture diagrams, data flow visualizations - Improve: Add decision trees for observability tool selection, alert priority matrices #### 🔊 Aural learners - Present: 45-minute panel discussion with expert commentary - Improve: Shorter 5-minute audio summaries of key concepts #### 🗣️ Verbal learners - Present: Clear definitions of metrics/traces/logs, practical analogies - Improve: Glossary of observability terms, comparison tables #### 🧩 Kinesthetic learners - Present: Step-by-step implementation examples - Improve: Hands-on labs for trace implementation, interactive dashboard building #### 🧠 Logical learners - Present: Clear hierarchy of observability signals, decision frameworks - Improve: Flowcharts for incident investigation, ROI calculation worksheets #### 🧑‍🤝‍🧑 Social learners - Present: Panel discussion format with multiple perspectives - Improve: Discussion prompts for team observability strategy sessions #### 🧘‍♀️ Solitary learners - Present: Individual reflection on current observability practices - Improve: Self-assessment checklists, personal learning path recommendations #### 🌿 Naturalistic learners - Present: System ecosystem thinking, boundary condition analysis - Improve: Environmental impact of data collection decisions ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Good heading structure, descriptive timestamps - Improve: Add ARIA labels for video controls, transcript with speaker identification #### Keyboard navigation - Present: Standard video player controls - Improve: Keyboard shortcuts for timestamp navigation #### Contrast and readability - Present: High contrast video player interface - Improve: Closed captions with customizable styling #### Text resizing and reflow - Present: Responsive video player - Improve: Ensure transcript text reflows properly at 200% zoom #### Media and captions - Present: Full transcript provided - Improve: Time-synced captions, downloadable transcript option #### Standards reference - Meets WCAG 2.2 AA for multimedia content with transcript ### 🌍 DEIB Assessment #### Cultural breadth - Present: Global technology challenges discussed (BBC, enterprise examples) - Improve: Include examples from different organizational sizes and regions #### Inclusive language - Present: Technical jargon explained, gender-neutral examples - Improve: Avoid assumptions about team structures and roles #### Neurodiversity support - Present: Clear section breaks, consistent terminology - Improve: Visual indicators for key concepts, summary bullet points #### Representation - Present: Diverse panelist backgrounds and experiences - Improve: Ensure examples don't assume specific organizational privileges #### Psychological safety - Present: Respectful discussion of common engineering challenges - Improve: Acknowledge different organizational constraints and resources ### ✅ Logical Validity #### Strong reasoning - Deductive strength: Decision-first approach logically leads to reduced noise (08:30) - Inductive strength: Multiple expert experiences support top-down methodology - Causal reasoning: Clear cause-effect between alert fatigue and unfocused instrumentation - Analogical strength: System boundary problems mirror organizational boundary challenges #### Fallacies - Appeal to authority: Heavy reliance on expert opinion without independent validation - Hasty generalization: Limited to enterprise/large organization perspectives #### Tone of voice - Overall tone: Collaborative and practical, focused on shared learning from experience ### 🛡️ DARVO Analysis --- ## Full Transcript [00:12] Hello everyone and a huge thank for [00:14] joining us today for a discussion on [00:16] what's your telemetry data really trying [00:18] to tell you. This is a leave dev online [00:21] panel discussion created in partnership [00:23] with uh cows. This discussion will last [00:27] roughly 45 minutes after which both [00:29] myself and the panelists will head over [00:30] to the lead dev slack panel to answer [00:32] some of your questions. [00:35] Um before we start uh could I have a [00:38] short introduction uh for my panelists? [00:41] So I'm going to start uh with you uh [00:43] Tani. [00:45] >> Sure. Hi. Um so I'm Tani. I'm founder of [00:49] uh proper science and basically just uh [00:52] helping organizations become more [00:55] effective. Um we deal with lots of [00:58] concepts around measurement and metrics [01:00] observability around the work system and [01:03] into the technology and um and my [01:07] background is also in in engineering and [01:10] computer science. So kind of came up [01:11] through uh different roles in [01:13] engineering and eventually looking at [01:16] the broader system of organizations [01:17] because I found there's lots of gnarly [01:19] interesting problems to solve there. [01:21] >> Cool. Nice one. Thank you very much. [01:24] >> Moving on to Andre. [01:26] >> Hi Andre Sherro. I'm a co-founder at [01:30] Coley. Cosley is a software company that [01:32] delivers a causal reasoning system. I'm [01:35] one of the engineers on the team. uh we [01:38] found it costly uh now more than three [01:40] years ago and and we're using it at [01:43] fairly large enterprises to help them to [01:45] automate identify and uh prevent [01:48] remediate and prevent problems in their [01:51] critical uh customerf facing digital [01:53] services. Prior to that uh I was a um [01:56] first engineer at another company called [01:59] Turonomic. Prior to that, I spent 10 [02:00] years at Goldman Sachs running their [02:02] systems and application management [02:04] engineering team. And I just find it so [02:06] much fun to help people to build better [02:09] operation, deliver better services and [02:12] uh u building on their observability to [02:15] actually make sense of out of all of [02:16] their data. So [02:18] glad to be here. [02:19] >> Yeah, that's pretty cool. I can sense [02:21] the passion all the way from here. And [02:23] uh and my name is Anthony. I am a senior [02:26] software engineer at the BBC. uh will be [02:29] currently your host. Um I've had [02:32] experiences uh with pretty much [02:35] everything related to a platform of some [02:37] sort, whether it's a data platform, [02:38] whether it's self-service, whether it's [02:40] experiments, whether it's audience, um [02:43] you name it. And I've kind of been [02:44] involved in this. Um I'm really pleased [02:47] to be here today. And the topic we're [02:49] going to tackle is drum roll or sorry [02:53] more specifically telemetry data. So [02:56] when we spoke about telemetry data, I [02:58] think what people what comes into [02:59] people's minds are like logs, traces and [03:02] the metrics of like everyday life and [03:04] all of which are like key components of [03:06] uh what we assess as to be the [03:09] observability of your engineering [03:11] product. In broad terms, metrics are [03:13] usually an indicator that there's a [03:14] problem somewhere. Traces uh would do a [03:17] similar job with an emphasis of what [03:19] components or where exactly you're [03:21] experiencing a problem. And last but [03:23] certainly those lists um logs uh they [03:26] often consult them in order to debug or [03:28] audit our services. But any of those [03:31] three activities which are part of the [03:33] telemetry data can quickly turn into a [03:36] hassle whether they are automated or not [03:38] and combined uh they can be incredibly [03:40] useful. However, there are many [03:42] pitfalls. Have you ever heard of death [03:44] by dashboards, alert fatigue, finding a [03:48] needle in a hay stack, and my favorite [03:50] as a junior developer when I looked at [03:52] my colleague and asked, "How do I access [03:53] the logs?" And he started by, "Let me [03:56] get you on the system. Do you have the [03:58] right uh do you have the rights to [04:00] actually do that?" And then we do you [04:03] know how to SSH into a box? But fear [04:06] not, today I have this exciting panel [04:08] with me and um we're going to start with [04:11] question one. So amazing panel, what is [04:14] the latest thing that you have done that [04:17] made an improvement or a difference in [04:19] your team company's telemetry data? Uh [04:22] for example, uh something very simple. I [04:24] implemented a log retention policy [04:26] across our cloud estates and I saved up [04:29] quite a bit of money and reduced noise [04:31] when aggregating logs. So who wants to [04:34] take this one? [04:39] >> I guess I could take this one. [04:41] >> Yeah, please. Yeah. [04:42] So not a very long answer to this one, [04:45] but I think [04:46] >> that's good. [04:46] >> Um the approach in terms of the the [04:50] something that have improved was [04:52] shifting the emphasis from more [04:53] telemetry to better learning by [04:56] decision- making better learning and [04:58] decision- making. So it's kind of um [05:03] change shifting the approach from [05:04] collect all the things to what are we [05:07] trying to achieve and then what can we [05:09] reliably see decide and then do under [05:12] real constraints. So I did that when in [05:15] one of my last engagements in the large [05:17] corporation with several leadership [05:18] teams helping them think through um what [05:21] are the right things to measure in their [05:23] in their tech environment. [05:25] >> Yeah, that's nice. That's really nice. I [05:27] think yeah, people often just jump in [05:29] without really knowing what they're [05:30] looking for. They just like they just [05:32] have this because we must have it, [05:34] right? It's observability, but could be [05:36] a bit more [05:39] refined. Um Andre [05:44] >> um well I think that uh one of the most [05:46] fun was as we were building our current [05:48] uh software offering uh we're offering [05:50] it as a SAS is that I wanted to start [05:53] with proper instrumentation that are [05:55] helping us to actually operate this SAS [05:58] as they they um um as a number of [06:00] customers grow and volume of data grows. [06:03] Uh so it was good to start with that [06:04] from day one as opposed to having it as [06:06] an off. often see people struggle like [06:09] oh I wanted to have better observability [06:11] but it's too late because I'm already in [06:12] business I don't have time to do this so [06:15] I'm just glad that we were able to do [06:16] this and continue to to u instrument all [06:20] the components in the way they interact [06:21] with each other so it's easy for me to [06:23] see things that are working or not [06:25] working in our s things that are not [06:26] working in our customers environment how [06:28] data is flowing through the system I'm [06:30] just really proud of what we built so [06:32] far to demo it to all of our other [06:34] customers like this is how you should do [06:36] Thanks. [06:38] >> Yeah. Yeah, you're absolutely right. I [06:39] think that too often people fall into [06:41] like the the pitfall of just saying, [06:42] "Oh, yeah, it's it's an afterthought or [06:44] I didn't think about it. Oh, it's a bit [06:46] late. I wish I'd known and stuff." And [06:48] and that is great that you know that you [06:50] two have this kind of approach that that [06:51] can make people think perhaps to to get [06:55] things um going earlier. Um, in our [06:58] quest to make monitoring meaningful and [07:01] relevant leveraging that telemetry data, [07:03] how do you, dear panelists, avoid alert [07:07] fatigue or worse, death by dashboard? [07:10] Andrew, I think you have something to [07:11] say about that. [07:12] >> Yeah, I think so. So, so I think that [07:15] what I often see as I work uh with a [07:18] number of customers over the past couple [07:19] of years is that people think about [07:20] like, let me just instrument everything, [07:23] collect everything, and then I'll figure [07:25] out what I'm going to do with this data. [07:27] And then it becomes all the things that [07:29] you mentioned about that by you know [07:31] dashboards. Uh so so what I am driving [07:34] internally as well as across our [07:35] customers is to think about this problem [07:37] top down. What are the critical user [07:39] journeys? What are the critical paths [07:41] critical data flows in your system? How [07:44] do you focus on instrumenting, observing [07:48] and exposing those things as opposed to [07:50] an army of data? And how do you also [07:52] minimize the data so that you only [07:54] collect things that are relevant to your [07:55] critical journey? Um, so really thinking [07:58] taking a top down approach to [07:59] observability rather than a bottoms up [08:01] approach. [08:02] >> Nice. Ti, do you have anything to add? [08:06] >> Yeah, one one other thing that popped to [08:08] mind was this idea of time horizon. So [08:11] separating signals by time horizon too. [08:15] So you know if you need something that's [08:18] um minutes you know to know something is [08:21] broken right away uh versus things that [08:24] need attention over and some triage and [08:27] some thought maybe trending information [08:30] and then [08:31] >> maybe improvement type data trend those [08:34] are more trend and systemic type fix [08:36] type things. So that prevents the [08:39] everything is urgent type [08:42] >> uh approach which is one of the things [08:45] that contributes to the fatigue. [08:47] >> Yeah, you're absolutely right. That is a [08:49] very good point like you know we tend to [08:50] look at everything just all at once or [08:52] or we don't we create an alert as [08:54] engineers but we don't really prioritize [08:56] those alerts. We just have to look at it [08:58] and then we just about where some [09:00] prioritization beforehand could actually [09:02] really help you just like focus on what [09:05] what needs to be looked at. Well, this [09:06] this just leads nicely to the the [09:08] question number three, which is more of [09:10] a work culture question, but like again, [09:13] often as engineers, we we build [09:15] dashboards like we build alert, you [09:17] know, um we we build them just because [09:21] it's something that we think it's a must [09:22] have into organization, but we don't [09:25] really look at them as much as we [09:26] should. Once it's being built, it kind [09:28] of like sits there. The alert sits [09:29] there. You know, it's there, you tested [09:30] it, it worked, but you don't really [09:32] iterate on it. And we're trying to [09:35] surface observability to stakeholders [09:36] without actually checking if they [09:38] understand the data and the logic that [09:40] we as engineers put on display. And I [09:42] think Tann you did uh we spoke earlier [09:45] and you did mention how you you managed [09:47] to bridge that gap into your approach. [09:50] >> Yeah. From [09:53] from a standpoint of thinking of that [09:56] the service and product that you're [09:58] providing. Mhm. [10:00] >> So in terms of [10:03] not always just trying to guess what [10:05] people need to see [10:06] >> and say hey here's here's what we've [10:08] created now it's going to solve all your [10:10] problems like and let me train you how [10:12] it works and all that kind of thing. So [10:16] liazing collaborating [10:19] with the stakeholders understanding what [10:22] are the decision- making that this data [10:24] is informing again time horizons can [10:27] come into play. Is this trend [10:29] information? Is this something's broken [10:32] right now and I need an alert kind of [10:34] information or is it something that's a [10:35] little bit more strategic long-term [10:37] improvement type? [10:39] Um what like what is what is this data [10:42] informing? Um because I remember reading [10:46] a white paper a long time ago that said [10:47] that um if if a met if some data or [10:51] measures are not informing some kind of [10:53] decision-m they could be waste. So that [10:55] could be that's one place to start. So [10:58] there's that collaboration of like what [11:00] is the context? What is the work? What [11:03] is the what what is infor what are these [11:05] this what is is this information [11:07] informing what is it supporting [11:10] um to what end? And then you know taking [11:12] a stab at that based on what you hear [11:14] maybe back briefing here's what I heard [11:16] here's what we understand here's a [11:17] sample and iterating through that in [11:20] collaboration with stakeholders. I think [11:22] that's really important. So almost like [11:23] kind of a product thinking a product [11:25] management um product development hat um [11:30] while while doing this uh is really [11:32] helpful I think necessary in order to [11:34] meeting the needs of those stakeholders. [11:37] >> Yeah. Yeah. I absolutely love like that [11:39] bit when you say um yeah if the data [11:41] doesn't serve a purpose why are we [11:42] keeping it why are we storing it? You [11:44] know Andre what do you think about that? [11:47] So um our my the founder of uh Kosley as [11:51] well as the founder of the previous [11:52] company Turbino make sure every time we [11:55] create a new dashboard he asked me and [11:58] everybody else like what decision are [12:00] you going to make based on this [12:02] >> and before you create the dashboard ask [12:04] yourself like what is it I'm actually [12:06] going to take what action I'm going to [12:07] take with this and it's just a really [12:09] good way to think about this like is [12:11] this going to help me make a decision [12:12] and if not don't even create action or [12:15] the report and uh it you know it stuck [12:18] with me. So I think about this every [12:20] time I you know do I collect this data? [12:22] What am I going to do with this? [12:24] >> I think it's a good way to help. [12:26] >> I've thought of one more thing to that [12:28] line is that the the whatever charts [12:31] dashboards [12:32] um their tools they have to live inside [12:35] the the rituals or the operating rhythms [12:38] and the operating cadence the daily use [12:41] the daily cadence of work. So that [12:44] context I think is really important part [12:46] of the thing that's really important [12:47] like what is how does work actually [12:49] happen and because these are kind of [12:52] going to slot in there versus just being [12:54] something you walk by and ignore. [12:56] Hopefully there's something that you're [12:57] actually using and providing value. [13:00] >> Yeah, that makes sense. Is there like um [13:02] do you reckon perhaps sometimes some [13:06] dashboards like um I remember setting up [13:08] a dashboard for for all of my um data [13:11] analysts that were using like um the [13:12] data warehouse I was working on and it [13:15] was all about like the trends like you [13:16] know the heat map of queries like when's [13:18] the best time to fire a query without [13:20] you know for like not competing with [13:22] resources CPU usage like a lot of like [13:24] um information is going how many [13:26] billions of rows are being scanned etc [13:28] and and it wasn't like a graphana kind [13:31] of a little bit complexish looking [13:33] dashboards. Do you think that sometimes [13:35] like the way you present that data can [13:37] affect how people are keen to engage [13:40] with it? [13:44] >> Oh, definitely. Uh I think that's part [13:47] of the feedback to to to to gather and [13:51] to take. And I think um sometimes [13:57] progressively elaborating is better than [14:00] starting with a lot and progressively [14:03] add in as needed. Uh is a good is a good [14:06] way to [14:09] >> hit them try to hit as close to the mark [14:11] as you can on what what uh what is [14:14] useful and what is easy to read and what [14:16] is engaging. [14:17] >> Yeah, fair point. Fair point. and you [14:19] Andre what's your most what do you think [14:22] your stakeholders find the most engaging [14:23] visually [14:25] >> so um I think that what T said is really [14:28] important to fit into how people operate [14:30] and how people work there are teams that [14:32] I'm working with that are actually [14:34] directly primarily responsible for [14:36] operations and they are sitting in front [14:37] of a a dashboard and and now integrating [14:40] into their dashboard is probably not [14:42] overwhelming them but relevant data into [14:44] their dashboards in the context of you [14:46] know I'm operating this application this [14:48] is an addition piece of information [14:49] that's useful in that context is useful [14:52] rather than like I create another [14:53] dashboard for you whenever you want to [14:55] go there and people will not um but but [14:58] then as a developer I don't sit in front [15:01] of dashboards um so do how do I uh push [15:05] information either into a slack channel [15:08] or into my IDE which then links me back [15:11] to okay to be able to know more about [15:14] this this is the link to either a piece [15:16] of information or to a dashboard if you [15:18] will um so so those are very different [15:21] use cases that they may lead you to the [15:23] same background information but in the [15:25] place where I live otherwise if nobody [15:28] looks at it it's not much use [15:32] yeah absolutely absolutely and and [15:34] talking about these types of information [15:37] um there's one that I personally [15:38] struggle with a little bit so I'm going [15:40] to play the devil's advocate here is is [15:42] traces I found in the past traces are [15:45] sometimes a bit time conssuming to [15:47] implement meant and by the way that's [15:50] the question number four and um for [15:52] those you know like me we don't really [15:54] believe I mean there is a use case for [15:56] it but I just feel like the effort [15:58] needed uh to add that to my telemetry uh [16:02] especially when I'm using lots of like [16:04] different components and I have um some [16:07] of his information given to me by like [16:08] let's say a cloud provider why would I [16:10] go through the trouble of adding it [16:11] myself could you could you convince me [16:13] otherwise Andre can you could you make [16:15] me use it [16:16] >> use it [16:16] >> absolutely Yeah. So, so, so um um I'm [16:21] obviously very interested and involved [16:24] in the open telemetry community. I'm one [16:25] of the contributors. One of the projects [16:28] that I'm working fairly closely with it [16:30] is the OBI project, EBPF based auto [16:33] instrumentation. And the point of this [16:35] is to reduce that burden so that you [16:37] don't feel overwhelmed by the amount of [16:39] work that you need to do to u instrument [16:42] your application but generating traces [16:44] for example. But the more important part [16:46] is what is the benefit? Why is this [16:48] important in the first place? Because [16:50] that's really an offset to okay do I [16:52] need to do something or or something. Uh [16:54] I think that as I am operating [16:56] developing and operating complex [16:58] distributed microservices [17:00] um tracing as a signal gives me an [17:04] insight into how these services [17:06] communicate depend with each other. Some [17:09] of those services may be bespoke [17:11] business logic that I wrote. Some of [17:13] those services may be cloud managed [17:16] services. [17:17] If I only monitor them independently [17:20] through logs and metrics, I don't know [17:22] who calls who, who depends on what. The [17:25] failure of this will cause errors or uh [17:28] latency uh someplace else. Traces are a [17:32] key uh signals to be able to understand [17:36] that dependency. um if you're running a [17:39] monolith it's not important but I think [17:41] nowadays an increasingly larger number [17:43] of applications and teams that are [17:45] arcting applications are developing [17:48] distributed microservices that are not [17:50] just [17:51] broken down into components but they're [17:53] actually data is flowing through AC [17:55] across different applications maybe [17:57] across different lines of business [17:59] relying on external data providers and [18:02] to understand the data flows between [18:04] across them and then based on that to be [18:07] to reason about the system behavior. [18:08] Traces are key and as long as we can [18:11] make it easy and cheap for people to [18:13] instrument the application, people will [18:15] see the benefit or realize the benefit. [18:19] >> Okay. Okay. I'm I'm partly convinced. [18:20] Johnny, do do you think I don't know how [18:22] do you feel? Do you join Andrea on this [18:24] quest for like implementing traces or [18:26] are you a bit like me just sitting on [18:28] the fence still wondering like, okay, [18:30] should we do it? Shall we not do it? [18:32] What would you recommend? [18:34] I think uh [18:38] I'm probably a little bit of both [18:41] depending on the scenario. by such a [18:43] simple [18:44] >> but what but what's on my mind as I'm [18:46] thinking when I've been thinking about [18:48] traces is and I'm thinking of like the [18:50] comp like the system systems thinking or [18:52] the system complex system thinking hat [18:54] is [18:56] um the the time consuming objection [18:59] which could be like more like the [19:00] pragmatic thinking like we need to get [19:03] things done and and and [19:06] you're doing this for the sake of more [19:08] like to generate learning which is a [19:10] more of an epistemic versus pragmat [19:12] pragmatic approach approach. So I was [19:14] thinking using some good engineering [19:15] practice like thin slicing and find you [19:18] know figuring out the right user [19:21] journeys to trace in through the system [19:23] with and to get where you need the key [19:25] dependencies like think through those [19:27] things I think what are those the high [19:28] high value highest value things where to [19:30] start with traces that I'm thinking of [19:33] those things to over where to overcome [19:35] the time just the time consuming [19:36] objection because [19:38] >> there's a like things that take too a [19:40] time often get objected to, but they're [19:42] often worth it because that time you [19:44] spend increases learning which saves you [19:46] time elsewhere. You got to pay the time [19:49] somewhere. [19:50] So, I was thinking like what could be [19:52] the analysis you put into what are the [19:54] traces to build and where to start and [19:57] that's kind of what came to mind. [20:00] >> Yeah, I can see that Android nodding and [20:02] uh and yeah, I'll have to lend to the [20:04] both of you like you are right. Yeah. [20:07] Okay. I will try and push my team to [20:09] implement some more traces now. [20:11] Thank you for that. [20:13] >> Happy to help. By the way, [20:15] >> if there is any open source project that [20:17] we could, you know, give advice on to [20:19] you and to anybody else in the audience. [20:21] By the way, I'm I'm really passionate [20:22] about making this easy and to Tion's [20:24] point, valuable. Um, you know, this is [20:28] regardless of the business that we are [20:29] in, I think it's it's going to help us [20:31] to understand our systems and think [20:33] about their behavior. [20:36] >> Yeah. Yeah. Absolutely. No, I mean, I'm [20:37] I'm all in favor. I think this is this [20:39] is a great invitation to to collaborate. [20:43] We're now going to move on to another [20:45] part of uh the um telemetry data which [20:49] are the logs. Very often the logs are [20:51] considered as is for me the backbone of [20:53] any engineering product because it can [20:55] tell you a very useful story either for [20:57] auditing or for debugging. But then [21:00] there's like so many questions around [21:02] them which I'm going to ask both of you [21:03] like you know how many logs or how much [21:05] logs is enough logs? uh how do you best [21:07] store and organize them? How do you [21:09] avoid the rush and door situation of [21:11] logs when like one logs from let's just [21:13] say an Apache airflow just takes you to [21:15] another logs with another pipeline that [21:16] was broken down somewhere else and then [21:18] to another logs because it was something [21:20] else broken down. So how how do we so [21:24] let's just sorry let me go back to that [21:25] first uh question that has how much logs [21:28] or how many logs is enough logs [21:30] according to either of you who wants to [21:32] go first [21:37] >> oh you're so polite but it's amazing [21:42] >> I don't know the answer to that question [21:44] uh how much long is it I don't know if [21:47] it's about quantity but about whether we [21:50] can answer key operational and audit [21:53] questions quickly without drowning. [21:57] >> So yeah, that's kind of what I'm [21:59] thinking. And then um the you know logs, [22:04] you know, metrics tell you where [22:07] something is [22:09] um in general generally tell you [22:12] something is wrong and logs give you [22:15] some local facts. And so just figuring [22:17] out um the right again using that [22:21] analysis the same you would do for like [22:24] traces analysis for what what the [22:26] logging needs to accomplish and and [22:28] therefore back into [22:31] what are the right things to collect and [22:32] that gives you um [22:36] uh that then you'll end up with with [22:39] what you need. I think um I think I [22:41] think that's that's what I've got right [22:43] now on this. [22:44] >> Yeah, of course. Of course. Andrew, do [22:46] you want to jump in? [22:46] >> Yeah. Yeah, absolutely. Exactly along [22:48] those lines. So, what we do today in our [22:51] own business and I'm promoting elsewhere [22:52] is that traces help you understand the [22:55] system behavior and to see where you [22:59] need to focus. Now once you say like [23:01] okay I need to focus on this service it [23:03] makes sense to drill down and say like [23:04] okay in this service that is apparently [23:07] causing all these degradations what are [23:09] the logs especially what are the error [23:12] and critical you know logs maybe some [23:14] warnings um that now understands [23:17] specifically what's happening within [23:18] that service. So I'm basically using the [23:22] understanding of the system by the way [23:24] through tracing to narrow where I need [23:26] to focus and then instead of looking at [23:29] every single log all over the place I am [23:31] in that context looking at the the the [23:35] logs in the time window and in the [23:37] context and especially the ones that are [23:39] critical to to then reason about it and [23:42] and what we're doing in our own uh [23:44] product is then to say that instead of [23:46] like let's say I want to apply some [23:48] generative TVI to automatically [23:51] remediate a problem if I were to send [23:53] all of my high volume logs to any model, [23:57] you know, it would be either expensive [23:59] or non feasible to begin with. But if I [24:01] can narrow it down to like this is [24:02] service, here's the five lines of logs [24:04] that I'm actually uh uh observed, I come [24:08] up with pretty good answers of like [24:09] okay, I think that you know improve uh [24:12] you know locking in your code or improve [24:14] the the memory management in your code [24:16] or avoid this null pointer exception in [24:19] your code and and I really only needed [24:21] like five lines of logs but I need to [24:23] know where to look and in what context [24:24] and when. [24:26] I see like a lot of people often like [24:29] you know describe telemetry data to me [24:31] as like you know like in that order like [24:32] metrics to surface something traces to [24:36] know where it is being surfaced and and [24:38] logs to actually as a combination of the [24:41] two above you've narrowed down to like [24:43] exactly what type of logs you need. So [24:45] are we in agreement that's this is the [24:46] way you should look at it like almost [24:48] like a like a ski slope or something. [24:51] >> Okay. One thing that came to mind from [24:53] an organizational perspective and I I'm [24:55] not sure maybe get you guys's opinion on [24:57] this is in an organizational perspective [25:01] when we're we're looking across a an [25:03] organizational system the places where [25:05] you see failure or problems most happen [25:09] or are in the boundaries boundary [25:11] crossings bound like the boundary [25:13] conditions and the boundary agreements [25:15] are what you need and I think that [25:16] mirrors in systems as well and that [25:18] might be a place as well for for logging [25:20] looking at boundary classes [25:22] uh crossing state change type areas, [25:25] places where you know that you can often [25:28] um have signs of and have have issues. [25:32] >> Yeah, totally. I I hear this almost in [25:34] every one of our customer conversations [25:37] that that I am depending on these [25:39] external services that are are either [25:41] completely outside of my business or [25:44] they are in my business but they're [25:45] operated by a different team that I [25:46] don't have any control over it. And it's [25:49] like [25:51] >> I mean I can't say that the person never [25:53] has a problem but often you know a good [25:55] chunk of their problems is caused by [25:57] something else. So, so to be able to [25:59] identify that this external service is [26:02] causing degradations in my service and [26:05] then to be able to say okay now I'm only [26:08] going to look at the client logs that [26:10] are interacting with that service is [26:12] helping me to see what do I need to do [26:14] is it something that I can improve or [26:17] can I point the third party provider or [26:20] the other lines of business other [26:21] application owner to say that like I [26:23] think that you need to do something [26:24] about this [26:26] >> this part of your service. Um, so and [26:29] and tion, you're absolutely right. I [26:30] think that not that it's obvious when [26:32] it's all under your control, but [26:34] crossing boundaries is clearly where [26:36] things become hairy. And I think with [26:38] microservices, distributed systems, data [26:41] being processed through multiple [26:43] providers is becoming more prevalent. [26:46] >> Yeah. [26:47] >> Hence, tracing is important. But in [26:50] general, ju just just just go back to it [26:53] like you find a way to sneak that one [26:54] in, you know. Absolutely. [26:58] No, you are absolutely right. I mean I [27:00] do see I'm going to take like the BBC [27:02] example of like more like of an internal [27:04] like um communications or sorry between [27:07] the logs and all those like metrics like [27:09] we have a lot of dependencies between [27:11] our products and obviously like for [27:14] example the I player front end team and [27:15] the I player backend team and the I [27:17] player data team like we all are [27:18] separate entities operating according to [27:22] like you know legacy kind of like [27:24] challenges and and yeah and sometimes we [27:27] are just like pulling our hair apart and [27:28] just thinking [27:29] Oh, you sending this information through [27:31] the logs, but we have like kind of like [27:33] no way of actually making use of it or [27:36] we have to go and surface some some [27:38] other problems to another dependent team [27:40] which they had no idea they've been [27:41] having these issues. It's um it's a bit [27:44] challenging but with good communication [27:47] and and with just like making sure you [27:49] iterate for the better uh there is [27:51] nothing you can solve. But we the the [27:54] way we might not use traces is because [27:56] we have such determinant ring fence [27:58] teams that it basically means that if [28:01] there is a problem in that area then we [28:03] know this team and we'll be looking at [28:04] it somehow and we don't need traces for [28:06] that respect but it just depends on how [28:09] how people are formulated. So how teams [28:12] are created within an organization. [28:17] >> Oh that was nice. Uh thank you for that [28:20] chat. Um now we're going to move on to [28:24] question number six. [28:27] Um right given the sheer volume of data [28:30] sources available um you know in terms [28:33] of like metrics log etc and all the data [28:35] in the world you can think of in the era [28:38] of big data uh we often see automation [28:41] as a tool uh being used in that field to [28:44] to gather and aggregate those data. But [28:48] to what end? So again like this this [28:50] question is more about like like in [28:52] testing you don't automate everything [28:54] you don't want to have 100% automation [28:55] coverage that's impossible same you [28:57] can't have 100% data gathering but so [29:00] how would you recommend using automation [29:02] in your telemetry approach and can the [29:05] rise of AI help in this setting do you [29:07] reckon? [29:09] >> Can I start with you Andre? [29:11] Yeah. So, um, two things. I think I kind [29:14] of alluded in my previous point that [29:17] >> sending large volumes of data to any [29:20] model is almost a non-starter. If I were [29:23] to take every one of my alerts and send [29:26] it as a prompt, uh, I would probably not [29:29] be able to figure out what's going on. I [29:30] get a half like 50% good hallucinations, [29:34] 50% not so good hallucinations, and it [29:37] would cost me a lot of money. So I think [29:38] that everything that that Tian said [29:40] earlier about you know what is the [29:42] context um you know what part of the [29:44] business flow this is and minimizing the [29:47] the the amount of information that [29:50] either a human or in the case in this [29:52] case an AI needs to consider and provide [29:56] that context around this so that instead [29:58] of just looking at like a single alert [29:59] you say like okay data is flowing [30:01] through these systems here is the metric [30:04] that triggered this here's a couple of [30:05] final logs I think that is actually both [30:09] a more focused smaller amount of data [30:13] that can be uh genuinely processed [30:15] through um uh a a model and to then [30:19] drive um uh automation possibly all the [30:23] way to creating a PR to fix a problem if [30:25] it also has access to the code has [30:28] access to a coding agent. Um, I'm I'm [30:30] very optimistic about this this flow [30:33] becoming increasingly more uh uh [30:36] automated. Uh, you could argue to what [30:39] extent you need a human in the loop, but [30:41] hopefully increasingly less so or maybe [30:44] decreasing less so, but but I think that [30:46] that that is helpful. One other key [30:48] point, I'm sorry I'm taking long um is [30:51] is that this what I think is also [30:54] important is that [30:56] >> to the extent that the data can be [30:58] better structured [31:00] semantically defined helps a lot in any [31:04] model to say that like okay this is [31:06] coming from a database this is coming [31:08] from message broker uh you know this is [31:12] a gRPC method that I'm invoking uh this [31:16] is the client this is the server um what [31:19] the open telemetry project is helping [31:22] with is to lay down some of the [31:24] foundations about these semantic [31:26] conventions that within a particular [31:28] context text helps you to say like okay [31:30] this is what I'm talking about this is [31:33] the type of the thing that I'm dealing [31:34] with which then again helps both a human [31:38] or a process or an AI agent to more uh [31:42] efficiently correctly uh automate [31:45] actions based on that information. So [31:48] I'm very passionate about you know [31:50] driving not just open telemetry semantic [31:52] conventions but in in driving a a a uh [31:56] definition a top down definition of of [31:58] what is the system that we are dealing [32:01] with and how the observative system the [32:02] observative signals are describing that [32:04] system semantically. [32:08] >> Wow [32:09] it's a that's a pretty good one. Um I [32:12] know it's hard to follow but Tani do you [32:13] have something good? [32:15] >> I don't think I can add too much to [32:17] that. I think I I would summarize from [32:20] the standpoint from uh [32:23] >> of at least just what I've thought about [32:25] on this topic in terms of automation [32:28] um is that the and it we've said this in [32:32] in various ways in the past you know [32:34] half hour or so [32:36] >> I don't think the automation should [32:37] exist again just to collect more [32:40] telemetry [32:41] >> so [32:42] >> it's it's going to I think it should be [32:45] a helper for shortening time to [32:47] orientation, protecting attention, [32:51] turning incidents into learning, keeping [32:54] that at the forefront is important for [32:55] any automation um that you create. Um [32:59] and the AI AI could then help as maybe a [33:01] sense making assistant in some respects. [33:04] So correlating some signals, maybe [33:06] summarizing what's changed, maybe [33:08] proposing queries or drafting [33:12] stakeholder updates kind of thing. But [33:14] it has to be grounded in evidence. it [33:16] has to keep humans accountable for [33:17] decisions. So I think um it's uh you [33:22] know wise use of the tools for [33:24] automation and for helping with analysis [33:27] kind of thing. [33:30] >> Nice. That's very nice. I mean what what [33:31] I usually say is that you know like if [33:33] you can't follow up on someone just [33:34] challenge them. So I'm going to [33:35] challenge Andre because there's nothing [33:37] else I can do at this point. You you've [33:38] covered it all up. is so according to [33:41] you, do you reckon like you know AI in [33:44] the process of like to you as well [33:45] Channy um AI in the process of like you [33:48] know like capturing those logs and if [33:50] you use them like you know you use like [33:51] labelled data like you were saying you [33:53] know um trying to refine it to save on [33:56] processing cost training your model etc. [33:59] Are you using it to then take some [34:01] actions when the problem arise or are [34:04] you using to maybe are you trying to [34:06] maybe look beyond that and offer like [34:08] you know what like a K cluster modeling [34:10] can offer which is something like [34:12] surfacing problems you didn't know you [34:14] had if that makes sense because that's [34:16] another way AI can look into things is [34:18] by finding patterns that a human [34:20] wouldn't find them. So Andre what you [34:24] want? [34:24] >> So so I I think that AI is a fairly [34:27] broad term. Um I think that has been [34:32] around for a while and uh one of the uh [34:36] more be more you know longer term aspect [34:40] of it is is clustering is to be able to [34:42] identify things that are are similar. Um [34:46] and I think that is really relevant to [34:48] reduce the volume of data or to find [34:51] things that are occurring uh you know [34:54] frequently um just to differentiate them [34:58] from a generative AI which just [35:01] inherently probably not the best tool to [35:03] reduce large volumes of data. It does [35:06] reduce data but not necessarily large [35:07] volumes of data. So I think if you [35:09] combine those AI techniques together um [35:12] in one hand you could narrow down that [35:14] this is the area that I want to focus [35:17] this is the you know cluster of problem [35:20] that I want to deal with this is the [35:22] relevant metrics and logs associated [35:25] with it and now how do I go about well [35:28] at least explaining what this means but [35:31] even I'm I'm going further typically is [35:34] to say that like okay how do I actually [35:35] propose a solution to that problem uh [35:39] preferably in the form of a a code [35:41] change. [35:42] >> Yeah, [35:42] >> I don't know if I want to propose an [35:44] architectural change, but at least in a [35:45] code change that to say like okay based [35:47] on what I'm observing, I think that this [35:49] is what is causing your failure and this [35:51] is how you fix it. Um and I think to [35:54] point is very important you at one point [35:56] you do want people to have trust and to [35:58] have oversight and a level of control uh [36:01] over this process. I just think that I'm [36:03] I'm very excited, especially over the [36:05] past couple of months, how that shifts [36:08] from being only be able to explain [36:10] simple things to actually being able to [36:12] come up with relatively uh high uh uh [36:16] you know, fidelity solutions to to [36:19] problems. Yet I I I don't just blindly [36:23] merge things that the PR that an AI [36:26] generated still [36:28] >> of course but you see an improvement in [36:29] accuracy of the answers and and Okay, [36:31] that's that's that's great. Yeah, [36:35] Tani, do you want to [36:38] conclude? [36:38] >> Nothing to add. [36:39] >> No. Okay, [36:40] >> good. [36:40] >> Fine. You know, you get mic drop in. [36:43] >> Yeah. [36:44] >> No, thank you. That was that was very [36:45] good. I I think we are now nearing the [36:49] last question. So I was hoping that you [36:52] know like we didn't have to worry so [36:54] much about time and I said oh this [36:56] question you're going to have to answer [36:57] in one sentence but because we have [36:58] plenty of time you can actually answer [37:00] it in more than one sentence. So um this [37:05] is uh for me a way to wrap up this but [37:07] also to give like a t a key sorry not a [37:10] t a key takeaway uh to to like the [37:13] people watching us and we're going to [37:14] watch uh this as being recorded but what [37:17] does good observability means to you and [37:21] what is your number one advice to [37:23] achieve it? So I'm going to start with [37:25] Tani. [37:27] What is good observability and your [37:29] number one advice to achieve it? number [37:31] one advice. Oh my goodness. Um [37:36] let's see. Okay, so um good [37:40] observability [37:41] makes [37:43] the system legible to the people [37:46] responsible for it. So allows you to [37:51] see things, not always see answers, but [37:53] see where things are out of balance and [37:56] see where you can go dig deeper. M [38:00] >> and so my number one advice is to embed [38:04] it in into the rituals and the ways of [38:06] working whether it's release reviews, [38:08] incident reviews, weekly health check, [38:10] whatever the whatever the work is that's [38:13] happening for the particular audience [38:15] and how they do things that that [38:18] observability is available where and [38:21] when they need it. So they aren't just [38:23] shelfware. [38:24] >> That's really good. I really like it. [38:26] It's another like powerful sentence for [38:28] you just said is like you know [38:29] observability is not about answers it's [38:30] about like you know imbalance is about [38:32] like you know really helping you [38:34] balancing out what you need to and yeah [38:36] I guess having as part of your daily [38:37] routine like your almost like your daily [38:40] care to your engineering product that's [38:43] yeah that's very good thank you very [38:44] much I really appreciate I hope you've [38:46] taken note to people because that's a [38:47] it's a good line to say to your manager [38:49] tomorrow at the standup you know [38:53] what's the second good line you could [38:54] say to that of uh engineers at standup. [38:57] >> Well, maybe this is going to be [38:59] overlapping a little bit, but I think [39:00] that um good observability leads to a [39:06] understanding of your system, [39:08] understanding of your system behavior. [39:10] Everything else is just data. um to the [39:13] extent that the observity information [39:17] allows you to learn and to understand [39:20] and to reason and to improve how your [39:23] system operates either as a first level [39:26] second level operation people to say [39:28] like okay I understand that this thing [39:30] is not working because this data is [39:32] flowing in because someone started a [39:34] marketing campaign and there's like [39:35] 2,000 new users that are pounding on our [39:38] site or or or there's a Super Bowl and [39:40] everybody's betting or like what [39:42] whatever The thing is is is rather than [39:44] looking at things independently of each [39:46] other to think about this as a system [39:48] and especially what I'm really [39:49] passionate about is as an architect to [39:51] be able to use this information almost [39:54] every day or at least every sprint and [39:56] say like okay nothing has broken but I [39:59] see how my system behaves. These are the [40:02] points of failures. these are the the [40:05] places where data going through my [40:07] system is being choked because I try to [40:10] you know push this into a postgress [40:11] database and that doesn't scale or like [40:15] how do I continuously be able to use [40:17] this information to improve both my [40:19] day-to-day operation as well as my week- [40:22] toeek monthtomon architecture. [40:25] >> Yeah, that's really good. like that [40:27] actually when you were talking it just [40:29] made me think of like a few instances [40:31] where the BBC like we didn't see things [40:33] coming like um one was like I think five [40:36] or six years ago like we had like [40:37] Strictly Come Dancing which is like a [40:39] very famous show where like um stars go [40:41] dancing and basically people members of [40:44] the public are asked to vote and in [40:46] order to vote they need to log in the [40:49] amount of password requests like you [40:50] know I've forgotten my password because [40:52] we got within 10 minute of the voting [40:54] window open was like in millions. I kid [40:57] you not. We got a million emails saying, [40:59] "Can you remind me of my password so [41:01] that people could actually vote." We [41:02] didn't see that one coming. And then the [41:04] other one was um again a user thing was [41:09] um Glastonbury, which is a huge huge [41:11] festival music. And at the end of [41:13] Glastonbury, there's a curfew. And um [41:16] and somehow at the end of the curfew, we [41:18] we we put everyone into the homepage of [41:21] our service. And so there was already [41:23] some other people on the homepage [41:24] service just like you know looking at [41:26] news or doing whatever. And then [41:27] suddenly we had like a huge amount of [41:29] like people watching the the live stream [41:31] just go onto the homepage and the [41:33] homepage crashed obviously because it [41:36] didn't crash. It just take five minutes [41:38] to recover like people were just loading [41:39] and stuff. So you are right being able [41:42] to know your user behavior. I mean the [41:44] external part that isn't the product [41:47] itself but what integrates with it is [41:49] equally as important as as what you have [41:51] in it. So, thank you for that wisdom. I [41:54] just very good. All right. Is there [41:58] anything either of you would like to add [42:00] before I call it a closure on this um on [42:03] this conversation? [42:06] >> This is fun. [42:07] >> Yeah. Yeah, it's good. Yeah. I think we [42:09] should just like, you know, stop [42:10] recording and just carrying this [42:11] somewhere like just the three of us. [42:15] Okay. Well, thank you very much [42:17] everyone. Uh thank you Tani. Thank you [42:19] Andre for your participation. Uh I am [42:22] now bringing this um conversation about [42:25] what's your telemetry data really trying [42:27] to tell you to an end. Uh it is being [42:29] recorded. Feel free to watch it in your [42:31] own time. And yeah, hopefully you'll be [42:33] able to shine from today or to or Monday [42:36] onwards uh when your when your new [42:39] higherups ask about what telemetry means [42:41] to you and observability. So take care [42:43] everyone. Bye. You [42:56] don't know.  📄 **Full text:** ![[Full Document Contents/Articles/What’s Your Telemetry Data Really Trying to Tell You|What’s Your Telemetry Data Really Trying to Tell You?, YouTube]] ## Highlights - experiences uh with pretty much everything related to a platform of some sort, whether it's a data platform, whether it's self-service, whether it's experiments, whether it's audience, um you name it. And I've kind of been involved in this. Um I'm really pleased to be here today. And the topic we're going to tackle is drum roll or sorry more specifically telemetry data. So when we spoke about telemetry data, I think what people what comes into people's minds are like logs, traces and ([View Highlight](https://read.readwise.io/read/01km99bkxfn48d5c5x0acqcrd5)) - Tags: 👻 ai highlighted - three activities which are part of the telemetry data can quickly turn into a hassle whether they are automated or not and combined uh they can be incredibly useful. However, there are many pitfalls. Have you ever heard of death by dashboards, alert fatigue, finding a needle in a hay stack, and my favorite as a junior developer when I looked at my colleague and asked, "How do I access the logs?" And he started by, "Let me get you on the system. Do you have the right uh do you have the rights to ([View Highlight](https://read.readwise.io/read/01km99bkxs692m2x8b0hncegwn)) - Tags: 👻 ai highlighted - customers grow and volume of data grows. Uh so it was good to start with that from day one as opposed to having it as an off. often see people struggle like oh I wanted to have better observability but it's too late because I'm already in business I don't have time to do this so I'm just glad that we were able to do this and continue to to u instrument all the components in the way they interact with each other so it's easy for me to see things that are working or not working in our s things that are not working in our customers environment how data is flowing through the system I'm ([View Highlight](https://read.readwise.io/read/01km99bkybxyypzav6w4w0e48g)) - Tags: 👻 ai highlighted - relevant leveraging that telemetry data, how do you, dear panelists, avoid alert fatigue or worse, death by dashboard? Andrew, I think you have something to say about that. >> Yeah, I think so. So, so I think that what I often see as I work uh with a number of customers over the past couple of years is that people think about like, let me just instrument everything, collect everything, and then I'll figure out what I'm going to do with this data. And then it becomes all the things that you mentioned about that by you know ([View Highlight](https://read.readwise.io/read/01km99bkz40j9nay63x492kybx)) - Tags: 👻 ai highlighted - >> So in terms of not always just trying to guess what people need to see >> and say hey here's here's what we've created now it's going to solve all your problems like and let me train you how it works and all that kind of thing. So liazing collaborating with the stakeholders understanding what are the decision- making that this data is informing again time horizons can come into play. Is this trend information? Is this something's broken ([View Highlight](https://read.readwise.io/read/01km99bkzv9s1q5tm2ay1se5dk)) - Tags: 👻 ai highlighted - to understand our systems and think about their behavior. >> Yeah. Yeah. Absolutely. No, I mean, I'm I'm all in favor. I think this is this is a great invitation to to collaborate. We're now going to move on to another part of uh the um telemetry data which are the logs. Very often the logs are considered as is for me the backbone of any engineering product because it can tell you a very useful story either for auditing or for debugging. But then ([View Highlight](https://read.readwise.io/read/01km99bm4bs3aygrbs6n6z9xbv)) - Tags: 👻 ai highlighted