What’s Driving the Rising Cost of Observability?  ## Metadata - Author: [[The New Stack Podcast]] - Full Title: What’s Driving the Rising Cost of Observability? - Category: #podcasts - URL: https://share.snipd.com/episode/fa136380-9a35-4284-b418-6f9335cc2790 ## Document Notes ## Highlights - **Flaws in Traditional Observability Tools** - Traditional logging, monitoring, and APM tools each have fatal flaws in the cloud-native era. - Logging lacks scalability, monitoring lacks flexibility, and APMs struggle with polyglot microservices. Transcript: Heather Joslin What are some of the most widely used types of observability and what are some of the biggest challenges with each? Christine Yen I think that I don't tend to think about observability in terms of types, but what I imagine you're reaching for is recognizing that the roots of observability lay in logging tools, Monitoring tools, and APM tools. Is that where you're going with that? Yes. Yes. Great. Well, when I think about those three distinct categories, which were how engineering teams for many years figured out what was happening with their software, I really think that each One sort of had or has a fatal flaw that makes them a challenge to use in today's cloud native, much more flexible, much larger scale world. If we look at logging tools, classic logging tools, they're incredibly flexible. Many engineers, myself included, grew up on being able to put whatever you want in a logline. However, loglines and that whole approach were really optimized for humans reading out the data with our eyeballs. And the scale of software that we're operating today, especially in the cloud, simply is no longer human scale. And using log lines as a substrate to answer questions often means that the experience using logging tools to answer questions about your software can be very slow or require a lot of Overhead and maintaining indices to try and eek some additional performance out of your logging data. Monitoring tools, in contrast, and again, these are traditional monitoring tools with lots of dashboards and metrics. I really see the fatal flaw there as optimizing for speed over flexibility, almost to the other end of the spectrum from logging tools. There, you can pull up a chart. You can capture and display metrics at impossibly large scale, but it becomes very hard to go, I've got this chart that shows me, let's say, latency over time in my system. Now show me this chart, but for this one endpoint, or show it to me for this one endpoint and this one user agent. Something super granular. And the reason that monitoring was so popular then and it's starting to creak now is that in this cloud-native world, I mean, containers and microservices have been the default way That we deploy software for the last, what, 10 years now? And a world where you had a small number of app servers that you were watching and an infrastructure that didn't change very much, optimizing for speed over flexibility made a lot of Sense. Today, where folks have, instead of one monolithic app on five app servers, you might have 50 microservices on top of 500 containers and then Kubernetes pods that are cycling through Those containers and restarting things. Application or your user experience, you have this explosion of complexity in your system that traditional monitoring tools struggle to be flexible enough to support, at least in A cost effective way. When I think about APM, the last category, I think APM's fatal flaw was trying to be too magical. Now, I grew up as an engineer using and loving New Relic. And APM tools in the era of Rails apps and convention over configuration and these very consistent ways of building software meant that those vendors were able to offer magical experiences. You drop their agent in, they'll just slurp up all the right telemetry that you might need to diagnose the 50 most common problems that a Rails app might run into. You log into the tool and it has all these magical out-of experiences that tell you you've got these problems. We detected them for you. And that made a lot of sense when there was this consistency in how lots of software is written. But like monitoring case, we're not in that world anymore. So many engineering teams these days have polyglot storage and microservices where different components are written in different languages and different frameworks. It's increasingly harder for a magical approach to cover the breadth that we see in modern infrastructure. ([Time 0:01:19](https://share.snipd.com/snip/b6465a71-c2b2-4046-8f60-1dfed55cf73b)) - **Why Observability Costs Are Rising** - Modern software scale and complexity drive up observability costs. - Traditional tools charge more for granular, flexible metrics needed by today's engineering practices. Transcript: Heather Joslin Yeah that's I mean it and it does seem like um speed is is very important especially now because because um things are so just like you said things are so distributed things are much more Complex and if you you have a problem, it can cascade much faster. Well, let's talk a little bit. You mentioned the cost. I mean, what do you think are some of, we're hearing a lot about, you know, we're in a period of, you know, a lot of global economic uncertainty. We're starting to hear more from organizations that are trying to contain costs. And part of that conversation is the cost of observability right now. What are some of the things that, in your view, inflate that cost? And where do you see that conversation heading? Christine Yen I think there are some, there's a lot of factors that contribute to inflating costs. Some are necessary facts of the world, like what I mentioned about how we're architecting software today. It is just now a fact that we're using lots of containers and Kubernetes pods, and there's all of this additional components in our infrastructure that used to not exist. So that is a driver of complexity. There's also a driver on the practice side, on how engineering teams today talk about their software differently than they may have in the past. When I think about trends in how engineering teams think about their system, I think about DevOps, platform engineering, and SRE. And I think of all of these trends as shaping where engineering teams focus, how they operate, and how they think about delivering a great service. Specifically, I've loved the discussion around SRE and especially the interest in SLOs, service level objectives, because there's such a clear emphasis with the rights of SRE on The experience of the end user. It's not enough for our infrastructure to be healthy. What matters is whether we are upholding our standards of delivering a great service to those end users. And because the conversation is moving in that direction, it means that attributes that engineering teams are now looking to describe their software in are things that include things Like user ID or shopping cart ID or maybe even the skew, pretend we're an e-commerce site, the skew of the item that is being touched by the software or being referenced by the software. There's a human and a process need for an increase in complexity of the language that we use and talking about our software and what's being impacted that is also contributing to an increase In complexity in the data, which contributes to an increase in complexity in the costs. And I hear, as a vendor, I talk to a lot of customers who are trying to adopt SRE practices and SLOs and are suffering because their observability tooling is punishing them from trying To track the things that matter to their business, trying to help them understand the customer impact or business impact of a given engineering change, and as a result, are pulling Back on the very data that they need to understand impact. And it hurts to watch because folks are so accustomed to working within the constraints of traditional logging, monitoring, and APM tools that often it takes some conversations to Open up their eyes to the idea that modern observability tools are actually built for this level of complexity and expressiveness in telemetry. Heather Joslin When you mentioned their tools, they're punishing them. In what way? In that there's just too much data or there's too many alerts? Christine Yen Yeah. I alluded to this when I was talking about monitoring tools. The way that monitoring tools, which, again, typically optimize for speed over flexibility, the way that they allow for the flexibility, if, say, an e-commerce site said it's actually Really important to be able to break down latency by merchant, not just by, you know, not by, you know, HTTP status code, of which there are very few. If a customer says, actually, it's really important to break down by this thing that I have hundreds of, the way that traditional monitoring tools tend to approach that problem is, Great, of course, you have 500 customers or 500 merchants on your platform. Of course, we can show you latency by merchant, but we're going to charge you for every one of those. And you start to have this explosion in, oh, well, you want that flexibility? Well, we'll charge you for it. You want custom metrics that capture latency, or whatever metric you're interested in, latency by merchant and skew, and whether there's a sale going or not, well, that'll cost you. And those sorts of, that is an instance because the tool, because that traditional approach of how monitoring tools are architected and expects to handle data, they pass those costs On to customers. ([Time 0:06:47](https://share.snipd.com/snip/079af1a7-8dc2-4979-91dc-22535aec6056)) - **Choose Modern Observability Tools** - Use flexible, cost-effective observability tools that handle complexity without punishing teams. - Avoid cutting essential data needed to track business impact despite cost worries. Transcript: Heather Joslin So so it's so it's that's part i mean that's part of it the mism between the older tools and the more complicated scaled-up systems. I mean, AI has definitely been a subplot in a lot of the stories we've reported on in the last couple of years. Where do you see AI presenting challenges for observability tools? Christine Yen Well, I think there's there. Okay. I think that there are more opportunities for observability tools than challenges. Let's talk about that. Well, when I think about, all right, this is going to be a broad ranging answer. So I'll try to bring you through it in a, in a structured way. I think generative AI has maybe like a threefold impact on observability tooling. There is the obvious, you know, every SaaS wants to use generative AI to improve our own experience for our users. And I'm really excited for Honeycomb. We've got some cool stuff brewing where we're, of course, using some of the latest technologies, latest models to make it easier for folks to understand how to use Honeycomb, to approximate Some of that magic that the traditional APM tools had, but being able to really learn and adapt to a given customer's data. In May of 2023, Honeycomb was one of the first companies to release in production a natural language query interface. And so instead of using our query builder to construct queries over their data, users could just come in and type in, you know, can you show me errors that users are encountering when They're trying to log in? We're going to translate that. So I think that piece, generative AI has as much a role for observability tools as any SaaS to just make the experience better. But given that we serve builders of software, the second side of how generative AI plays into the observability story is that so many software engineering teams out there right now Are being told, go put some generative AI into your product. Use this black box, figure out how to make it work. And I would love to be able to say that generative AI APIs are as stable the ones we know and love. You know, Twilio, Stripe. Oh, you know, here's this really predictable input, output, you know, standard expected behavior. But that's not how users are using generative AI. ([Time 0:13:03](https://share.snipd.com/snip/bdb7f7c9-adb6-4ccf-874b-9f947539e0ea)) - **AI's Dual Role in Observability** - Generative AI greatly enhances observability usability and problem-solving. - AI code generation adds unpredictability, increasing the need for flexible observability tools. Transcript: Heather Joslin Where do you see AI presenting challenges for observability tools? Christine Yen Well, I think there's there. Okay. I think that there are more opportunities for observability tools than challenges. Let's talk about that. Well, when I think about, all right, this is going to be a broad ranging answer. So I'll try to bring you through it in a, in a structured way. I think generative AI has maybe like a threefold impact on observability tooling. There is the obvious, you know, every SaaS wants to use generative AI to improve our own experience for our users. And I'm really excited for Honeycomb. We've got some cool stuff brewing where we're, of course, using some of the latest technologies, latest models to make it easier for folks to understand how to use Honeycomb, to approximate Some of that magic that the traditional APM tools had, but being able to really learn and adapt to a given customer's data. In May of 2023, Honeycomb was one of the first companies to release in production a natural language query interface. And so instead of using our query builder to construct queries over their data, users could just come in and type in, you know, can you show me errors that users are encountering when They're trying to log in? We're going to translate that. So I think that piece, generative AI has as much a role for observability tools as any SaaS to just make the experience better. But given that we serve builders of software, the second side of how generative AI plays into the observability story is that so many software engineering teams out there right now Are being told, go put some generative AI into your product. Use this black box, figure out how to make it work. And I would love to be able to say that generative AI APIs are as stable the ones we know and love. You know, Twilio, Stripe. Oh, you know, here's this really predictable input, output, you know, standard expected behavior. But that's not how users are using generative AI. That's not how this black box in particular works with our systems. Pulling generative AI can build these into a product mean you're introducing non-deterministic behavior. Each of these API calls is extremely parameterized. So there's so many different factors that go into different behaviors. And I think that the need for observability tooling that is flexible and fast becomes more and more important as these software engineering teams are trying to figure out often live What's happening with this generative AI feature why is it not behaving why is it hallucinating how can we get a handle on why it's hallucinating and how to, how to, how to guide it back Into reality. So that continues to be really exciting. The last piece is a little bit more of a challenge. Um, but it's, again, it's a, I think it's a little bit more of a challenge for the broader engineering industry where we are hearing more and more teams are going all in on co-pilot, you Know, entire software engineering teams are buying licenses for their teams to be able to have AI-assisted code be out there in the wild. And I think it is very exciting to have tools to help with code authorship. And we know that when, I mean, we started this conversation talking about how observability is for when your software doesn't do what you expect. And we believe that as these AI code generation tools are used more, there's going to be more code out there in the wild that we don't fully have, you know, we don't maybe understand quite As well or starts doing things that we didn't expect. And I think I look forward to the era, which is coming of AI code ownership. What does it look like to own the code that tool spat out. And I think that that will be, we're all going to have to navigate it together. And observability tools are, I think, are necessarily going to play a part in understanding, okay, this code is out there. What is it doing? And how do we bring it back in? ([Time 0:13:21](https://share.snipd.com/snip/280dee2d-6d6d-4384-bcec-a2e2895f98e4)) - **OpenTelemetry's Growing Importance** - OpenTelemetry (OTEL) is key to standardizing telemetry data and improving observability. - OTEL adoption fosters vendor ecosystem compatibility and better telemetry control. Transcript: Heather Joslin Yeah, absolutely. I wanted to touch on another big topic in observability, open telemetry. It's kind of poised to have a, potentially this can be a big year for adoption. ([Time 0:18:46](https://share.snipd.com/snip/d87a5f4a-c016-49d9-ac1d-49516a954380)) - **AI Tools Should Amplify Humans** - AI SRE and AI DevOps solutions are multiplying due to the difficulty of making sense of telemetry data. - Effective observability tools amplify human judgment rather than replace it with automation. Transcript: Christine Yen There has been a predictable explosion in the number of AI SRE or AI DevOps engineer companies. And it is interesting because on one hand, trying to make sense of telemetry and trying to understand, trying to decipher what some paper trail is telling you about your software and How to resolve it. That is one of the just hard problems of building reliable software today. So absolutely understandable that there's a lot of smart founders and VC dollars chasing that problem. And from our experience sort of being in this world for a little bit now, there's a lot of context that doesn't yet exist in the telemetry itself. There's a lot of context that a human brings to distinguish between a DDoS and a load test that might look the same in telemetry. Our approach has always been to focus on building mecha suits instead of robots. How can we amplify the context and the judgment of the humans on a team, point them in the right direction, allow the humans to do what humans are great at, which is, hmm, that looks weird, And allow the machines to do what the machines are good at, which is crunch through a ton of data and offer you options. I think this will be really interesting to keep an eye on. We're certainly thinking through how we can leverage AI to be a little bit more of that teammate. But I think there's going to be a huge variety of approaches over the next few years that we're all going to be experimenting with together. ([Time 0:22:16](https://share.snipd.com/snip/f25c6183-5cb7-42fa-bb50-af09deaa8302))