How Can We Solve Observability's Data Capture and Spending Problem?  ## Metadata - Author: [[The New Stack Podcast]] - Full Title: How Can We Solve Observability's Data Capture and Spending Problem? - Category: #podcasts - URL: https://share.snipd.com/episode/427819cd-269c-438e-b4e5-25cda2c8c55d ## Document Notes ## Highlights - **Observability Is Continuously Evolving** - Observability expectations change continuously as new tech and releases appear. - Jacob Yackenovich says tools must adapt weekly to shifting production realities. Transcript: Jacob Yackenovich A great question. The state of observability, I wish there was a standard answer, but what we are seeing on a constant basis, to your earlier point, is that answer changes by about 2% every week. Where you thought you had a good understanding of that definition and the set of use cases that the industry expects from observability tools to be able to perform. And a new technology has announced or a new release like we've seen at KubeCon this week with the most recent release of Kubernetes being able to now handle AI workloads that much more Effectively and efficiently is redefining then what do you do in production and then how do you observe the circumstances that you need to see to answer why systems are behaving the Way that they're behaving as these new technologies, new principles, new cultures and processes get added into the mix of operations. ([Time 0:01:43](https://share.snipd.com/snip/b2bce4c2-6ea7-4659-80f4-8a846c3693bd)) - **Two Patterns Of AI Failures** - AI introduces two failure patterns: add-on and blocking components with different impacts. - Jacob Yackenovich says blocking AI failures can effectively take an application down and harm user trust. Transcript: Jacob Yackenovich The key is I'm taking advantage of the latest and greatest technology to go faster and do more in my business and facilitate my end users that much more effectively and efficiently. The primary case that we see today is where technologists are adding what we call a blocking, excuse me, not a blocking style, but an add-on style. So I'm adding in a new service or experience to my existing application workload. That's perhaps a chat bot that helps the end user interact with the application. If they have a question or answer that they want to get addressed from the data set that's inside the application, they can do so by asking an English-based text question and get a response, But at the end of it, they can still conduct the business of the application. If there's an error or an unresponsiveness to the generative AI component of the application, that's okay, the customer can still do what they came to do. There are other conversations I mentioned a moment ago, called a blocking style, where customers and industries are adding in generative AI capabilities to check for false information That's coming in the application, or fraud details, and in those circumstances, well now I've injected a microservice that is generating content but it's part of the overall workflow Of the job to be done in the given application. Now when that circumstance becomes in terms of the response be it toxic or hallucinatory or biased or just wrong from the end user who's interfacing with that application, the application Is down. Yeah, correct. To give a silly example, if I'm a flight reservation system and I have an application for booking a flight, say, from Madrid to London, and I get a response back from booking a flight, And suddenly I get a recipe for a banana blueberry pie. Which happens. Regardless of what that might sound delicious, the question of the user says, well, why did I get a recipe for food when I'm trying to book a flight? And I got a question whether or not I even need to conduct my business with that company. I'll go with a different company to book my flight because this is so erroneous. What else is wrong? So what we're seeing in this era is not just latency, responsiveness, throughputs and errors as a golden signal, but an emergence of accuracy as a fifth golden signal, into the representation Of, is a application healthy, performant, and responsive? ([Time 0:03:15](https://share.snipd.com/snip/31f6e234-b3c1-4d59-bd48-5670cdb69603)) - **Accuracy As A Golden Signal** - Accuracy becomes a new 'fifth golden signal' for AI-driven content generation. - Jacob Yackenovich argues we must monitor content accuracy alongside latency, errors, and throughput. Transcript: Jacob Yackenovich What else is wrong? So what we're seeing in this era is not just latency, responsiveness, throughputs and errors as a golden signal, but an emergence of accuracy as a fifth golden signal, into the representation Of, is a application healthy, performant, and responsive? Is the content itself accurate from the end user's perspective. B. Cameron Gain And how does that happen? Is that the user's responsibility? No. Jacob Yackenovich Or that's the, how would a observability platform gauge that, or these signals, and what kind of signal would it be? There are several different areas that we're seeing in terms of how to approach it. One of them is to evaluate the prompts, the inputs that the user of the application are providing to the application, and the responses that we're getting back. So being able first to evaluate what is those answers back, are there good quality responses that are provided, are they hallucinatory, are they toxic, is there bias in the actual responses? And then gauging those from an overall perspective of, is the application providing that information that the user is expecting, or are there things that we need to get and raise alerts To the operations teams and the AI engineers that are participating in that conversation to say, I need to retrain my model, I need to improve what I'm doing, or maybe I need to rethink Which particular model I'm using altogether in terms of providing the value that my customers are expecting from my given application. B. Cameron Gain Because I'm thinking with traditional, with the workloads, the AI, so-called AI workloads, you are using, you're still going to use metrics, logs, and traces. Yes. But this is an additional signal, as you said, and I'm wondering though, I mean, in some respects, it's, you're looking at observability of observability, in the sense of where you Have the AI from the LLM, giving you solutions, but then you have to have another signal or another way of verifying these solutions to make sure they're not an hallucination or erroneous Or causing problems. Jacob Yackenovich Correct, metrics, events, logs, telemetry data are still critical to understand the overall health and performance of the application, but in the case of adding artificial intelligence Workloads into the application ecosystem, now you've got a technology that is generating content that's never been generated before. That's when accuracy comes in, in my opinion, as a fifth golden signal into this conversation. Frankly, accuracy has always been there, but if you think about it, because the end user interacts with the application, they provide a fact. It's linear. They get a fact, and that's it. Sure, there might be a couple of things, or maybe a person's name is spelled incorrectly, but those are error edge cases. The primary circumstances, facts go in, facts come out, job is complete on the application. But now, I have a portion of the application that is generating content that has never been generated before. So it's more than just understanding is the microservice that represents the AI component of the ecosystem responsive, latent, is there the right throughput, is there any errors In the actual construct of the microservice itself as an entity, but it's also now inspecting what's going into those AI workloads. How is the model processing that information and what's coming back to the end user through that overall application workflow? ([Time 0:05:30](https://share.snipd.com/snip/b2a2d79f-cf0f-407a-9401-1b3fe0824d96)) - **Inspect Prompts And Responses** - Evaluate both prompts and responses to detect hallucinations, toxicity, or bias. - Alert AI engineers to retrain models or change models when content quality drops. Transcript: Jacob Yackenovich One of them is to evaluate the prompts, the inputs that the user of the application are providing to the application, and the responses that we're getting back. So being able first to evaluate what is those answers back, are there good quality responses that are provided, are they hallucinatory, are they toxic, is there bias in the actual responses? And then gauging those from an overall perspective of, is the application providing that information that the user is expecting, or are there things that we need to get and raise alerts To the operations teams and the AI engineers that are participating in that conversation to say, I need to retrain my model, I need to improve what I'm doing, or maybe I need to rethink Which particular model I'm using altogether in terms of providing the value that my customers are expecting from my given application. ([Time 0:06:08](https://share.snipd.com/snip/d3110226-94f7-400d-987a-1cf5f0c0e033)) - **Capture Full Per-Second Telemetry** - Collect 100% of telemetry at per-second granularity to avoid missing transient anomalies. - Jacob Yackenovich warns sampling or polling risks losing life-cycle events and key signals. Transcript: Jacob Yackenovich Absolutely, the keys that we look at from a observability perspective is first understanding at the speed in which the business is changing and the interaction speeds in which the Users are interacting with the application, you have to collect everything. Okay. And what do I mean by everything? Yeah, that's good. I mean by collecting the metrics, events, logs, and telemetry data at per second speed, and capture 100% of the telemetry data that's happening from the end user to the application, Throughout the entire application ecosystem, and back. Why? There are, as we're at KubeCon today, seeing a continuously increasing quantity of parts of those applications that are taking advantage of the latest and greatest technologies. And that's not just replacing the entire application, that's adding additional microservices, or adding additional cloud-native technologies to the existing way the business Performs the tasks that it needs to do. But what ends up happening is you end up with a combined set of technology in an application that is heritage technology along with cloud native technology in the same application. And so in those circumstances, you have entities that might never have any type of change or maybe on a quarterly or monthly basis, you might also have instances of elements that will Have their elements constructed, the task performed, and their elements deconstructed, never to exist again, in the time it takes me to finish this sentence. And in those circumstances, if you're not collecting the data about what the application is doing at per second granularity and at 100% capture, you end up missing a life cycle, two, Three, multiple of what was actually occurring inside the application ecosystem. ([Time 0:10:13](https://share.snipd.com/snip/e407dcc7-b856-412d-831b-2e9969a6678b)) - **Hybrid Stacks Create Visibility Gaps** - Hybrid apps mix legacy and ephemeral cloud-native components, causing visibility gaps. - Missing short-lived components obscures root causes during incidents, Jacob Yackenovich says. Transcript: Jacob Yackenovich But what ends up happening is you end up with a combined set of technology in an application that is heritage technology along with cloud native technology in the same application. And so in those circumstances, you have entities that might never have any type of change or maybe on a quarterly or monthly basis, you might also have instances of elements that will Have their elements constructed, the task performed, and their elements deconstructed, never to exist again, in the time it takes me to finish this sentence. And in those circumstances, if you're not collecting the data about what the application is doing at per second granularity and at 100% capture, you end up missing a life cycle, two, Three, multiple of what was actually occurring inside the application ecosystem. You'll miss those peaks and valleys that give you those key signals about something that is erroneous and or anomalous in nature because you're polling or you're doing a derivative Or sampling and then you miss those particular circumstances. Then it becomes the proverbial white rabbit to chase an ops because you don't have a good quality foundation of the information that actually took place. ([Time 0:11:10](https://share.snipd.com/snip/c67f7a94-d5da-4664-96b1-b69fefcba445)) - **Prefer Predictable Observability Pricing** - Avoid forcing limited visibility for budget reasons; prefer predictable pricing for observability. - Jacob Yackenovich recommends fixed-price models so teams can innovate without surprise bills. Transcript: Jacob Yackenovich Of the approaches that we take from an observability perspective at IBM is to understand you should not have to be forced to turn a blind eye into your operations ecosystem because the Observability price is too expensive. You're in the business of making money. You have a quantity of applications that are facilitating what your business is trying to do. It should not be a judgment call to say, oh, I can only afford certain observability levels for certain parts of my business. Fair enough, yes. Especially in circumstances today where all it takes is a smart business leader and a site reliability engineer or a data scientist to say, hey, I have a great idea. Let's take these particular individual applications and make an additional set of solutions and combine a new value that were these two individual applications together. And that afternoon, now you have a brand new application that absolutely no one knew about, but they were treated as two distinct things. If one of those was not under the level of observability to find out what's going on and why systems are behaving the way that they're behaving, now operations is in the dark on what is A brand new and potentially critical area because the business made that decision to create that value for a reason, make more money. Yeah. So it's a key conversation. We see you need the entire set of use cases, regardless of the number of users, and regardless of the number of use cases of the job to be done as an observability product. This should not be, ah, well it's another dollar here, or five dollars here, or you have a large quantity of a Bram and a certain host, it's an extra amount of money, and it ends up becoming 10, 12% of your annual recurring revenue in terms of the bill. Or more. Yeah. It's an area that requires a re-analysis, and the right technology and the right partner to be able to help you succeed at the level of granularity and intelligence and understanding That you need in operations to figure that most important question out and observability, why? Why are systems exhibiting the behaviors they're exhibiting? So the solution then should not be exorbitantly expensive. ([Time 0:13:15](https://share.snipd.com/snip/42e5be7d-bae3-4343-9f9b-8940913e4891)) - **Use Analytics To Prioritize By Business Impact** - Let analytics surface headlines so engineers don't sift raw telemetry for answers. - Jacob Yackenovich says observability should prioritize incidents by business impact, not opinion. Transcript: Jacob Yackenovich We see a mixture of culture and tools and process all combined into this conversation. Part of it is you want to make sure the skilled engineers and operations owners are interfacing with the truth of what's going on in the environment with the right level of analytics, So they're not having to sift through all of the data. They're provided the insights based on what the analytics are present in the observability tool. Sure, you can go look in the flame graph and see the particular transaction calls that are made on a particular application and users, singular transaction, and that's always valuable. Getting down to the lines of code analysis to see what actually took place and why a particular transaction was anomalous or erroneous in perspective. At the same time, our point of view is, a business leader should never need to get to that level of specificity and granularity. They're going to trust the smart people to look at the insights that are coming from an analytic engine of giving you the headlines. These are the things that you need to pay attention to as an operations leader. Where we see the business leader then saying is, it's great that I can rely on an observability tool to trust that my operations teams are looking at the right things, the right errors, The right circumstances, and dealing with the right impact that I have to my business. But I don't think of my business as one application or eight microservices. I think of it as a service that I provide to my end users. And I want to know, is my service healthy performance from the end user's perspective, and if there is a problem, are my operations teams not working with their intellectual property Between their ears, a judgment call of their fiefdom, or they think this particular area is more important, no, they're working on particular issues, be it, P3s or P2s, in the context Of the relative impact that circumstance has to the overall business. Yes, okay. That's what I want to see as a business leader is, are my people, that are really smart and really good at what they're doing, are they working on the things that are the most significant Or most important in the way that I run my business, and then is the observability solution. ([Time 0:17:54](https://share.snipd.com/snip/f191975e-52f0-4a35-baa3-d2b996931191)) - **Let Leaders Define Service Context** - Let business leaders declare service definitions and time-based scopes for monitoring. - Use those declarations to auto-prioritize incidents against declared business impact. Transcript: Jacob Yackenovich Usually that follows the sun, be it through a season or an hour of the day or a week of the year kinds of characteristics. And then being able to aggregate that information so that I can see that perspective. And then as a return on that investment that you're making into an observability tool, have the observability tool itself automatically prioritize orprioritize the issues or incidents That are under investigation based on the relative impact that the circumstance has to the business. ([Time 0:20:47](https://share.snipd.com/snip/9a73aac4-194a-439d-bb45-d6ee818823df))