OpenTelemetry: What’s New With the 2nd Biggest CNCF Project?  ## Metadata - Author: [[The New Stack Podcast]] - Full Title: OpenTelemetry: What’s New With the 2nd Biggest CNCF Project? - Category: #podcasts - URL: https://share.snipd.com/episode/86ee5e4d-8342-4b52-a2d9-228dd6f4decc ## Document Notes ## Highlights - **Challenges of Multiple Signals** - OpenTelemetry faces the challenge of supporting various signals like tracing, logging, metrics, and profiling. - Each signal presents unique complexities for documentation and management. Transcript: Alex Williams So OpenTelemetry has come a long, long way. But the pace of innovation has just been unrelenting for decades, really. And it's no different now. And it seems like you guys have a really good idea on how to think about tracing. But logging and metrics brings a whole different set of challenges. Morgan McLean And profiling to come and, you know, that list will continue to grow. ([Time 0:06:47](https://share.snipd.com/snip/61e4b582-1e5d-44a8-916c-026b76b792c7)) - **Simplify OpenTelemetry Setup** - Simplify OpenTelemetry's setup and configuration to improve user experience. - Automate processes to reduce manual configuration and make it as easy to use as proprietary agents. Transcript: Alex Williams Agreed. And I'm curious on your approach because, I mean, you can read around and say, oh, it looks like, you know, OpenTelemetry does need to do some improvements in logging and metrics and Such, you know, and, oh, OpenTelemetry, you know, has documentation problems. It's not different than a lot of other you know projects out there but i'm just curious in your approach you know to tackle those types of issues i would actually say like getting started Morgan McLean Out of all those just like getting into it in the first place like starting to use it is probably the biggest challenge and that's a function of documentation that's a function of some Features and design decisions we've made as well as other things i mean to me there's like two approaches there's the actual like very concrete specific things that we're doing right And then there's the overall structure of the project i'll talk about the second one first like i i agree with your assertion like to me like open source project management in terms of Like how the organization is structured and who's working on what and things like that is not really radically different than running a project in a company or really in any kind of environment. It's all just human project management at the end of the day. The biggest difference, I guess, is that in open source, there's no sort of direct management chain. The people on the hotel governance committee can't go instruct someone to go build something or do something fundamentally everyone can vote with their fee but it's a pretty healthy Organization in hotel the the culture and attitudes and things are strong enough that when you know when we have applied some structure to the project like there's a technical committee A governance committee um there's of the work is divided into these, what we call special interest groups within OpenTelemetry that focus on things like the collector, which has a SIG, or Java, which has a SIG, and so on. I think that structure has actually worked out relatively well, especially for a project of this size. We've maintained the velocity of the project, but as you mentioned, we've added support for metrics, and then logs, and now for profiles, and various other things that i at least i'm Curious what outsiders think but i at least am satisfied that the the overall structure of the project is not slowing things down and that includes things like documentation right You asked about documentation like yeah could the docs be better absolutely uh for an open source project of the docs, pretty good. I, you know, I would, I would argue yes. Like they're not the most perfect thing I've ever seen, but compared to most open source software I use, they're incredibly good. And we've been making major investments the last two years in documentation quality. There's lots of people in the community have spent a lot of very productive time on that. So that's the project structure but i think the more interesting part of this question is how like what actual specific things are we doing in the project to address this documentation Is one so yes there's a lot of work going on that um it's fairly sort of self-evident i think if you're regularly using those docs how the quality is improved the other one though is just Making it easy to use without having to configure things. The biggest complaint that I've seen, and even I've had myself with using OpenTelemetry, is that there's just a lot of parts to it. If I have a Java application running on a Linux VM, I need to go install the OpenTelemetry Collector onto that VM. I need to go configure it. That might be a lot of configuration I'm doing. It's all this really long YAML statements to go capture the right data from the host or any third-party apps. Now I need to decide if I want to have Java Auto Instrumentation or Java SDK. Then I need to go deploy that. And then I need to go configure that. It's not really turnkey. Like all the parts are there that I need, it's not a trivial thing to get started with and so there's a lot of work that's been going on like we've been doing it's so much pluck that's that's Going to get donated some pluck this year um and there's other work that's gone to the community for the hotel operator and other things like that to totally change the dynamic there Where it's going to still use all those components but now the experience is I go and put the OTEL collector or the operator and I apply it to my infrastructure and I'm done because it just Does all the rest for me automatically. And if I want to tune things, I can, but for like 90 something percent of use cases, I don't have to touch it at all. And so I think as you see OTEL enable more and more of those scenarios, all of which wouldn't have been possible like two years ago because we just didn't have the underlying components That they needed. But now those components are there as we go and enable that, I think that's going to totally change the game, right? Because now it becomes just as easy to use as a proprietary agent of old, and yet it includes all of the things that you want that your older agents wouldn't have had. And it's still tunable, it's still open source, you're not wedded to a specific vendor, you can send the data wherever you want, it customizes as you see fit. And you really get the best of all possibilities with that. So I think, I think as you see more and more improvements get made to open telemetry that just make it automatic, or as it gets built in to more platforms and runtimes just out of the box, You're going to see, I mean, it's already rapid adoption, but you will see that accelerate even further because the, the few sort of reasons of the few points of frustration that people May have had with it evaporate. ([Time 0:07:37](https://share.snipd.com/snip/802a52a3-4790-430b-87d5-c304f67d7ff1)) - **Profiling in OpenTelemetry** - Profiling provides deep insights into application performance, including CPU and memory consumption of functions. - OpenTelemetry's profiler allows correlation with traces and other signals, which is currently lacking in most tools. Transcript: Alex Williams Okay. That's interesting. When you think about that, you think about that, you know, focus that you're putting on, like the profiler, for instance, you mentioned the profiler, perhaps you could help, you know, Us understand, you know, what you're doing with the profiler and how that reflects, you're trying to do this year. Yeah. Morgan McLean So the broader context here is, and you were touching on this earlier, is open telemetry supports many different types of signals. So the first one was tracing that we added. And tracing was, I think, important because there was no real standard in the industry for that. There was open tracing and open census, but those literally became open telemetry. And so if you wanted tracing support that wasn't just a part of a product, you had to go use open telemetry. We added application metrics and system metrics after that a few years ago. We added logging, I think, just over a year ago to open telemetry. And those are the three main signals that people use. But profiling is one that I work on directly, as do many, many other people in the OTEL community. And profiling is the fourth major signal that's getting added. Profiling is somewhat unique amongst all these signals. Traces, metrics, infrastructure, they give you insight into the performance of a, sorry, trace metrics logs give you insight into the performance of a particular piece of infrastructure, Or maybe the performance of an application that's running on an infrastructure. With profiling, you're actually getting insight into the performance, like a level deeper into the application itself and the actual functions or methods inside of that application. You can see their memory consumption. You can see how they call each other. You can see their CPU consumption. You can see their sort of wall time for each of them, how long they take. This is very, very potent. And it's potent in a place where I think most people haven't really taken advantage of profiling in the past, right? If you're a client application developer, if you work on video games, for example, right? Like I was video game dev early in my career. You're pretty used to profiling, right? Like it's really important. I just have to ask a question. Yeah. Alex Williams Have you read the book Tomorrow and Tomorrow and Tomorrow? Have you read that? I have not. Oh, you got to read it. Okay. Is it about code profiling? It's about, no, it's about video game development. Oh, okay, cool. Yeah. No, it's a novel, but it's so engaging. It's so good. Morgan McLean Yeah. I did a... I did this internship, but I was at BioWare working on Mass Effect, and we were struggling at various points to get it to run on certain types of hardware. And this is a common thing across all video game developers. Video games, as people probably can appreciate, are very performance intensive. And so for games, and really for any client software there's a variety of profilers available yeah there's ones built into like visual studio or whatever but if you're writing cloud Software historically there has been no sort of really good equivalent of that that works at scale in your production environment um there were some efforts at google when i was there I launched their sort of profiler that external customers could use, and some big customers use it, but uptake was relatively tepid just because it was kind of a standalone thing. And there's many other profilers that have launched us that we've won. At Splunk, most observability vendors have them, but they're all somewhat niche. I am very excited for profiling to arrive in OpenTe uh just because like the power that it can give you in cost savings and insights at extreme scale is immense like unless you've actually Used one of these in a really big really expensive production environment it's not really possible to grok how easily you can use one of these and look at it and be like wow like look at This horribly inefficient code that i wrote that i thought was a trivial thing that i never would have thought of optimizing because i didn't think it was important and golly it's burning Like 20 of our aggregate cpu like tens of millions of dollars a month like the power of that is immense and yet profilers have remained somewhat niche with profiling being added to open Telemetry i'm excited both because i think it'll break it out of that niche profiling code profiling become much more common and perhaps even more importantly it'll be used for more Than just cost savings it you know with open telemetry the big benefit of having all of these signals in one big project is that you can correlate them. And so now if I am inspecting traces or calls across services that are slow and that data is captured from traces, I can look at the profiles that are attached to those specific instances, And I can understand what was making that service slow. That sounds like a thing you'd expect from every observability tool, yet many of the most of them struggle with it in most instances. Once they have this data, it's powerful. ([Time 0:12:51](https://share.snipd.com/snip/88254f30-9910-4028-9584-97a10cf037e2)) - **Universal Profiler** - A universal profiler's key feature is the ability to correlate data with other signals, like traces. - This enables pinpointing the root cause of performance issues by connecting slow services to specific code segments. Transcript: Alex Williams So what does a more universal profiler provide? What are its characteristics? What are its feature sets that distinguish it from the niche variety? I think it's the second thing that I was touching on. Morgan McLean It's really the ability to properly correlate that data, right? Most profiling tools, and I'm sure someone's going to listen to this and come in and say like, hey, Morgan, you know, I'm going to use this one and it's well integrated. But I would argue very strongly that most profiling tools in the market right now, open source or otherwise, generally gives you a deep view of function or application performance In isolation. So they give you that production view. You can see it across all your instances. It's great. But unless you're on a big drive for reducing compute spend, which is a very valid thing to do, but unless you have that very specific scenario, the utility is somewhat limited because If you've ever used one, you'll know you're just shown a giant flame graph for the service. You can see all the function calls and how they call each other. And then on the x-axis, you can see the CPU memory consumption of each. What you're actually lacking is, you know, you're, say, a big bank or something and a customer opens the banking app and they try to transfer money and it takes forever the developer Or ops person has been tasked with tracking down why this is broken today with their combination of of tracing or apm and profiling tools they can get on one side they can isolate which Service might have caused that to be really really slow but jumping from this service was slow to, well, what happened to my code? Where was it blocked? Like what was going on when it was being slow, remains relatively challenging. And that's because those signals are not particularly well correlated today. Right? That's the power is that you can now have those hops where we're guided by the tool, or I mean, the tool just tells you it has the, at least the information it needs to say, well, this was Slow because of this actual piece of the code and you should go fix that. ([Time 0:17:34](https://share.snipd.com/snip/8cadb7b2-8632-4722-bf3f-b3bf3375fad2)) - **OpenTelemetry in Various Industries** - OpenTelemetry can be used to capture telemetry data in various industries, including automobiles. - The integration of OpenTelemetry with natural language processing and AI is an evolving area. Transcript: Alex Williams Okay. So then when you're talking about profilers, you're talking about, you know, the hard work that's been done in tracing and login metrics. Now we're you know looking more towards you know a much higher degree of scale yeah and we're seeing that in all kinds of you know all kinds of ways uh you know automobiles any machine you Morgan McLean Can think of has telemetry data right and i can tell you there's some automobiles from two brands that i know of that have open telemetry in the car captured data. Alex Williams And so then you also, though, and have, you know, all kinds of new data being purposed in all kinds of ways through natural language processing, for instance. Yes. And I'm just curious on how that plays into your work, especially as Kubernetes changes quite a bit and is adapting to GPUs and stuff. And there's a whole new kind of level of data that's coming through that. So I'm just curious on how you're thinking. How do you map this? What are some of the outcomes you'd like to see when you look back? ([Time 0:21:02](https://share.snipd.com/snip/1e5d6918-e9f6-4550-8fde-8cb739de39b4)) - **OpenTelemetry and AI** - OpenTelemetry's structured data is well-suited for AI-based analytics in observability. - This structured data allows for better insights compared to messy log-based systems. Transcript: Morgan McLean And I guess you're sort of asking, and correct me if I'm wrong, but you're really asking about this sort of conversion to nexus of AI and open telemetry, right? And there's numerous different dimensions to this. Probably the most relevant one is, you know certainly if you're looking at the observability business or you like to use observability tools and you want to see how they're going to Get better is how does the data that otel generates is that you know can we now apply deep learning can we apply new types of analytics to that data uh and to me the answer is overwhelmingly Yes right like you know anyone who's used a fairly advanced observability tool that has this like topology maps and entities and things like that in the last few years has probably sat Down and thought like gee the computer can probably a sufficiently advanced like ai-based analytic system can probably provide deeper insights to this and yes that's what like many Observability companies, including Splunk, are working very hard on. To make those systems useful, you need the data that's going into them to be well structured. And so I think OpenTelemetry is probably better positioned than any data source in the industry to provide that because OpenTelemetry actually defines like what a service is what A host is what a you know certain type of thing is that it's capturing data from or or how certain interactions work right because it has specification that says like this is an http request And it has these fields and you know you can trivially determine the difference between a span for an http request versus one for gRPC versus one for a database request and so on. Because we've done the really hard work of defining those semantic inventions in OpenTelemetry, it means that your analytic systems, whether they be traditional ones or future AI-based Ones, can actually define, or not define, can extract insights from that data very, very easily. I would contrast that to more log-based analytic systems where the data tends to be very messy. That means that your signal-to ratio is relatively low. Well, whether it's a human or an AI looking at this data, that's going to be more challenging, just fundamentally. And so I think OTEL is, in fact, I would argue almost a prerequisite for people having real success with AI-based observability analytics. There's other aspects of the story as well. Like we commonly get asked, like, hey, if I'm doing AI training, like I'm actually doing this and I want to improve my infrastructure costs or improve my performance, should I use OTEL? Can it provide me the data I need? And like, yes, there's a whole bunch of people working on that. It's already being used in various places. But to me, it's the AI insights based off the highly structured OTEL data. That's the real long term, the really interesting long term story here. Alex Williams Yeah. And, you know, when we talk about signals, too, there's this whole new set of golden signals, aren't there, that you have to be thinking of when you're thinking of like latency of LLM Responses or token usage? Morgan McLean It's fundamentally, I think the signals stay roughly the same, right? Latency, error rates, throughputs, things like that. But the amount of data processing and these days, the lack of insight that a lot of firms have into why was the latency poor, right? Why did I get errors on these things? That tends to be even more opaque with these systems because you don't fundamentally understand how parts of them work. Like, yeah, it's doing a bunch of matrix multiplication, things like that, but it's far more inscrutable as a software engineer than looking at code where you know exactly what it does At any time. And so open telemetry is also very, very useful for that, for the reasons I was mentioning of it, you know, we're, we're working very hard as an industry and as a project to ensure we can Extract all the information you need, whether you're a person, a customer who calls into these AI services, or if you're actually the provider of these AI systems and you yourself want To find out why some of these signals are not meeting your expectations. ([Time 0:22:13](https://share.snipd.com/snip/43061b08-8361-4da9-b8c2-1abc2f5827b3)) - **Existential Threat to OpenTelemetry** - An "existential threat" to OpenTelemetry is its widespread adoption as a standard. - If all software emits OpenTelemetry data natively, the need for separate instrumentation would decrease. Transcript: Alex Williams Thank you so much for taking some time to talk to us about OpenTelemetry. It's a cool project. There's lots of people involved. Just maybe a last question. About existential threats to, you know, the open telemetry project overall, what are they? Existential threats. Morgan McLean I mean, one, and this is like a positive threat, would be things building open telemetry into it, into them today. So right now, and for the rest and for all of the project's history, a lot of our work has been on providing instrumentation, ensuring that OTEL can capture data from every language, Every framework, every runtime, every database that people use. Long term, we never wanted to do that. Like that's always been kind of a bootstrapping thing for the project. Ideally, we would have the language run times and the web frameworks and the databases just speak otel and in that case you don't need most of the components that otel provides today Right like there are some sort of you know versions of this like.net uses open telemetry internally and can emit open telemetry data and there's there's various databases that do it But like you, you know, ideally if that becomes, if OTEL becomes like the big adopted standard where every single person in the world who provides any kind of software says, yeah, it Needs to emit OTEL data, then most of the surface area of the project, which is that instrumentation, doesn't need to exist anymore. In some ways, that's an existential threat because it means OTEL doesn't need to be nearly as large of a community. But it's a positive one because it means we've succeeded in our mission right like everything just emits the right data you don't need this huge library huge community of people going And pounding on the keyboards to go build and continually maintain those integrations morgan ([Time 0:27:56](https://share.snipd.com/snip/4e3f9d71-271e-4c96-ae66-97c5c18a94c9))