The Value Of Code Profiling With Opentelemetry by Greg Leffler 🔗 **Source:** [The Value Of Code Profiling With Opentelemetry by Greg Leffler](https://www.youtube.com/watch?v=yMSXaOGIcnM) > [!note]+ Document Notes > tags: [opentelemetry, code-profiling, observability] > author: Greg Leffler > source: YouTube > reference: https://www.youtube.com/watch?v=yMSXaOGIcnM > date created: 2024-12-28 18:00 > > # 🏷️ Title: The Value Of Code Profiling With Opentelemetry > <!-- deep-read:sections start --> ## 🧰 Action Inbox - [H1] Set up OpenTelemetry injector for zero-config instrumentation — install collector and restart app #opentelemetry - [H2] Enable profiling with splunk_profiler_enabled parameter — ~4% performance impact acceptable #profiling - [H3] Join OpenTelemetry End User SIG for community input — contribute to making tooling better #community ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ### 🔗 References - [OpenTelemetry Project](https://opentelemetry.io) - [Ticket Blaster Game](https://ticketblaster.splunkgames.com) ## 📝 Summary - Greg Leffler demonstrates how OpenTelemetry's sampling code profiler solves traditional profiling problems - Traditional profilers had high performance overhead and were proprietary; new approach samples periodically with ~4% impact - OpenTelemetry injector enables zero-config instrumentation for Java, .NET, and Node applications - Demo shows finding performance issue in door game application down to specific line of code causing delays - Bottom line: Code profiling is now practical for 24/7 production use with open source tooling ## 📖 Main Content Code profiling traditionally analyzed CPU and memory usage but came with significant drawbacks: high performance overhead making 24/7 deployment impractical, and proprietary tooling with licensing costs and vendor lock-in. OpenTelemetry's new sampling profiler addresses both issues. The sampling approach periodically captures snapshots rather than instrumenting every function call, reducing performance impact to approximately 4%. This makes continuous profiling feasible in production environments. As an open source project with contributions from 40+ observability vendors, it eliminates proprietary concerns. OpenTelemetry provides zero-configuration instrumentation through the injector technology for Java, .NET, and Node applications. Installation requires only deploying the OpenTelemetry collector and restarting applications - no code changes needed. The system automatically captures traces, metrics, logs, and now profiles using semantic conventions for consistent naming across platforms. The demonstration uses a Monty Hall door game application where door #3 exhibits 5-second delays. Standard tracing shows the delay occurs in the door service but cannot identify the root cause. Enabling profiling reveals the exact function (doorchecker.preCheck) and line number causing the delay - a sleep statement calculating 300 + (70^door_number) milliseconds. ## 🗒️ Notes #### ⏱️ [00:01:31](https://www.youtube.com/watch?v=yMSXaOGIcnM&t=91) — [H1] Introduction and Background - Key idea: Greg Leffler is Director of Developer Evangelism at Splunk with extensive SRE experience - Evidence: "I was an SRE before for eBay advertising and at LinkedIn. I've used observability tools, I've been on call" #### ⏱️ [00:04:23](https://www.youtube.com/watch?v=yMSXaOGIcnM&t=263) — [H2] What is Code Profiling - Key idea: Profiling analyzes CPU and memory usage to determine what threads run and methods are called - Evidence: "It's a way to determine where in the code performance problems are coming from" #### ⏱️ [00:05:45](https://www.youtube.com/watch?v=yMSXaOGIcnM&t=345) — [H3] Profiling vs Standard Observability - Key idea: Standard tools show "service X has problems" while profiling shows exact location - Evidence: "Like getting a dispatch that says something in Raleigh is on fire vs. 117 Main Street second floor first room on the right" #### ⏱️ [00:07:12](https://www.youtube.com/watch?v=yMSXaOGIcnM&t=432) — [H2] Historical Profiling Problems - Key idea: Performance impact and proprietary nature prevented widespread adoption - Evidence: "Code profiling historically has had a very heavy performance impact...nobody would leave a profiler running 24/7" #### ⏱️ [00:09:15](https://www.youtube.com/watch?v=yMSXaOGIcnM&t=555) — [H1] OpenTelemetry Overview - Key idea: OpenTelemetry is the standard way data gets from applications to observability systems - Evidence: "Over 40 observability vendors contribute. There's thousands of individual people who contribute" #### ⏱️ [00:12:30](https://www.youtube.com/watch?v=yMSXaOGIcnM&t=750) — [H3] Semantic Conventions Importance - Key idea: Semantic conventions standardize naming across different systems and languages - Evidence: "CPU_use on this one, CPU on that one, CPU0 load...your observability backend would like to aggregate them" #### ⏱️ [00:15:45](https://www.youtube.com/watch?v=yMSXaOGIcnM&t=945) — [H1] OpenTelemetry Sampling Profiler - Key idea: New sampling profiler can run 24/7 with minimal performance impact - Evidence: "People that have this enabled on average suffer about a 4% performance impact" #### ⏱️ [00:18:20](https://www.youtube.com/watch?v=yMSXaOGIcnM&t=1100) — [H2] Door Game Application Demo - Key idea: Demo application shows Monty Hall problem with deliberate performance issue - Evidence: "Door number three takes a long time to open...we could figure it out pretty fast but go with me" #### ⏱️ [00:22:15](https://www.youtube.com/watch?v=yMSXaOGIcnM&t=1335) — [H1] Zero-Config Instrumentation - Key idea: OpenTelemetry injector requires no code changes for Java, .NET, Node applications - Evidence: "Install the collector on the host, restart the app. That's it. There's not a step three" #### ⏱️ [00:25:30](https://www.youtube.com/watch?v=yMSXaOGIcnM&t=1530) — [H3] Trace Analysis Without Profiling - Key idea: Standard tracing shows service calls but not internal code execution details - Evidence: "We made an HTTP request, 5.2 seconds later we got our answer...doesn't show you what the request was doing" #### ⏱️ [00:28:45](https://www.youtube.com/watch?v=yMSXaOGIcnM&t=1725) — [H2] Enabling Profiling Configuration - Key idea: Profiling enabled through simple Kubernetes manifest environment variables - Evidence: "splunk_profiler_enabled...splunk_profiler_memory_enabled...that's it" #### ⏱️ [00:32:10](https://www.youtube.com/watch?v=yMSXaOGIcnM&t=1930) — [H1] Finding the Root Cause - Key idea: Profiling reveals exact function, file, and line number causing performance issues - Evidence: "32 call stacks...doorchecker.preCheck...it tells you what line it's on" #### ⏱️ [00:35:25](https://www.youtube.com/watch?v=yMSXaOGIcnM&t=2125) — [H2] Code Analysis and Fix - Key idea: Sleep calculation uses 70^door_number formula causing exponential delays - Evidence: "70 to the zero power is one...70 squared is 4,900...300 plus 4900 is 5200" #### ⏱️ [00:38:40](https://www.youtube.com/watch?v=yMSXaOGIcnM&t=2320) — [H1] Session Recap and Resources - Key idea: Profiling is now practical with open source tooling and minimal performance impact - Evidence: "Performance and proprietary concerns stopped adoption...now it performs great, it's open source" ## 📚 References - [OpenTelemetry Project](https://opentelemetry.io) - [Ticket Blaster Profiling Game](https://ticketblaster.splunkgames.com) - [Splunk Observability Cloud](https://splunk.com/observability) - [OpenTelemetry End User SIG](https://github.com/open-telemetry/community/tree/main/sigs/end-user) ## 🗣️ Paraphrase - Traditional code profiling tools were too resource-intensive for continuous production use and typically required expensive proprietary licenses - OpenTelemetry's sampling profiler solves both issues by taking periodic snapshots instead of constant monitoring, reducing overhead to ~4% - Zero-configuration instrumentation through OpenTelemetry injector eliminates the need for manual code changes in supported languages ## 📰 Newsletter 🔍 **Game-Changing Development in Production Debugging** The days of choosing between application performance and deep observability are ending. OpenTelemetry's new sampling profiler brings continuous code profiling to production with just 4% performance overhead - finally making it practical to run 24/7. **Key Benefits:** - **Zero-config setup** for Java, .NET, and Node applications - **Pinpoint accuracy** down to specific functions and line numbers - **Open source** with contributions from 40+ vendors - **Production-ready** sampling eliminates traditional performance concerns The demo shows finding a 5-second delay caused by a single sleep statement - something impossible with standard tracing alone. This level of insight transforms incident response from "something's wrong somewhere" to "line 47 in doorchecker.java has the issue." **Getting Started:** Try the interactive [Ticket Blaster](https://ticketblaster.splunkgames.com) game to see profiling in action, or explore the [OpenTelemetry documentation](https://opentelemetry.io) for implementation details. Resources: [Learn WordPress](https://learn.wordpress.org/), [Developer Docs](https://developer.wordpress.org/), [WP-CLI](https://wp-cli.org/) ## ✅ Action Items #### DevRel - Developer advocacy: Present profiling capabilities at conferences, create hands-on workshops - Education: Develop curriculum comparing traditional vs sampling profilers - Developer experience: Document zero-config instrumentation workflows - Developer marketing: Create interactive demos like Ticket Blaster game - Community management: Moderate profiling discussions in OpenTelemetry channels #### OSPO - Policy & Governance: Evaluate OpenTelemetry adoption policies across engineering teams - License Compliance: Review OpenTelemetry Apache 2.0 license compatibility - Security Management: Assess profiling data collection and transmission security - Software Inventory: Track OpenTelemetry collector deployments via SBOM - Developer Support: Provide training on profiling configuration and tuning - Community Engagement: Support OpenTelemetry End User SIG participation - Strategic Alignment: Define profiling adoption KPIs and success metrics #### Community - Maintainers: Contribute to OpenTelemetry profiling component development - Contributors: Submit language-specific profiler implementations - Sponsors: Fund OpenTelemetry profiling feature development - Ecosystem: Coordinate profiling standards across observability tools #### WordPress - Core Performance: Implement OpenTelemetry profiling for WordPress.org infrastructure - Playground: Add profiling capabilities to development environments - Community: Share profiling best practices at WordCamps - Training: Create profiling courses for WordPress developers - Documentation: Document profiling setup for WordPress hosting providers - Meta: Monitor WordPress.org performance with continuous profiling ## 🧪 Hypotheses - Sampling profilers with 4% overhead will achieve >90% adoption rates in production environments within 2 years - Zero-configuration instrumentation will reduce time-to-value for profiling from weeks to hours - Open source profiling will increase incident resolution speed by 50% compared to traditional debugging methods - OpenTelemetry standardization will drive consolidation of observability tooling across the industry ### 📊 Methodology - Study design: A/B test profiling-enabled vs traditional monitoring for MTTR comparison - Data sources: Production incident response times, developer survey data on debugging efficiency - Experiments: Deploy sampling profiler across percentage of production hosts, measure performance impact and debugging effectiveness ### ❓ Questions - How will open source observability platforms integrate OpenTelemetry profiling data? - What security implications arise from continuous code execution profiling? - Can sampling intervals be dynamically adjusted based on application load patterns? ### 🔮 Future Research - Integration patterns for profiling data in existing observability dashboards - Machine learning applications for automated performance anomaly detection from profiling data - Cross-language profiling correlation in polyglot microservice architectures ### 💡 Implications - Practice: SREs can identify root causes faster, reducing escalations to development teams - Policy: Organizations need profiling data retention and privacy policies - Security: Profiling data may expose sensitive application logic and execution patterns - Education: Engineering curricula should include continuous profiling best practices - Funding: Reduced debugging costs may offset observability infrastructure investments ### 🧠 Learning Styles Assessment Reading level estimate: 12th grade (technical conference presentation) #### 👀 Visual learners - Present: Flame graphs, waterfall diagrams, Kubernetes manifests, code screenshots - Improve: Architecture diagrams showing profiling data flow, performance impact charts #### 🔊 Aural learners - Present: Conference presentation with clear narration and explanations - Improve: Audio summary of key profiling concepts, podcast-style technical discussion #### 🗣️ Verbal learners - Present: Analogies (firefighter dispatch), clear definitions, step-by-step explanations - Improve: Glossary of profiling terms, comparison tables vs traditional approaches #### 🧩 Kinesthetic learners - Present: Ticket Blaster interactive game, hands-on demo with door game - Improve: Sandbox environments, guided profiling exercises, configuration checklists #### 🧠 Logical learners - Present: Mathematical examples (70^door_number calculation), performance metrics, decision trees - Improve: Profiling adoption decision matrix, cost-benefit analysis framework #### 🧑‍🤝‍🧑 Social learners - Present: OpenTelemetry End User SIG recommendations, community contribution opportunities - Improve: Profiling study groups, peer review of configurations, moderated implementation discussions #### 🧘‍♀️ Solitary learners - Present: Self-paced demo walkthrough, individual problem-solving exercises - Improve: Reflection questions on profiling strategy, personal learning checklist #### 🌿 Naturalistic learners - Present: Ecosystem thinking about observability tooling evolution - Improve: Environmental impact considerations of profiling infrastructure, sustainability metrics ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Clear heading structure, descriptive slide content - Improve: Alt text for flame graphs and technical diagrams, ARIA labels for interactive elements #### Keyboard navigation - Present: Standard presentation navigation - Improve: Keyboard shortcuts for demo interactions, skip links for code sections #### Contrast and readability - Present: High contrast slides with clear text - Improve: Dark blue on black text (mentione <!-- deep-read:sections end --> ## 📺 Video  --- # 📄 Full Transcript *Auto-generated from YouTube captions · timestamps roughly every 30 seconds · [watch on YouTube](https://www.youtube.com/watch?v=yMSXaOGIcnM)* **[00:05]** Okay, we're going to go ahead and get started. It's uh 11:31. So, thank you all for being on time. I appreciate it. Uh I'm here to talk to you today about the value of code profiling with open telemetry. Uh and before I start, I want to sort of ask like how many of you are familiar with open telemetry by surprising number. All right, great. Last year I asked this question and there were like three hands that went up and this year it's almost all of you. So, that's awesome. Um how many of you familiar with code profiling? Okay. Wow. Well, I'm happy to see that. Um, this presentation may be a little introductory at parts then. Um, but if **[00:37]** you have in-depth questions, there'll be time for Q&A. Uh, feel free to ask me at the end. Um, but I'm just going to go ahead and jump right into it. Uh, this is my picture. This is my name, my title, all that sort of stuff. Um, I currently am director of developer evangelism at Splunk. Uh what that means is I come to things like this and talk about technology and products that we uh either support, contribute to or sell. Um in this case, this is something that we contribute to pretty heavily. Um I was an SR before for eBay advertising and at LinkedIn. Um I've used **[01:08]** observability tools. I've used code profiling. I've used all this stuff a lot. I've been on call. I've had, you know, the nightmares of Christmas mornings ruined by being paged. Um yeah, I'm old enough that there were pagers. Um, but you know, uh, this is sort of sort of what life was like for me. Um, and, uh, we didn't have code profiling, uh, easily accessible at least at eBay or LinkedIn when I worked at either of those places. Um, and I really wish we had. Uh, and hopefully by the time you're done with this presentation, you feel the same way and you're comfortable **[01:38]** trying to get it where you work as well. So, this agenda looks like a lot. I promise it's not. We will finish early. You'll be able to go get in line for lunch. Don't don't worry about it. Um, but we're gonna cover uh several things. We're going to cover what code profiling is. Uh, why you might want to use it. Uh, I'm going to do a little bit of an intro to open telemetry for anybody who's not familiar with hotel um or who just needs a refresher. Um, I'm going to show you the sample application that we're going to use for this. Uh, how we instrumented the application, how we **[02:09]** turned on profiling, how we found the performance problem. Uh, and then give you some resources where you can go if you want to learn more stuff. Um, and I'll also, you know, feel free to have you connect with me on LinkedIn. You can ask me things directly as well. So, we're going to start off by talking about what is profiling. Um, and profiling is uh a way to analyze the CPU and memory usage of your application is the short answer to that, right? If you ask ChatGBT, that's probably what it would tell you. Well, I mean, it would probably tell you in a few million more uh a few million more uh words than **[02:41]** that, but that's pretty much what it is. Um, and it's a way to determine what threads are running and what methods or functions are being called by those threads. Um, and to see the impact of what those threads are doing. It helps you figure out where in the code performance problems are coming from, right? So, um, you know, from standard observability tools like there is a problem with service X, right? The CPU usage is high on that host that it's running on. And that's all you know. You don't know where in service X the problem is. you don't know why service X **[03:11]** is having high CPU usage. All you get is service X and you might not even get that. You might just get oh it's this host and then which service it is. Um so code profiling is a way for you to go beyond sort of there is a problem somewhere in the universe to where exactly the problem is. And uh if you were a firefighter it'd be like getting a dispatch that says hey something in Raleigh is on fire. Good luck. um or getting, hey, it's at, you know, 117 Main Street on the second floor, first **[03:42]** room on the right and the outlet at the bottom. Like, you know exactly where the fire is. You know exactly where to start looking. Um and so it's a way for you to really accelerate uh the time it takes to solve problems. Uh unfortunately, uh computer systems have lots of problems and sometimes it's difficult to figure out why those problems are occurring. Code profiling can tell you very quickly why the problem is happening. Well, actually, why maybe isn't correct. Uh, it can tell you what is running when it's broken. Why it's doing that is, you **[04:13]** know, up to you, [laughter] but you can figure out, you know, where the problem exists in your code pretty quickly. So, hey, that sounds pretty awesome, right? Like, why should we use it? I mean, it's I just kind of told you like it it gives you a lot more information. It tells you what you should do. Um, so I'm going to be a little sneaky with this and talk about why should you not use it or why do people not use it. Um, and the there's one big word why people don't use code profiling and it's performance, right? Roughly half of you **[04:43]** raised your hands about that you've used code profiling before. Uh, how many of you have noticed a performance impact from running code profilers? And it's about that same crew, right? Code profiling historically has had a very heavy performance impact. You set up a profiler and it's doing a lot of work. It needs to sample constantly to figure out what threads are running, how much memory are they using, what functions are they calling. And it's something that historically has been very difficult to like leave a profiler running. Like nobody would leave a **[05:13]** normal profiler running on their code 24/7 uh unless they had a very simple application or running it on a very beefy machine. Um you would only turn on the profiler when there was a problem. Great. But you know once you have the problem it's a little late at that point right like you don't get the insight in advance that could tell you hey this is something that you should fix and that how you could fix it with code profiling. So historically performance has been the biggest problem right code profiling is an expensive operation. It's something that you can't leave all the time. Uh so a lot of people like **[05:44]** play with it in school or they set it up once and then they're like I'm never using this again. Like in production it slows down my app too much. we can't afford to pay AWS three times extra to always have the profiler turned on. The other one is uh prop proprietary or but that that profilers are proprietary. Um I couldn't really find a good single word that encapsulated that. But uh in the past code profilers have generally speaking been pretty closely guarded secrets by vendors, right? Like **[06:14]** vendors would make a profiler available um that is sort of a universal profiler, right? A lot of languages of course ship with profiling support for that language um and profilers for that language, but ones that are integrated with observability tools or that give you data that you can use in the same interface you're using to troubleshoot the rest of your application have historically been proprietary. Um and proprietary we don't like at this conference for obvious reasons, but for one of them is there's cost, right? You generally had to pay to license these. Um the other one is that you don't know what they're doing. you can't tell, you **[06:46]** know, if they're doing anything other than profiling your code. You don't know how well they perform. You can't fix it if something's broken in them. Um, so performance and proprietary, the proprietary status uh of profilers have generally been the two big stumbling blocks and the two big reasons people don't use profilers. So keep that in mind. Uh, like I said, I'm going to take the the open telemetry detour here for a little bit and I'm going to cover open telemetry in 60 seconds. No, I'm I'm I'm not going to cover up until 60 seconds. That's probably not **[07:16]** possible. Um and I'm going to cover it quickly. Um because again, a lot of you said you were already familiar with it, but um if at any point you're not like feel free to yell at me and I probably won't hear you because half of you are in the back of the room, but if you want me to stop and you know, give you more depth on anything, I'm happy to do that or there'll be Q&A time at the end. uh open telemetry. The first sentence definition of this is that it is the way that data gets from applications into observability systems. Uh I used to say it was a way or like an an emerging standard or something like that. It's **[07:48]** the standard now that there's over 40 uh observability vendors contribute. There's thousands of individual people who contribute to the project. It it one it's it's open telemetry. That's the way that data gets from applications into observability platforms. And um if you aren't using it already, you should [laughter] because uh basically every vendor is on board, including all of the ones that are uh sponsoring the show except for one. Uh I'll I'll leave you all to figure out which one that is, but uh pretty much **[08:18]** everybody else is fully on board the open telemetry train and uses it as either the main data format or a first class data format. Um the Splunk observability products are all native. We built them on open telemetry. Uh we employ a bunch of open telemetry maintainers. We're very big supporters of the project. But it's not a Splunk project, right? Like it is a very busy, very popular project. Uh it's the second most busy CNCF project. Uh the first one is I don't remember. It's some hobby project called Cube or something. It's **[08:49]** never going to catch on. It's not something we don't need to worry about. Um but it's a very popular project. It's something that is used a lot. Um, and like I said, if you don't know it, you should probably learn it. Uh, because it's going to cover pretty much anything. Sure. >> Uh, yes. So, the question was, is open telemetry a standard or is it code that runs? And the answer is is yes. Um, open telemetry provides uh flexible lightweight instrumentation that is part of the code. Uh, so there are libraries, **[09:20]** SDKs, APIs set up for how you actually instrument your applications. the project ships uh reference code or sample code for dozens of languages and frameworks and all that sort of stuff. Um so it is code. Um it's not it's mostly not something that you like say I installed hotel on this box is generally something you probably wouldn't say. Um but there is something called the open telemetry collector which is the project doesn't like people saying this but it's basically like an **[09:51]** agent in other observability systems. Um and it is a binary that runs on a machine and collects metrics and traces and logs from that machine uh processes them and then ships them off you know to wherever they're going to be used. So it's both uh code and it's a standard as well. The biggest standard that the project has come up with are what are called semantic conventions. Um the hotel project sometimes is maybe a little overly academic and things. Uh semantic conventions is an academic way to say what you call stuff. So if you have a metric that captures the percent of CPU **[10:23]** that's currently in use on your host, right? Um there's a semantic convention for what you name that metric. And the reason that semantic conventions are important is that if you look across all of the different like cloud providers, the different operating systems that run your services, the uh different frameworks that run your applications, the different languages, like what those things are called could all be different as the default naming of a metric. Um, but they all mean the same thing. and your observability backend would really **[10:54]** like to be able to say, "Hey, I know all these metrics are just CPU. So, I'm going to show them in the same way. I'm going to aggregate them across each other. I'm going to let you slice and dice that across the different cloud providers." Um, without you having to manually go in and say, "Okay, it's CPU_use on this one. It's CPU." On that one, it's CPU0 load, right? Like, um, it sounds really petty, but it is actually very important, right? the the two hardest problems in computer science are you know naming things cache invalidation **[11:26]** and off by one errors right so like you really need to pay attention that you name stuff the same across all the things where you're going to use it where you're going to call for it um like I also said optel provides instrumentation for a lot of languages and frameworks pretty much every common one um like I said I work at Splunk I work at a vendor we sell an observability product we've seen people that have pretty esoter caric setups that we're still able to instrument with open telemetry. Um, as you might guess, a lot of the **[11:57]** more modern, we'll say, I don't want to use a derogatory term. I don't want to say they're hipster languages, but a lot of the more modern languages, um, have very good support because people want to like build my first project in Erling or whatever, and so they build a notel collector, they put libraries on for it. Um, so there's actually a lot of support for a lot of languages you might not expect, but of course all the big ones, Java, net, node, you know, all those sorts of things are very well supported. um and are used extensively. Uh [snorts] we have uh even just at Splunk, we have some very huge customers who you've definitely heard of who use uh open **[12:29]** telemetry every day. One of the things I didn't put on this slide is that a lot of the components in open telemetry are still classified by the project as alpha or beta. Um and if you work at a corporation, a lot of times people see that and they're immediately like, oh no, I'm not going to run anything in production that's beta, right? Um Gmail was on beta for 14 years. So like beta is beta is just a word. Um and the project actually has a of course they do have a very long document explaining when they call things alpha, when they call them beta, when they call them **[13:00]** production um or stable, sorry, as they call it. Um we have people running very important, very critical code in beta um all across the world um and don't have problems with that. So um if you want to talk about naming stuff, like feel free to talk to me after, but there's lots of stuff that's in alpha and beta and hotel. Don't be scared of that, right? It is good enough for production even if it's just in beta. Uh and one of the latest innovations from the project is a sampling code profiler. So, um I'll talk a little bit about what that means uh **[13:31]** here in a second, but the basic version of this is this is a profiler that you can run 247. You can leave it running all the time. And the reason that you can do this is that it doesn't collect every single function call. It doesn't instrument every single assist call. It doesn't know every single memory allocation, right? Periodically, it wakes up, it figures out what's going on, it samples that as a snapshot in time. And that's good enough in the overwhelming majority of use cases to figure out what's going on. And because it's not constantly hooked into every **[14:03]** single call that gets made, you can comfortably leave it running. We see that people that have this enabled on average suffer about a 4ish% performance impact in their applications which you know if your app can't handle a 4% performance hit then you need more infrastructure or better architecture or both. Um but like the value of this data really can't be uh can't be hidden. Did they finish the uh toxic waste cleanup? So, [laughter] so, uh, like the value of having the **[14:33]** profiler running is is super critical. And the fact that it's sampling means you can leave it running. You can feel confident about it running and not hurting your infrastructure and you can use it before you know you need it, right? Like you don't have to worry about like one of the worst things is when you have a problem that doesn't that you can't reproduce very well, right? And um, one of the [clears throat] worst worst times of that is when it's an important user that runs into it. Uh, like I mentioned at the start of this, I worked at LinkedIn for a while. Our CEO was a very aggressive LinkedIn user, as you might guess, and uh, when he had a problem, **[15:05]** what he would do is he would take a screenshot of an error page and email it to the knock and and that was it. That was the troubleshooting you got. Like, you don't know what he was trying to do. You don't know what the operations were. You don't know what he expected to happen. All you get is presumably something's broken and a question mark. Um, and the CEO is a very different LinkedIn user compared to normal LinkedIn users, right? He would have hundreds of thousands of connections, whereas every other user was capped at 10,000 at the time, right? He would have **[15:35]** um he would be in dozens of groups. He would do stuff that nobody else would do. So, there were lots of things that were weird for him, right? And we could not reproduce those. There's nobody else's account that we could say, "Let's go in and try it with our test account that has 500,000 connections." Like, we didn't have such a thing. And so we really would have liked to have been able to have this profiling to go in and say, "All right, well, when he was running that transaction, what was going on, right?" And we could figure that out without having to try to reproduce it, right? But if we didn't have the **[16:05]** profiler running at the time he made the request, what were we going to do? Call him and be like, "Hey, Jeff, could you run that request again?" You know, like, no, obviously we weren't going to do that. Um, and so having the profiler running constantly is I would recommend that you do it if not in your full production environment, at least in a percentage of your production hosts so that you have a chance that the thing was was profiled. Um, but again really like we see people run this, they do not suffer performance impacts that they care about. Um, and the value of the data that they get is is worth it. **[16:40]** >> Sorry. Oh, it's it's just OTEL. It stands for open telemetry. Yeah. Yeah. It's just the short version. Yeah. Yeah. >> Yeah. So, the question was what are the differences between instrumenting and profiling? And I will talk about that shortly. So, uh first I'm going to show you our sample application. This is our sample application. It's a very cool, very slick web 2.0 you know, sort of thing. Um, it's called the door game. **[17:12]** And if you have taken a statistics class or if you're over 4550 maybe, um, you'll be familiar with this. Um, basically what you're doing is you're playing a game. The game is you are at the final part of a game show and there are three doors in front of you and you get to open you get to pick one of those doors and after you pick one, the host will pick one of the doors you did not select and say, "Hey, this door did not have the prize behind it. Would you like to switch doors? How many of you would switch doors? **[17:44]** All of you that have your hands down are wrong. Um, [clears throat] you should switch the door. Um, and if you want to argue with me about the math about that later, feel free. But, uh, basically, this game exists to have that be a simulation so that we can show people who have their hands down. All right. Well, here, go play this game a couple of dozen times and then tell me if you would switch, right? Because you get to play it and see like, oh, I switched and it was the correct choice. Or, oh, I switched and it was wrong. Which does happen. And it's not a guarantee you're going to be right, but um anyway, so let's show you what a play of this game looks like. So we have the game. We **[18:14]** click on let's play. And apparently this was the part in the demo where I told you all this. There we go. So it fetches a game from the server. This takes a while. Um I have another version of this where I talk about why it takes a long time to start up, but we're not going to do that today. Um instead, you see what I told you, right? We picked door number two. They say don't open door number one. Here there's a goat behind it. We stay at door number two and it's still a goat. So in that case we should have switched. We pick door number two again and you know we didn't **[18:45]** believe us last time so we stay with door number two and this time we win. So we're 50/50. Not bad. Uh this time we're going to stay with door number two. We won last time. We're on a hot streak, right? Um and so but this time though, you know, I heard we should switch. So I switched to door number three and well that that took a while. Um and I still lost. So that's not great. Um, [laughter] but maybe, you know, maybe the game doesn't like me switching. Like, I'll start with door number three this time. All right. And, uh, I I feel good about door number three. So, I'm going to stay with door number three. **[19:15]** We're going to see what happens with door number three. All right. Clearly, something is wrong with door number three, right? Like, this is taking way longer than this than than door number two is. Let's look at door number one. We stay with door number one. Now, I don't even care about winning. I care about the performance of this. And door number one opened real fast. I have to worry about anything behind door number one, right? Uh so what we've determined here is like this is our application. It's a really simple, really straightforward application, but what it's showing us is one of these paths has a performance problem, right? And look, the total **[19:47]** source of this game is like 80 lines. We could figure it out pretty fast, but you know, go with me for this. So we don't know why door number three takes a long time to open. [clears throat] So it'd be cool if we could look in our observability system and have it tell us what's going on, right? So before we can do that, we need to go through a process called instrumentation. Instrumentation is adding stuff to your application. Um code traditionally, but uh as you'll see, it's not always code, but instrumentation is adding stuff to your application that can collect data from **[20:18]** it and send it out to your observability system. So when we say I instrumented this application, all we mean is we added stuff to it that lets it collect data on performance data, metrics, trace data, log lines, whatever. um and send those to wherever we want to analyze them. I kind of lied though. We didn't actually do any instrumentation for this application um because this is a Java application and it's a Java Spring application. So both of those things are supported by uh what we call zero config instrumentation in open telemetry uh through a technology called the open **[20:49]** telemetry injector. Uh you might not find many Google results for that today. Um check back after CubeCon, you'll find a bunch of them. Uh CubeCon's next month if you don't know. Um but the open telemetry injector is how we were able to accomplish this. Um the way that this works is you install the open telemetry collector on the host that this application is running on and you restart the app. That's it. There's not a step three. There's not anything you have to do like you install the collector. You configure where you want the data to go in the collector but you don't have to touch anything in that **[21:20]** application. You don't have to go in you don't have to figure out how to add the libraries. You don't have to figure out how to say like this is something you need to care about. Like it does it all for you. It's awesome. It's really cool. Java.NET and Node are currently first class citizens. Other languages are being added but um and all their common frameworks as well are automatically instrumented. So, you know, we talk about how do you instrument an application? Well, it ranges from do nothing to like okay go into the source and flag everything that is critical data manually add the open telemetry API **[21:51]** and the SDK for your language. you know, emit all the things that you want to do. Like that's when we talk about instrumentation, that's what a lot of people think about. Um, but really it can be as simple as install the cluster and restart the app. Um, if you want to know how that works, um, this is the first operating system we ran this on was Linux, which may may give you a clue. Um, we hook into LD preload. So we are able to inject stuff through that way. That's how we do it. Um, but so that's it. That's how you instrument it. And now I'm going to show you what you get from instrumentation, right? So from **[22:22]** an instrumented application um you can look in an observability system of your choice. Uh I work at Splunk so obviously this is Splunk but they're all going to look pretty similar to this. Um and what you'll see here is uh we have traces from our application right so we can see uh there are a few traces here on the screen. I sorted them by the longest running time. The duration there is the third column. Um and you can see there's some that are pretty long, right? Like there's one that was 30 seconds. Um, this is due to a slow database thing that, like I said, I'm not going to **[22:53]** cover in this session, but uh, and I'm not going to cover it just because right now the database query monitoring stuff is proprietary to Splunk. So, I didn't want to talk about it here, but we can tell you if your database is having problems, that's why. Um, but there's one in a blue box that's one that we ran uh, that we want to see like, okay, why did this one take 5 seconds, right? If we look at when we picked door number three, if any of you were counting, it takes about 5 seconds for door number three calls to finish. So, what happened there? Well, when we click on the trace, we get sent to a waterfall. **[23:23]** It's pretty sad waterfall, right? There's only two lines in it, but [laughter] you get the idea that as uh your application is calling [clears throat] services inside of your infrastructure, automatic instrumentation can capture that, right? Right. So what we can say is this application door game uh there was a get into that service and it was getting a game you uh ID of the game that the a door was picked which door was picked and getting the outcome of that. It calls into the door service uh a method called doorgame.get outcome and it took 5.2 **[23:55]** seconds and that's it. We can't tell why it took 5.2 seconds. We can't tell what in the code was running for 5.2 seconds. All we basically have is we made an HTTP request, 5.2 seconds later, we got our answer. That's still pretty cool. Like if this was a more complicated service and a more complicated app and there were 30 services involved or 200 services involved or 17 services involved like when I worked at LinkedIn [laughter] like knowing which services were being called would be pretty cool, **[24:27]** right? But uh in this case, this isn't super helpful because there's only one service. But um you know, regular automatic instrumentation just shows you where the request was going. It doesn't show you what the request was doing, if that makes sense. So let's talk about enabling profiling, right? Um I sort of alluded earlier that, you know, oh maybe we're going to have to do, you know, we're going have to do more work like instrumentation is different than profiling. Like how how bad is it going to be? It's it's not too bad. Uh um but I do want to start the **[24:58]** section with a big caveat that like this is going to be different in the open source version of the profiler. Right. Um this demo was done using the Splunk distribution of the open telemetry collector uh and the Splunk distribution of the profiler. Um you'll [clears throat] be able to see that very easily because the config parameter is called splunk_profiler enabled. Um but there will be a way to do this that is probably as easy in the open source release of the profiler. Um, I just don't know what it is yet. Uh, because there aren't currently any open source observability tools that use this data. This is all very fresh. Um, so **[25:30]** we'll see. But just so you know, this is kind of how you will do it. Um, this application is deployed in Kubernetes, right? Like we're a cool hip and groovy place. We've been using Kubernetes for a while. You know, tomorrow door game might become Google and we might need to serve it to everybody in the world. They might be want to play the door game. So, we run into Kubernetes. Um, so all we're going to do is edit the manifest and redeploy it. Um, there are some tuning settings for profiling uh that you can edit. As I mentioned earlier, this is a sampling profiler. Um, and I saw like two people's like little hamster wheels **[26:02]** start turning of like, oh well, how often does it sample and how do I configure what it's going to do? And how do I figure out like what if there are threads I don't want sampled? And you you can tune all that. Um, we're we're not doing that today. Um, the the settings of the profiler are set where they are for a reason. um changing these in what you feel like are pretty insignificant ways can have very significant performance impacts. Like this is definitely not something that has been super well tested if you're going to talk about going in and changing the sampling intervals or modifying the other settings. You can do **[26:32]** it, but like with pretty much any tuning operation, you really should know what you're doing. Like um don't don't Google something and copy and paste like, oh, I'm just going to change these because somebody else said to. Um so we're not going to edit any, but they do exist. So we talk about enabling profiling. What do we do? This is a Kubernetes manifest. Um it's YAML. Sorry, it's the best we got. Um but you can see um that we have a couple of parameters set up around the open telemetry instrumentation. Those strictly don't need to be here. Like it would have worked with the injector um without **[27:03]** having this information. But uh it looks a little bit nicer in the UI if we name the service here. If we call it doorgame. Um, but what you'll see right below where the cursor is is, uh, the parameter of Splunk_profiler enabled. Um, and then Splunk profiler memory enabled. Uh, those enable the profiler for CPU and for memory respectively. Um, you may wonder why the CPU one doesn't have CPU in the name. I don't know. Maybe the open source version will fix that and they'll both be descriptive of what they are. Um, but the one that's just spunk profiler **[27:34]** enabled is CPU. The one that has memory is for memory. Um, call stack interval is the sampling interval and there are also other tuning parameters that we didn't even put in here. Um, generally, uh, I do this as a workshop sometimes where you all can play like hands-on and do this yourself and people will change a sampling interval. Um, so I'm like, all right, I'll put the we'll put it in there. Um, and then they see, you know, what that does for performance. Um, but so we make this change, we run cube cuddle apply, we wait, the app restarts, and you know, we're running now with profiling enabled. So now we want to **[28:04]** find the problem, right? All right, we want to figure out what's going on. And uh we enabled profiling. We played the game a couple of times. Uh you know, same thing. It didn't magically fix itself by restarting. Wouldn't that be nice? [laughter] It's still doing the same thing. Um so we know, all right, let's look at the profiles. Let's see what's going on. Um and what we're going to see is of course the file, the source file on the disk, where the method is running, what line of the code is running. like you actually get exactly what you need to figure out what the problem is, right? Assuming you can read **[28:36]** whatever language this game the the app is written in. Um, this one's Java, pretty elementary Java, so hopefully it should be okay. Um, and so you see where it is and then you can fix it, right? We're uh we're cool SR and we're going to fix stuff in production, right? We find out what's wrong. We're just going to delete some of the source and just redeploy. It's probably fine. Um, so let's go in and let's look at our let's look at our observability system again, right? And I showed you this already. This was where the traces were coming from the application. Um, and this looks exactly the same, right? [laughter] Like nothing nothing on this **[29:07]** screen is different. Nothing has changed. Uh, but what does change is when you go to look at the trace uh at the waterfall for a particular trace, you'll see what's different. So, I'm going to sort these by longest running traces. I'm going to scroll down a little bit until I find a long running one that also has a time stamp after I set up the profiler because obviously there not going to be anything in there from before. And I'm going to go look at that particular trace. And on this trace, what you'll see now is this also looks pretty the same. Uh but there is one difference. If you look at the bottom box of the trace in dark blue, uh **[29:40]** it's black on dark blue, so it's probably pretty hard to read for you all, but it says 32 call stacks, right? So what this is telling us is that the profiler was enabled and was running when this trace when this trace happened. So we can see what was going on at the time of that trace. second. It's either 22 or 32. I can't tell either. It's it's black on blue. [laughter] But uh what you'll see over here on the right hand side is uh a flame graph uh and a basically a stack trace, right, of what was running at the time. If you've ever debugged Java, you've seen a million of these. Um but **[30:11]** this is one that you didn't have to manually trigger. And as I'm mousing over each one of these things, you can see what functions being called. The top of this is all libraries, system standard stuff. The bottom of it is our application code, which you can tell if you see it starts with com.splunk. So, this is the code that we wrote ourselves. And you can see this as we're scrolling down through here. We can see it's spending a pretty pretty long amount of time on uh on a sleep right there. Well, that's probably part of the problem. Um, but then the function that's running is called uh doorchecker.prech, right? And it tells you, which uh **[30:44]** unfortunately I moused away from it, but it tells you at the end of that line uh that it's in doorchecker java. It tells you what line it's on, right? So, all right, something is calling sleep. We don't know why it's sleeping. Uh, it's sleeping because we contrived a scenario for the demo, but I mean, you get it. In a real production problem, you don't know why it's doing whatever it's doing. Um, so we know now what function to look at. We know what file it's in. We know what line it's on. We know what's going on. So, [sighs] let's go look at this file. Um, I have the cursor on line 36, uh, as you can **[31:15]** tell from the status bar. Uh, and you can see, yeah, there sure is asleep right there. So, uh, that's the cause of our problem. just delete it and and go on with life, right? Um well, okay, like sure, but maybe it sleeps for a reason. Like, let's try to figure out why it's sleeping. And if we look up a little bit more, um what you'll see is this pre-check function that gets called uh is calculating something first before it goes to sleep. So, it's calculating extra. And this extra that it's calculating, uh which is the line right above the cursor if you're not following **[31:46]** along, um is is taking 70 to a power of the dor number that we selected. Uh so Java is a zero index language. Uh so if you select door number one, that's actually door number zero. 70 to the zero power. Anybody? One. Uh all right. So if you pick the first door, then it's the extra is one. Um if you pick the second door, it's 70 to the first power, which is 70. Uh and so if you pick the second door, then at the third door, then it's the value is 2. 70 squared, which is 4,900. If you look at our sleep **[32:17]** there, uh it's 300 plus extra. 300 plus 4900 is 5200. Seems a lot like 5.2 seconds that we were encountering for this function to take. Um and so you can see right like this pre-check function that is sleeping for some reason based on the amount based on which door you selected um is why this application is slow. It's why door number three is taking longer than the other ones. Like obviously the profiler is not magic. We can't say like based on this that I can just remove that. like there may be some reason it **[32:48]** sleeps and there may be some reason it sleeps longer for certain doors that are picked. So obviously you'd want to check with engineering or somebody to say like hey it's asleep for a reason can we change this or whatever but it's a much easier for you to know and when you call the engineering on call to say hey line 47 of file blah blah blah has a sleep why is that there right and they can go much more easily and say I don't know let's see right and hopefully they have comments or the pull request has history or hopefully there's **[33:19]** something that explains why that sleep is there. Um, but the point is is that you as the person who's responding to this um or you as the engineer who's helping that that person um can say like, "Hey, I know where the problem is. I found it right away. This is where it is. This is how I can fix it." And it's in the same observability platform you use for everything else. It's really straightforward. It's really easy to figure out why things are not performing the way that they should. So, we're just going to nuke that sleep line and rerun our application just to show you that that was the cause of it. **[33:50]** Right. So if we go back into the door game and we start with door number three because that was the one that had a problem and we stay with door number three, it's you know 300 370 milliseconds, right? So uh we fixed the problem and the other doors are still performing correctly as well, right? Like u but now everybody can play the Monty Hall game as much as they want and figure out that you know switch uh the odds of you winning are two-thirds if you switch by the way one third if you stay. um it is not 50/50. Um if you're really convinced that it's 50/50, feel **[34:21]** free to come talk to me later, but you're wrong. Uh so, uh let's talk about where we can go next. Uh I do want to provide you with some resources and things you can do if you want to start using code profiling or if you want to learn more about it. Um but first, here's a session recap. If you'd like to take a photo of this, feel free. The session was recorded, but um if you want to have like the most important stuff on one place, that's why I put this slide here. Um profiling is super cool. you should probably use it. Um, now that there's an open source version available and it's part of the Otel project, you can start using it. Like I said earlier, **[34:52]** there aren't any open source tools that use this data yet. I mean, I'm certain, you know, the open source tools will start using this. Profiles are now metric uh an OTEL pipeline type. So, they're the same level as metrics, traces, and logs. So, all the tools will start to adopt them. Um, in the past performance and concerns about the code being pro proprietary were things that stopped adoption. Um, now it performs great. It's open source. So those two things kind of knocked away. Um, open telemetry is is both the future and the present um of data ingestion into **[35:23]** observability tools. Um, that is where all the development work is going. Like it will be highly unlikely I think that anybody will be installing vendor agents in a year even, right? Like pretty much everybody's on the hotel train. So, um, if you're not, hop on because it's something that, uh, is it's something that there's still time for you to learn and contribute to as well. Like I mentioned, a lot of the components are alpha or beta stage. So, they're looking for contributors or looking for maintainers. Um, and uh, the project is also incubating at CNCF. It's really **[35:54]** close to graduation. Um, but before a project can graduate, there need to be maintainers for all the components. So, some of the components are really looking for maintainers. If you know cotlin, come come talk to me. But [laughter] like um you can still participate. You can still join the open telemetry community. Uh you're definitely not too late for that. Uh you can automatically instrument applications thanks to the open telemetry injector. Um it's again as easy as installing the collector and restarting the app for most cases. Enabling profiling is easy. Uh it lets you find problems with specific lines of **[36:24]** code. You no longer need to gp across your entire source tree to try to figure out where a function comes from um or what it's being called. you no longer need to be a Recex Ninja. Um, like it's it's much easier to solve problems when you have this data. Three resources for you to leave with. Um, if this all went over your head and you would like to have a simpler version of profiling, uh, we did create a game called Ticket Blaster. You can access at ticket blaster.spunkgames.com. Uh, you can play it on your phone, but I'd recommend on a computer so you can see the screen, but it basically walks **[36:55]** you through like what do you see when you enable code profiling? How do you fix it? What does it let you do? Um, and it's an interactive kind of deal. Um, if you want to try it yourself, uh, like I said, it is currently available in Splunk observability cloud. There's a link to get a trial on the screen. Um, and I would encourage all of you to join the open telemetry end user SIG. Uh, unless you work at Dino Trace or Data Dog or something and then then probably not. But if you work anywhere else where you know you actually use open telemetry as a user, you should join the end user SIG. um because these are folks that **[37:25]** meet to talk about as a programmer, as an S sur, as a software engineer, how do I make this easier and better for you, right? Like how do we make the documentation more clear? How do we make it easier to instrument apps? Um and so these are companies, the end user SIG is all companies who use open telemetry day-to-day as you know part of their observability practice. There are other SIGs for data dog and data trace and us where you know the people who contribute to the project talk about um but the end user sig something you can join. They're always looking for members. It's a great way for you to help build your open **[37:56]** telemetry expertise, learn more about profiling. Um, and there also is a new open telemetry certification available that I should have put on this slide. It's called the open telemetry certified associate from the Linux Foundation. Um, they have a booth here, so you can go ask about it. Um, but it is something that if you want to learn about, you can learn about it um, pretty easily and then have something to show off that you learned about it. Like this is the standard, but it's still relatively new as far as things go. Um, so it's a great thing for you and your careers to learn about and have it as something in your back pocket. **[38:26]** So, thanks everybody. Uh, there's about six minutes left, so you can join the lunch line. Um, if you have questions, feel free to come up afterwards. Uh, thanks again.