Unveiling OpenTelemetry eBPF Instrumentation - OpenObservability Talks S6E05  ## Metadata - Author: [[OpenObservability Talks]] - Full Title: Unveiling OpenTelemetry eBPF Instrumentation - OpenObservability Talks S6E05 - Category: #podcasts - URL: https://share.snipd.com/episode/2632ed2e-8ca1-4654-9927-17c8b425cf1c ## Document Notes ## Highlights - **No-Code Application Instrumentation** - OBI provides no-code instrumentation by loading eBPF probes to capture request rate, errors, and duration without changing application code. - It runs user-space agent plus small kernel programs, giving rich telemetry with minimal runtime overhead. Transcript: Dotan Horovits Okay. Mario Macías Basically, OBI allows you to instrument your applications. What I mean by applications, I mean usually clients or services, and provide what we call red metrics, that is rate, errors, and duration. So you instrument at the application level the details of the communications between your applications. From that, this is not very different from any other OpenTelemetry SDK. The particularity is that Baylab doesn't require to reconfigure or rebuild, redeploy your application. You just drop OBI there, and it will use a technology that has been for some years in the Linux kernel named eBPF. EBPF basically allows you to inject some probes, some small programs into different parts of your system. So they are triggered. It's like a code-based debugger. Those endpoints, those probes are triggered on certain events. For example, when a new socket connection is open, when you receive traffic, when a given user-provided function is triggered. So from those endpoints, when those e programs are triggered we can read read the memory so you can we can see what's going on and start reconstructing information about your system Dotan Horovits Traces so it's i think it's important to emphasize that it happens in in kernel space so although you monitor the applications in this case, but it runs and this part of its powerful qualities Is that it runs in kernel space, right? Mario Macías Yes, it runs in kernel space. So you will need some privileges, some extra privileges to run OBI so it can hook into the kernel. Particularity of EVPF is that it's a just-in virtual machine that pre-verifies your code. So it takes care for dangling for potential, dangling pointers or null pointers. So it can make sure it won't load your program unless it makes sure that your program is safe that prevents you from hanging your system if if you load an ebpf program with a bug and that's Dotan Horovits Very important because as i said it's it's kernel space on on the one hand it's very powerful because you have direct access to a lot of the information on the protocol, the endpoint invoked And all. But then again, if something is messed up in the program, it's kernel space. It doesn't crush just your application. It crushes the entire thing. So the safety measures there are definitely a crucial feature of eBPF. Mario Macías Yes, yes but i think we never hanged uh i mean even developing we never hang the the the kernel it just you you just say that your program crashing saying wait this is not we cannot verify The safety of these of these code and yeah I'm pretty confident that project in EVPF software. Dotan Horovits So yeah, and obviously it's an open source project, obviously for the EVPF. And as you mentioned, it's baked into the Linux kernels and actually expanding beyond Linux. So it's pretty, because of the ecosystem that has been built around it, I think it's definitely much more robust than maybe four or five years ago when I started covering it for these Use cases. And I want to go back again to OBI, to, again, OBI for OTEL eBPF instrumentation, just for those who haven't picked up on the acronym. And essentially, so you instrument the app layer. I'm just summarizing for our audience. Although it's a program running on KernelSpace, it essentially provides you auto-instrumentation of your applications running, whether server, service, or client, and the communication Between them. And providing you, you mentioned the red metrics, so the request rate, error rate, duration, the classic golden signals that everyone works. So it gives sort of the baseline telemetry that people usually want for basic monitoring of their application, right? Mario Macías Yes. Just to be a bit more precise, OBI is not fully running in the kernel space. It's another user space application with few privileges, so it can load small programs in the kernel that will store this information in some data named eBPF maps. So from the application layer, it reads all the information and then reconstructs the traces and provides all the metadata. ([Time 0:09:09](https://share.snipd.com/snip/817cce5c-e709-43fd-a958-5dab8929f1d0)) - **Kernel Power With Safety Guards** - eBPF runs verified programs inside the kernel, requiring privileges but preventing unsafe code from crashing the system. - The verifier rejects unsafe programs so eBPF preserves kernel stability while enabling powerful probes. Transcript: Mario Macías Yes, it runs in kernel space. So you will need some privileges, some extra privileges to run OBI so it can hook into the kernel. Particularity of EVPF is that it's a just-in virtual machine that pre-verifies your code. So it takes care for dangling for potential, dangling pointers or null pointers. So it can make sure it won't load your program unless it makes sure that your program is safe that prevents you from hanging your system if if you load an ebpf program with a bug and that's Dotan Horovits Very important because as i said it's it's kernel space on on the one hand it's very powerful because you have direct access to a lot of the information on the protocol, the endpoint invoked And all. But then again, if something is messed up in the program, it's kernel space. It doesn't crush just your application. It crushes the entire thing. So the safety measures there are definitely a crucial feature of eBPF. Mario Macías Yes, yes but i think we never hanged uh i mean even developing we never hang the the the kernel it just you you just say that your program crashing saying wait this is not we cannot verify The safety of these of these code and yeah I'm pretty confident that project in EVPF software. ([Time 0:11:02](https://share.snipd.com/snip/b4de508b-6226-4790-9dd4-ddb13d693576)) - **Use OBI For Low-Overhead Telemetry** - Use OBI when you want instant, language-agnostic telemetry without redeploying applications. - Prefer eBPF instrumentation to reduce overhead in interpreted languages like Ruby or Python. Transcript: Dotan Horovits As you mentioned that before, no code, without code changes, this is very, very crucial because for those who have tried to do the manual instrumentation directly with the OTL APIs, That's a lot of code that you need to add beyond your business logic. Mario Macías So here we're talking about something that is a no code instrumentation right yes yes i think the the main advantage of of obi is the the no code so and you can get instant instrumentation Uh from the very beginning without having to instrument everything and redeploy it the other advantage is the performance compared with some sdks for example for go for java you might Have a current sdks might be performant enough but we observed or some users complained that in some SDKs of some interpreted languages might add an extra overhead because all this Code, all the instrumentation code is running in the interpreted language. With eBPF, since this is just-in compiled, you can also, with low-latency instrument, or get basic instrumentation of your services, even if they are written in Ruby or Python or other Interpreted languages. ([Time 0:14:56](https://share.snipd.com/snip/0b7fbcab-a688-4f5f-98c2-d1b1f5a6484b)) - **Protocols Need Explicit Parsers** - eBPF sees raw bytes, so OBI must implement protocol parsers explicitly to support new protocols. - Community contributions expanded support from HTTP/GRPC to Kafka, Redis, SQL, and MongoDB. Transcript: Mario Macías So sometimes I want to clarify that sometimes people think that EVPF is some kind of magic. So you say, you tell EVPF, please bring me the data and you get all the data. This is this is not real you what you see from evpf is just a bunch of bytes you need to know how the data is structured at the binary level so if you want to support new protocols you need to Explicitly add them it's not like for example in an SDK, with traditional SDKs, you are working at the programming language level. So you get the information, you know which arguments you have, you can code in your target language. With ABPF, it's a bit more difficult. You work in C, you just see a bunch of bytes. Need to understand them so you need for every new every new protocol you need to add explicit protocol we started with http and https which are the most common protocols and expanded It to to grpc but recently thanks thanks to OVI being open source we are having new contributors that are expanding some the number of protocols like Kafka, MongoDB, we already provided SQL but now the SQL support has been expanded and improved. I'd say that even we have Redis. So yeah, the number of protocols is growing. ([Time 0:20:17](https://share.snipd.com/snip/8cc6dd4a-8764-4d7e-a329-6d97949851db)) - **Tracing Faces Context Limits** - Tracing from eBPF is harder than metrics because context propagation often requires modifying requests or language-specific hooks. - OBI supports spans but distributed trace propagation remains limited and case-dependent. Transcript: Mario Macías Yes, we are providing traces also. Trace support is still a bit limited. So we manage to support distributed tracing. I mean, as long as you get red metrics, you can get single spans. So you get a single span. But for distributed metrics, sorry, distributed traces, being able to propagate the context through traces, this is a bit difficult to do from from ebpf because this involves not Only reading reading the information but also writing the information modifying the the user the user request to attach this trace parent. This is not possible in all the languages, especially in insecure communications, because you cannot rewrite the output but we we managed to get some tricks to to relate uh which uh When we measure separate spans we we did some tricks to try to relate them and then report them as part of of the same trace But this is, we should see case by case whether it is support or Not. Same as red metrics are generally available, traces it's still a bit more limited. We hope to keep expanding also the tracing support for more frameworks. ([Time 0:23:25](https://share.snipd.com/snip/c0ce8d40-706e-4e36-8cd8-9d58ee8c1dce)) - **Unexpected Service Connections Revealed** - Users discovered unexpected service communication via OBI, revealing gossip protocols and hidden traffic paths. - These findings exposed real production connections that static architecture diagrams missed. Transcript: Mario Macías It will get the basic trace information and then it will decorate it with some extra metadata like Kubernetes metadata. And then it exports it as OpenTelemetry or Prometheus. Also supports network flow informations that is agnostic to to your application you won't see application level information but for some custom protocols uh that don't get direct Support from the SDKs, you still can know that this data flow in from one endpoint to another into your cluster. What kinds? Dotan Horovits Can you give us concrete examples just to understand what kinds of use cases you found useful for that? Mario Macías Yeah, for example, one is a use case not at the application observability level, but at the network monitoring level. You can get flows or the flows going through your cluster. The communication between aggregated by node, by process, by pot. So you get similar to netflow information and it's also useful for for example to monitor your network costs in your in your cluster you can see external external network traffic but Also you can see traffic in your cluster within different availability zones which have a an extra cost so you can you can do this kind of measurements also we we found some users coming Saying i this this ovi is inventing the the data they are showing me this look at this service graph this is showing data that is not, that shouldn't be, that those connections shouldn't Exist. And then we discovered that maybe this piece of software that came from a third party was using some gossip protocol to communicate between different instances. So it makes you be aware of some traffic that cannot be seen via traditional instrumentation because you might miss it. ([Time 0:35:55](https://share.snipd.com/snip/54092e25-5ef7-4814-9a9d-7991142bc4cc)) - **Prefer Daemonset Deployment** - Deploy OBI as a daemonset for efficiency and full-node visibility in Kubernetes clusters. - Use sidecar mode only when you need instrumentation for a small subset of pods. Transcript: Mario Macías Yeah, the preferred way is a daemon set. So from one single instance, you will instrument all the processes in your system, the ones that you selected. Sidecar mode is also supported. So they run in the same pod. And this OBI instance will only see data of the pod where it is located. We generally recommend daemon set mode because it's more efficient due to economy of scale and a simpler deployment. But it might happen that you only want OBI for a very concrete application that is not deployed in all your nodes, only in a few subset of them. So you consider that it's not worth deploying something globally. So in that case, you can deploy it as a sidecar. ([Time 0:40:07](https://share.snipd.com/snip/6bf7e9f1-ccc9-43be-ae53-ab654aae53e0)) - **User-Space Enrichment And Export** - OBI enriches raw kernel data by mapping PIDs to container and Kubernetes metadata in user space. - It exports metrics via Prometheus and primarily uses OTLP for OpenTelemetry compatibility. Transcript: Mario Macías It it happens in the user space of the agent. The agent has to connect to the Kubernetes informers, to the Kube API, and get a snapshot of all the metadata into your system. So when you get the raw data from the kernel, only get information about process IDs, basically. Which process ID you have. So then you start need to do some matching, match this process ID with your container ID. So once you get that container ID, you can match it with the Kubernetes pod, deployment, and so on. Yeah, there is some internal matching. Dotan Horovits And another point that you mentioned briefly, just want to pause on that, the side of exporting the data to backends of choice. You mentioned Prometheus, you mentioned others. So which backends are supported? Mario Macías Where can I today send my data and in which formats uh you can export for metrics information you can export it in in prometheus and but but our our main our main exporter is OpenTelemetry. ([Time 0:41:29](https://share.snipd.com/snip/2c2dfa67-8f09-4db9-b491-c0dc4117deaf)) - **Bela Donated, Yet Continues Downstream** - Grafana donated most of Bela's code to create OBI while keeping Bela as a downstream distro with extra features. - OBI is feature-complete and Bela now builds on OBI as a library for Grafana Cloud integrations. Transcript: Mario Macías Yes, we've got many times this question. So we wanted to donate Bayla to OpenTelemetry, but there were some parts of Bayla that wouldn't fit into that project. So we donated 80, 90% of our code to OpenTelemetry. So all the basic application and network level instrumentation, the code relies on OBI. So OBI is feature complete. Everything I've mentioned until now runs in OBI. But there were some other features that we still wanted to to keep so bayla still is still existing and provide some extra features like process level metrics an easier seamless integration With grafana cloud for example and is the downstream distribution of OBI that Grafana is commercially supporting, despite we are still contributing to OBI. Actually, the warranty that they want separate paths is that all the code we donated to OBI has been removed from Bayla. So now OBI supports running as an executable, but it can be also run as a library. So Bayla uses OBI as a library. So when we want to add a new feature to Bayla, we need to add it first to OBI. An improvement, an evac fix. So once it's in OBI, it is automatically incorporated into Bela. ([Time 0:44:06](https://share.snipd.com/snip/a74bbb1a-eed0-41c4-a18b-0e7b95918a37)) - **Combine OBI With SDKs** - Combine OBI with SDKs rather than replace them; use SDKs for deep app-level telemetry and OBI for complementary system and network signals. - Coordinate OBI and SDKs to avoid duplicated metrics and enrich SDK telemetry with eBPF-derived details. Transcript: Mario Macías So you can get, because even if the SDKs are more complete, there is also some information that can be seen from EVPF that cannot be seen from the SDKs, like internal runtime information, Network level information, and so on. So we would like to get both SDKs and OBI working together without, for example, duplicating information. There's also some work to do with the system metrics SIG, in order to standardize those, the network metrics we already provide are not standard. So we will need to provide a proper semantic convention so OBI works together within the open telemetry ecosystem. Dotan Horovits Sounds good. And you mentioned already about the SDK, so maybe this is a good segue to talk about roadmap. So beyond, obviously, the official release, that is the immediate focus, can you tell us anything about what's upcoming for the project and the SIG beyond that? And the SDKs that you mentioned? Yes. Mario Macías On one side, we are working on expanding the number of protocols, the number of languages, the accuracy of the data, the improved the tracing, the distributed tracing support. But also another point to start working is to let OBI working better, interacting better with the SDKs. ([Time 0:53:56](https://share.snipd.com/snip/0014e11e-f6ef-42f2-abd1-d6812102fa08))