From Group Science Project to Enterprise Service: Rethinking OpenTelemetry  ## Document Notes ## Highlights - **Reactive Observability Fails MTTR** - Observability tools mostly alert after systems are already broken and don't significantly reduce MTTR. - Ari Zilka argues proactive operations are needed to act on telemetry before failures cascade. Transcript: Ari Zilka Sure. So first and foremost, MyDecisive is what we call proactive operations. It's designed to be a robot that does all your change controls for you. Why that? Because I had too many customers asking me, can you really reduce MTTR when I was at New Relic? And the honest answer was nobody was reducing MTTR. It's not a dig on New Relic. Nobody was reducing MTTR significantly. And that's because by the time you're broken, it's too late. What you want to do is make sure that the humans aren't in the loop making the changes and that you make every production change staring at your telemetry while you make the change. ([Time 0:02:12](https://share.snipd.com/snip/806b9cc2-eeda-49e2-9e32-5be7f369177c)) - **Automate Rollbacks And Rollforwards** - Automate change decisions using AI so systems can roll forward or roll back without humans staring at graphs. - Use runtime telemetry-driven actions to reduce human-in-the-loop delays and cut MTTR. Transcript: Ari Zilka Who better to stare at telemetry and make a change than an AI, right? Everyone wants an AI observability tool, but what all the vendors here, I counted, there's 23 at this show. What they're all showing you is a dashboard that has a magical red piece of text on it. Like that's the broken thing. It isn't the broken thing except a contrived use case is Alex. What really is easier to do is say that is a broken change, roll it back, or that is a healthy change, roll forward, keep making the changes, add it to more nodes. ([Time 0:02:54](https://share.snipd.com/snip/ca89bdea-2c3a-4b5a-8840-3bb3efdce6ee)) - **Otel As An On-Prem Enterprise Service** - MyDecisive runs as an on-prem 'bump in the wire' using OpenTelemetry, Kubernetes, Prometheus, NATS, and HAProxy. - It preserves customer control by running in the customer's environment and being open source. Transcript: Ari Zilka Then? Well, that's why we're here at KubeCon. It's open telemetry inside. It's Kubernetes. It's Prometheus. It's NATS. It's HAProxy. So it is exactly the same application footprint that your own apps are. And I had to make it that way because it runs on-prem. You run it. We don't run it. ([Time 0:04:30](https://share.snipd.com/snip/07911ad9-11b3-49bb-af98-51c2d8170ab3)) - **Streaming Media Built A Huge OTEL Team** - A large streaming media customer ran 10,000 OpenTelemetry instances after migrating from New Relic and Datadog. - They ended up building a 40–50 person OTEL team to run the stack and own platform engineering. Transcript: Ari Zilka I was working with a very large streaming media outlet. They have 10,000 instances of OTEL today. But if you go backwards to 2023, they had New Relic and Datadog. I was talking to the platform engineering team, and they said, we want more control. What can New Relic do? What can Datadog do better? And I was one of those two vendors and I said to them, hey, you know what? Let's put Otel in the middle. The kind of control you want should be done by Otel. And they said, well, can I send all my Datadog data and my New Relic data to one vendor through Otel? And the answer is yes, you can. Unfortunately for them, they went off and built an OTEL practice and community internally. So they have like 40, 50 people running their OTEL stack, and it is their platform engineering team. But the punchline here, Alex, is that with New Relic and Datadog, all they were doing is managing a vendor contract. They were begging teams, please use these observability tools, they will help you. The teams were saying, okay, I might not use it. But now with OpenTelemetry, they're in a two-way conversation. They're on the same side of the table instead of ops versus dev. How come? Because OTEL requires so much setup, programming, and maintenance that can only be done in the application code. ([Time 0:07:08](https://share.snipd.com/snip/75fd5ec3-cc7b-49de-bcef-be1e228872c5)) - **OTEL Creates Significant Platform Toil** - Large companies often staff 20–50 people to run OTEL, turning telemetry into a platform engineering burden. - MyDecisive aims to decouple that operational toil and make OTEL a platform choice rather than heavy custom code. Transcript: Ari Zilka But everywhere I go that has open telemetry without my decisive, you have 20, 30 person teams if the company is a multi-billion dollar company. If it's a hundred million dollar company, it's got two or three OTEL experts. But OTEL is being babysat by platform engineering teams. They love it because now they have a piece of software to program, code, and own, but they hate it because they are in locking arm in arm with dev to get it right. And I've decoupled it back to a platform selection, back to a platform team decision, but with all the power of open telemetry. ([Time 0:08:49](https://share.snipd.com/snip/ff31c337-8480-423a-bb97-b4996bdce497)) - **Introduce OTEL Without Disrupting Developers** - Keep existing vendor agents and drop MyDecisive as a wrapper to avoid disruptive migrations. - Let platform teams control OTEL centrally while developers remain mostly oblivious to telemetry plumbing. Transcript: Ari Zilka Well, step one in MyDecisive is keep your current agent. If a platform engineering team picks open source, open telemetry without the MyDecisive wrapper around it, they've got to slowly migrate their way away from the vendor. With me, keep the vendor in place. You don't have to ask the devs to do anything, so it becomes an autonomous decision. Secondly, we are giving the OTEL team a more powerful interface to open telemetry that is programmable and easy to live with. They don't have to worry about scale. They don't have to worry about production footprint. They instead worry about logical behaviors, runtime behaviors, automation opportunities like, hey, I want to do a release. And if the release hurts the system, roll it back. ([Time 0:09:34](https://share.snipd.com/snip/1a8ddd00-b0af-40d7-ad54-b8fb3a86d99d)) - **Filter Telemetry But Toggle During Incidents** - Deploy a filtering layer that you can toggle during incidents so you get full fidelity only when needed. - Avoid touching application code during incidents by managing OTEL pipelines on the wire. Transcript: Ari Zilka You mentioned we're a bump in the wire. That's exactly correct. Just drop us in like a network device, and then we start snooping the data, and we start suggesting, hey, can I filter this data for you? Humans don't have to put in an OTEL filtering pipeline. And then when you filter the data, you can come to me and say, hey I am in an incident. Stop filtering the data. I need full fidelity in Datadog or Splunk so I can debug it. You push a button in my system to stop it from filtering. ([Time 0:11:11](https://share.snipd.com/snip/f477fa49-7ec2-43ea-b14f-101271fc5c98)) - **If-This-Then-That For Explainable Automation** - A programmable 'if this, then that' interface on the telemetry plane lets automation run safely without humans in the loop. - This design yields explainable automation where engineers can see conditions and actions clearly. Transcript: Ari Zilka What we've done is we realized that what you need is an if-this interface to open telemetry. Open telemetry, you declare, I want to filter data dog. I want to reroute new relic. I want to scrub PII data from Splunk. And then you hire an OTEL developer and they write that pipeline for you. And then they call the app teams and make sure it's not breaking the telemetry. With my decisive, it discovers things on its own, but it's all built on this fundamental construct of if this, then that. So it's got the right hooks, the right interface where a human doesn't have to join the loop. I think five years ago before modern AI, we talked about human in the loop a lot. I don't want a human in the loop. They told me they don't want to be in the loop. They don't want to look at a dashboard. The last thing I'm going to do is move their eyeball from a dashboard to an open telemetry server. So I had to run myself. If I build an if this, then that, it's very easy for humans to explain to themselves what actions I'm going to take under what conditions. ([Time 0:11:59](https://share.snipd.com/snip/8122ba87-f452-4596-8bb3-7ac9e5cda143)) - **Supports Multiple Observability Backends** - MyDecisive supports different best-of-breed observability backends like Conviva, Honeycomb, and CloudWatch. - The product centralizes control so millions of OTEL instances can be managed as a single enterprise instance. Transcript: Ari Zilka Have teams who are using Conviva because it has the best in class streaming media tools. We have teams who are using Honeycomb for distributed tracing. We have teams that are using CloudWatch on AWS, RAW, to just watch the infrastructure, watch their lambdas, etc. And trying not to pay an observability vendor at all. Those are all the layers of the stack from the infrastructure all the way up to the user or client application. Otel has, you know, more power to it. It has, it has woven its way into the entire stack. Right. And so we get that free on board. Alex Williams So with the scale of, of Otel, it provides an opportunity, a window for your service, really. Yeah. Because then you can put your service on the wire. Yeah. And wherever Otel is, you know, your service is looking at what it's doing. And then it says there's problems over here with the New Relic data or the data data or the Honeycomb data. Ari Zilka Or the Peacock player or the Netflix player or the Disney Plus player, all the way in someone's house. And the most important part is that enterprise scale I keep emphasizing, enterprise class solution. You could control it from a single control point. ([Time 0:14:40](https://share.snipd.com/snip/2995816f-e4c6-40d1-b9e5-2f7ec97b8370))