"Observabilitying" the Future of Software With Charity Majors  🔗 **Source:** ["Observabilitying" the Future of Software With Charity Majors](https://share.snipd.com/episode/cfcd533f-5935-43a3-9863-1340c5b8a5aa) ## Highlights - **Traces Unite Telemetry Into a Single Source** - Observability is most powerful when telemetry types are united, not fragmented. - Charity Majors explains traces/structured data unify metrics and logs so you can reason across requests instead of treating signals as separate exhaust pipes. Transcript: Charity Majors Oh boy. You know, it's been really fun in the last two months. It feels like people have realized that you don't need metrics, logs, traces, all of the different signals. Actually, you just need traces. You just need structured data that is connective, connectable. You could call that traces with spans. You could call that, you know, wide structured logs, whatever. What you need is there's actually so much more power in having you data united, not fragmented. And metrics and logs, they will always be with us, but they're really, they're the exhaust pipe of your infrastructure, right? It's for the telemetry that you don't control. You can't change it. You just have to operate it, right? Those are your metrics and logs. But for your software, the software you do control, your telemetry is a product problem. ([Time 0:01:49](https://share.snipd.com/snip/3e1e2f72-0c55-4f6b-85be-5291b24eb087)) - **Debugging a Windows App By Observabilitying It** - Scott used an agent-based observability approach like debug print statements at scale to fix a Windows app quickly. - He built an MCP server and looped an agent over telemetry to reproduce and solve a visual UI issue within minutes. Transcript: Scott Hanselman This is one of my superpowers. And one of the things that's worst about me is my ability to make really bad analogies. But I feel like observability and then putting in the context of agents is like got here debugging at scale. Like before debuggers, before attaching debugger and stepping through a debugger, we would just put like console.writeline, got here. Console.writeline, got here, number two. And then you would just, your whole life was just a series of bisects as you're like, it's between here and here. And I was recently doing some agent work on a Windows app and it was very visual. And someone said, oh, you should use computer use. And I was like, no, you're going to take screenshots pixels and you're going to burn carbon and waste energy. This feels like a debug dot write line situation. So I just observability the hell out of the entire system and then made an MCP server that pointed to it. And then the agent looped on it and solved it within minutes. It was very much a got here debugging at scale problem. ([Time 0:03:12](https://share.snipd.com/snip/59ea608b-ea63-45bb-a858-859d821f0fbc)) - **Auto Instrumentation Lets You Ship With Telemetry** - Auto-instrumentation and OTEL make it feasible to build faster with telemetry than without. - Charity Majors says modern tooling can close feedback loops so engineers get faster, actionable signals during development. Transcript: Charity Majors And with OTEL and auto instrumentation now, I think it is genuinely possible to build faster with telemetry than without, which is huge. And I think underappreciated, you know, I don't blame software engineers for, I have the theory that like the entire 20 year journey of DevOps was really all about us trying to create One feedback loop that included both people writing the code and the code in production. And it completely failed because it was just too hard. ([Time 0:05:49](https://share.snipd.com/snip/19d21763-c13f-4eeb-ad44-f06adc33bdc5)) - **Invest In Observability To Close Feedback Loops** - Invest in observability to create feedback loops between code changes and production behavior. - Charity Majors advises that without observability you lack the connection that lets teams learn from deployments and improve systems. Transcript: Charity Majors Know what creates a feedback loop? Observability. Otherwise, you just have a bunch of things happening and a bunch of things causing things. It's only if you observe the connection between the two that you have a feedback loop. ([Time 0:12:18](https://share.snipd.com/snip/6e7a1dda-7d9f-4900-b816-422373df3f0e)) - **Design Telemetry As Part Of Your Product** - Treat observability as a product and instrument deliberately with context. - Charity Majors recommends designing telemetry as part of the software so you can slice and reassemble data for debugging and dashboards. Transcript: Charity Majors And then parts two and three are about, for practitioners, software engineers, how to instrument your code and how to understand your telemetry. And we've got two tracks there's doing it kind of i think of them as uh coke zero and coke classic doing it with uh ai and doing it you know without ai they're both uh they're they're both Really solid i had nothing to do with them but they're i think they're pretty good um and then we have uh part four and part five one One of them is deep dives into like use cases. ([Time 0:13:21](https://share.snipd.com/snip/d2ab61f6-1efd-462e-beb9-f69c75f10c62)) - **Credibility Trumps Authority For Technical Change** - Transformations succeed when senior engineers earn deep credibility and trust inside the org. - Charity Majors emphasizes influence, reciprocity, and demonstrating reliable results to cut through bureaucracy and drive change. Transcript: Charity Majors And we might do that. But yeah, because what CTO is going to like start two thirds of the way through the book? Yeah. Scott Hanselman Don't bury the lead. Don't bury. Charity Majors So it's systems thinking. And then there's some stuff about how to drive change in an organization with influence, not formal power, how to partner effectively. My actual favorite chapter in the book comes almost at the very end. And it's my favorite, not because I think it's brilliant, but because I don't think I've ever heard anyone write about this, seen anyone write about this. But it's when I think about how improbable every transformation is, like most of them fail, the vast majority of technical transformations fail. The ones that succeed, what do you need to make them succeed? Well, as a very senior engineer, you need trust and credibility inside your organization. People need to believe that when you say something, it is true. It will happen. It will be backed up. That cuts through bureaucracy like a hot knife through butter. It's one of the only things it does, that deep credibility. And when it comes to partnering with vendors, like when it comes to you partnering with people in your team, you need to understand that you're all on the same team. Everybody has their needs, but you're all in pursuit of the same goal and you need to keep reminding people of this, right? ([Time 0:15:16](https://share.snipd.com/snip/19cb972e-680c-4983-8889-dfc777af1b56)) - **Traces Are The New Unit Of Work** - The unit of work has shifted from single transactions to traces that span hours and multiple agents. - Charity Majors describes timelines and long-lived traces for agent-driven workflows like chat supervisors spawning many API calls. Transcript: Charity Majors It for most of my career, the unit of work has been the transaction. And it just isn't now. The unit of work is usually the trace, but the trace is just structured data. One of the things that I find really frustrating about people who have been in observability for a long time is that they get really wrapped around the axle about signal types often. And it's just data. They're just differently enveloped, wrapped bits of data, right? And the more fungible those data types are, the more powerful they typically are. And so I think that, you know, we just shipped something, I think it went like yesterday into TA called timelines, which is so cool because, you know, you can imagine if you, if you, if You're, if you're like intercom or fit, you, you build like this chat product, right? Uh, the interaction begins when someone asks a question and then you might have like a supervisor agent that kicks off and spawns many different agents. And each of those agents does a bunch of API calls and MCP calls and storage calls and everything. And then, know, at some later point in time, they converge, they bring back an answer and they return it. And then the user says something else and it kicks off another. And like knitting together all those bits, it's not a transaction. That might take place over hours, right? And so the primitives have changed and our need to like up the stack has changed. ([Time 0:19:26](https://share.snipd.com/snip/a6af5f5d-8a87-4b91-b42c-b341623f6ea7)) - **Treat Every Deploy As A Measured Production Experiment** - Acknowledge testing-in-production and build safe tooling like canaries and feature flags. - Charity Majors advises using progressive delivery and observability so each prod experiment can be detected and rolled back safely. Transcript: Charity Majors The only question is whether or not you acknowledge that that's what you're doing and build tooling to do it safely. There is no replacement for reality. Tested or examined or validated every single moment of intersection of software, infrastructure, deploy process, time, data, users, every single one of those is unique. There is no test for reality. There is only practice runs, right? You're always going to be tested by... Scott Hanselman It's Schrodinger's production. Charity Majors It really is. Scott Hanselman You just have to open it and find out the cat's okay. Charity Majors And none of this is an argument to not test. But I feel like we all have limited attention cycles. We all have limited engineering cycles, even with AI. And I feel like most people get so wrapped around the axle of making sure it's perfect that they're under-investing in the other side of it, which is how am I going to find it? How am I going to trace it? How am I going to react to it? ([Time 0:23:36](https://share.snipd.com/snip/a755f62d-13c6-44c2-bb5d-4110fe9658ef)) - **Use Feature Flags With High Cardinality Tracing** - Combine feature flags with high-cardinality observability to get true testing superpowers. - Charity Majors explains enabling a single app and tracing every request lets you compare behavior at one-in-a-million granularity. Transcript: Charity Majors Like the combination of feature flags and the kind of, the kind of high cardinality observability data that Honeycomb does, they're greater than the sum of their parts. I know a lot of people have used feature flags when all they have are metrics and aggregates and log lines. And feature flags are still useful, but they're not superpowers. But it is a superpower when you can break down by one in a million apps and enable just that app and then compare it against the baseline of all the other apps and trace every request. Like it's just, yeah, you can test in prod. Then it's like, why wouldn't you test in prod? ([Time 0:25:44](https://share.snipd.com/snip/273874ae-f940-45f6-a23a-ce1760d58874)) - **Keep Hiring Juniors To Scale Engineering Capacity** - Don’t stop hiring juniors; invest in early-career people for future capacity and impact. - Charity Majors argues companies that only hire seniors are effectively trying to automate people away and miss scalable leverage. Transcript: Scott Hanselman If a company says we only hire seniors now because AI does the easy stuff, what do you say to them? It tells me they're just trying to automate all people away. Okay. You don't like that. Totally unacceptable. Charity Majors I mean, I am a person and I would like to have a job. So yeah, I don't think that's very smart. Good. Scott Hanselman Okay. So if an A is an AI more like a junior engineer or an intern with bad judgment, or should we not anthropomorphize it at all? ([Time 0:26:33](https://share.snipd.com/snip/3b2efaec-9c87-461b-931a-bf130d9f939f)) - **On Call Sharpens Empathy Through Customer Pain** - Being on call aligned engineers with customer pain and taught empathy rapidly. - Charity Majors says putting developers on call makes pain a teacher and increases care for production quality. Transcript: Charity Majors I'm just kidding. I mean, it's, how important is empathy is kind of like saying do we really need oxygen let me rephrase how did being on call being on prod teach you about empathy that's the real question That is the thing that i love about being on call is how much it aligns you with the pain that your customers are going into we were talking about feedback loops earlier and i have told many People that the way that they get their developers to care is by putting them on call. Because when you align, when the pain is separated and you don't, you're not a good developer, but when your pain is aligned with your customers, it pain is nature's teacher. ([Time 0:28:04](https://share.snipd.com/snip/aa2b2ec3-421c-4d7a-9803-5bd0d4c5bea9)) - **AI Tutors Amplify Junior Engineers' Productivity** - Junior engineers plus AI tutors create large leverage for senior engineers who see many fixable problems. - Charity Majors and Scott note juniors use AI as a 24/7 tutor and can implement the many smaller but impactful fixes seniors identify. Transcript: Charity Majors In a different way but like most of them have just this running dialogue all day long with they have a 24 7 tutor personally devoted to them you know and and it helps them figure out how to Ask interesting questions and interrupt their more senior counterpart. Yeah. I just, they're people are, people are great. Yeah. Scott Hanselman I'm, I'm really looking forward to spending the last kind of thousand days of my career spending as much time as I can with early in career people and then going and teaching again at community College. Charity Majors And was just talking to jay gengel back from versell and he was saying you know he was expressing the same thing he's like man i see so many more bugs that i can fix i see so many more problems That i could i can't build all the things that i see and and i can't ask agents to go build those things or solve those things because they're too they're too big they're too they're the Right size for a more junior or mid-level engineer. And I could have so much more impact if I just had people to do this. And I was just like, I think that's the best pitch for junior engineers that I've ever heard. ([Time 0:30:00](https://share.snipd.com/snip/9ccc52d2-8bb1-4206-b2f1-746d0aa86059))