I Can't Believe Nobody's Done This Before...

I Can't Believe Nobody's Done This Before... ![rw-book-cover](https://i.ytimg.com/vi_webp/sFEDAkJy9Dc/maxresdefault.webp) 🔗 **Source:** [I Can't Believe Nobody's Done This Before...](https://www.youtube.com/watch?v=sFEDAkJy9Dc) > [!note]+ Document Notes > tags: #websockets #openai #api #performance #ai-agents #tool-calling #networking #infrastructure > author: Theo - t3․gg > source: YouTube > reference: https://www.youtube.com/watch?v=sFEDAkJy9Dc > date created: 2024-12-19 16:30 > > # 🏷️ Title: OpenAI's Revolutionary WebSocket API Change: 90% Bandwidth Reduction for AI Agents >  ## 🧰 Action Inbox - [H1] Test WebSocket implementation for agent workflows — evaluate performance gains #websockets - [H2] Implement persistent connections for tool-heavy workflows — reduce context resending #api-optimization - [H3] Adopt open responses standard across providers — ensure compatibility #standardization ## 🧭 Table of Contents - [[#📝 Summary]] - [[#📖 Main Content]] - [[#🗒️ Notes]] - [[#📚 References]] - [[#🗣️ Paraphrase]] - [[#📰 Newsletter]] - [[#✅ Action Items]] - [[#🧪 Hypotheses]] - [[#📊 Methodology]] - [[#❓ Questions]] - [[#🔮 Future Research]] - [[#💡 Implications]] - [[#🧠 Learning Styles Assessment]] - [[#📊 Accessibility Assessment]] - [[#🌍 DEIB Assessment]] - [[#✅ Logical Validity]] - [[#🛡️ DARVO Analysis]] - [[#🎯 Conclusion]] ### 🔗 References - [OpenAI Responses API](https://platform.openai.com/docs/guides/responses) - [Open Responses Standard](https://github.com/openai/openai-responses) ## 📝 Summary - OpenAI moved from REST to WebSockets for their API, achieving 90%+ bandwidth reduction and 20-40% speed improvements - The change addresses inefficient context resending during tool calls in AI agent workflows - WebSockets guarantee persistent connections to the same server, enabling stateful interactions - OpenAI open-sourced the standard, potentially making this the new industry default - Most beneficial for agentic workflows with multiple tool calls, less impactful for simple chat ## 📖 Main Content OpenAI has implemented a fundamental architectural shift from REST APIs to WebSockets for their AI services, addressing a critical inefficiency in how AI agents handle tool calls and context management. The core problem stems from how AI models are stateless - every tool call requires resending the entire conversation history because models don't retain state between requests. In traditional agent workflows with multiple tool calls, this means sending megabytes of context data repeatedly for responses that might only be a few tokens. WebSockets solve this by maintaining persistent connections to the same API server, enabling stateful interactions. Instead of routing each request to potentially different servers (requiring full context resending), the connection stays with one server that can cache the conversation state and only process new information. The technical implementation leverages WebSockets not as a superior protocol, but as a guarantee of server affinity. This eliminates repeated authentication checks, cache lookups, and context validation on every tool call. OpenAI has open-sourced this as the "Open Responses" standard, increasing the likelihood of industry-wide adoption across providers like Anthropic and Google. ## 🗒️ Notes #### ⏱️ [00:00](https://youtu.be/sFEDAkJy9Dc?t=0) — [H1] OpenAI's massive API change announcement - Key idea: OpenAI switched from REST to WebSockets with dramatic performance improvements - Evidence: "90 plus% bandwidth reduction and speed improvements by 20 to 30%" #### ⏱️ [02:45](https://youtu.be/sFEDAkJy9Dc?t=165) — [H2] Understanding context management in AI models - Key idea: AI models require full conversation history on every request due to stateless nature - Evidence: "Everything. Everything. Yep, you guys got it. Follow-up request isn't just you sending your new message. It's you sending the whole thing" #### ⏱️ [05:30](https://youtu.be/sFEDAkJy9Dc?t=330) — [H3] The tool call bandwidth problem - Key idea: Every tool call resends entire history, creating massive redundancy - Evidence: "Every single tool call sends the whole history back" #### ⏱️ [07:15](https://youtu.be/sFEDAkJy9Dc?t=435) — [H2] Why AI models need complete context - Key idea: Models are stateless and "die" between tool calls, requiring full context restoration - Evidence: "The AI isn't running when the tool call happens. The AI is paused effectively. It's dead. It's killed. It's off" #### ⏱️ [09:00](https://youtu.be/sFEDAkJy9Dc?t=540) — [H3] Cache limitations don't solve bandwidth issues - Key idea: Caching reduces compute but not data transfer since cache keys are context hashes - Evidence: "The cache is just to reduce the compute the model has to do...it's not changing how much data you send" #### ⏱️ [11:30](https://youtu.be/sFEDAkJy9Dc?t=690) — [H2] OpenAI's infrastructure routing challenges - Key idea: Requests route to different API servers and GPUs, preventing state persistence - Evidence: "Your first request might go to the first API box. Your second request might go to the fourth API box" #### ⏱️ [13:45](https://youtu.be/sFEDAkJy9Dc?t=825) — [H3] Inefficiency of processing large contexts for small outputs - Key idea: Systems process megabytes of context to generate just a few words - Evidence: "It might literally be sent two megabytes of text and then respond with eight words" #### ⏱️ [15:20](https://youtu.be/sFEDAkJy9Dc?t=920) — [H1] WebSockets as server affinity guarantee - Key idea: WebSockets ensure requests hit the same server, enabling stateful interactions - Evidence: "The websocket is less a protocol here, more a guarantee" #### ⏱️ [17:00](https://youtu.be/sFEDAkJy9Dc?t=1020) — [H2] Benefits of persistent connections - Key idea: Eliminates repeated authentication, cache lookups, and context validation - Evidence: "You don't have to recheck your auth. You don't have to resend the whole context" #### ⏱️ [18:30](https://youtu.be/sFEDAkJy9Dc?t=1110) — [H1] Open source standardization - Key idea: OpenAI open-sourced the responses standard for industry adoption - Evidence: "They actually fully open sourced it. Open responses is a standard that is inspired by the OpenAI responses API" #### ⏱️ [20:15](https://youtu.be/sFEDAkJy9Dc?t=1215) — [H3] Use case limitations for simple chat - Key idea: Benefits primarily apply to agent workflows, not basic chat applications - Evidence: "The benefit here is not huge in like the chat app use case" #### ⏱️ [21:00](https://youtu.be/sFEDAkJy9Dc?t=1260) — [H2] Performance improvements for agents - Key idea: 20-40% speed improvements for workflows with 20+ tool calls - Evidence: "speeds up agentic runs with 20 plus tool calls by 20 to 40%" #### ⏱️ [22:30](https://youtu.be/sFEDAkJy9Dc?t=1350) — [H1] Opportunity for infrastructure innovation - Key idea: Early AI ecosystem enables rethinking fundamental architecture patterns - Evidence: "Everything in our stack from how we network to how we send requests to how we store our code in Git is all up for change right now" ## 📚 References - [OpenAI Platform Documentation](https://platform.openai.com/docs) - [Open Responses Standard Repository](https://github.com/openai/openai-responses) - [WebSocket Protocol Specification](https://tools.ietf.org/html/rfc6455) ## 🗣️ Paraphrase - OpenAI replaced their REST API with WebSockets to solve massive inefficiencies in agent workflows where the same context gets resent on every tool call - WebSockets provide persistent server connections that enable stateful conversations, eliminating redundant data transfer and authentication checks - The change delivers 90%+ bandwidth reduction and 20-40% speed improvements, particularly beneficial for agent workflows with many tool calls rather than simple chat ## 📰 Newsletter **Major Infrastructure Shift: OpenAI Moves to WebSockets** OpenAI just revolutionized AI API architecture with a shift from REST to WebSockets, delivering 90%+ bandwidth savings and 20-40% speed improvements for agent workflows. **Why This Matters for WordPress Developers:** - **Plugin Development**: AI-powered WordPress plugins using multiple tool calls will see dramatic performance improvements - **Content Generation**: Automated content workflows with agent-based systems become much more efficient - **Resource Optimization**: Reduced bandwidth means lower hosting costs for AI-integrated WordPress sites **Technical Implementation:** WebSockets maintain persistent connections, eliminating the need to resend entire conversation contexts on every AI tool call. This is especially valuable for WordPress AI features like automated SEO optimization, content analysis, or dynamic personalization. **Next Steps:** Consider how your WordPress AI integrations could benefit from persistent connections, especially for multi-step content processing workflows. Resources: [OpenAI API Docs](https://platform.openai.com/docs), [WordPress REST API](https://developer.wordpress.org/rest-api/), [WP-CLI](https://wp-cli.org/) ## ✅ Action Items #### DevRel - Developer advocacy: Create demos showing WebSocket implementation for WordPress AI plugins - Education: Develop workshops on stateful AI interactions for WordPress developers - Developer experience: Update documentation with WebSocket integration examples - Developer marketing: Highlight performance benefits in AI-powered WordPress campaigns - Community management: Gather feedback on WebSocket adoption challenges #### OSPO - Policy & Governance: Review implications of persistent connections for data handling policies - License Compliance: Evaluate open responses standard licensing compatibility - Security Management: Assess WebSocket security considerations for AI integrations - Software Inventory: Track WebSocket dependencies in AI-powered WordPress solutions - Community Engagement: Contribute to open responses standard development #### Community - Maintainers: Evaluate WebSocket support in AI-related WordPress plugins - Contributors: Create examples of efficient AI agent implementations - Sponsors: Support development of WebSocket-based AI tools for WordPress #### WordPress - Core: Consider WebSocket infrastructure for future AI integrations - Core Performance: Evaluate performance benefits for AI-powered features - Core AI: Implement WebSocket support in AI pilot projects - Plugins: Update AI plugin guidelines to recommend WebSocket usage - Documentation: Create guides for implementing persistent AI connections ## 🧪 Hypotheses - WebSocket adoption will become standard across all major AI providers within 6 months - Agent-based WordPress plugins will see 3-5x performance improvements with WebSocket implementation - Bandwidth cost savings will enable more sophisticated AI features in budget WordPress hosting - The open responses standard will accelerate cross-provider compatibility ### 📊 Methodology - Benchmark WordPress AI plugin performance before/after WebSocket implementation - Measure bandwidth usage reduction in real-world WordPress agent workflows - Track adoption rates across AI providers and WordPress plugin ecosystem ### ❓ Questions - How will WordPress hosting providers adapt infrastructure to support persistent WebSocket connections? - What security implications emerge from maintaining stateful AI connections in WordPress environments? - How will this change affect WordPress plugin development patterns for AI integrations? ### 🔮 Future Research - Impact of WebSockets on WordPress multisite AI implementations - Scaling patterns for persistent AI connections in high-traffic WordPress sites - Integration possibilities with WordPress's existing WebSocket implementations ### 💡 Implications - Practice: WordPress developers need to rethink AI integration architectures for efficiency - Policy: Hosting providers may need updated terms for persistent connection usage - Security: Stateful connections require new security considerations for WordPress AI features - Education: Developer training needs updating to include WebSocket-based AI patterns - Funding: Reduced bandwidth costs may enable more ambitious AI features in WordPress projects ### 🧠 Learning Styles Assessment Reading level estimate: 12th grade (technical content with specialized terminology) #### 👀 Visual learners - Present: Clear diagrams of request/response flows, context management visualization - Improve: Architecture diagrams showing WebSocket vs REST comparison #### 🔊 Aural learners - Present: Video format with detailed verbal explanations - Improve: Podcast-style technical deep dive on implementation details #### 🗣️ Verbal learners - Present: Excellent use of analogies (Groundhog Day for stateless models) - Improve: More real-world examples of WordPress implementations #### 🧩 Kinesthetic learners - Present: References to actual code implementation and testing - Improve: Step-by-step WordPress plugin conversion guide #### 🧠 Logical learners - Present: Clear cause-and-effect reasoning for architectural decisions - Improve: Performance benchmark tables and cost analysis #### 🧑‍🤝‍🧑 Social learners - Present: Community discussion prompts about adoption - Improve: Forum discussions on WordPress-specific implementations #### 🧘‍♀️ Solitary learners - Present: Detailed technical explanations for independent study - Improve: Self-assessment checklist for WebSocket readiness #### 🌿 Naturalistic learners - Present: Evolution metaphors for API development - Improve: Ecosystem thinking about interconnected AI services ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Well-structured video content with clear section breaks - Improve: Detailed transcript with technical diagram descriptions #### Keyboard navigation - Present: Standard YouTube video controls - Improve: Interactive transcript with timestamp navigation #### Contrast and readability - Present: Clear video presentation with good contrast - Improve: High-contrast code examples in supplementary materials #### Text resizing and reflow - Present: Responsive video player - Improve: Scalable transcript formatting #### Media and captions - Present: Auto-generated captions available - Improve: Professional captions with technical term accuracy #### Forms and interactions - Present: Standard video interface - Improve: Interactive code examples with accessible controls ### 🌍 DEIB Assessment #### Cultural breadth - Present: Global technical audience consideration - Improve: Examples from diverse WordPress communities worldwide #### Inclusive language - Present: Technical content uses clear, professional language - Improve: Define technical jargon consistently throughout #### Neurodiversity support - Present: Well-structured content with clear progression - Improve: Content warnings for rapid technical concept changes #### Representation - Present: Focus on technical merit over personalities - Improve: Highlight diverse developers contributing to standards #### Psychological safety - Present: Encouraging tone about learning complex topics - Improve: Acknowledgment that not everyone needs to understand every detail ### ✅ Logical Validity #### Strong reasoning - Deductive strength: Clear logical progression from problem to solution - Inductive strength: Good use of specific examples to support general principles - Causal reasoning: Excellent explanation of cause-effect relationships in API performance - Analogical strength: Effective use of analogies (Groundhog Day, routing examples) #### Fallacies - Appeal to authority: Minor reliance on OpenAI's claims without independent verification - Hasty generalization: Assumes all agent workflows will benefit equally #### Tone of voice - Overall tone: Enthusiastic but technical, educational and accessible while maintaining technical depth ### 🛡️ DARVO Analysis - Deny: No denial patterns present - Attack: No attacks on accusers or critics - Reverse victim/offender: Not applicable to technical content - Emotional cues and rhetoric: Positive enthusiasm for technical innovation, no manipulative rhetoric ## 🎯 Conclusion OpenAI's move to WebSockets represents a fundamental shift in AI API architecture that solves critical inefficiencies in agent workflows. By maintaining persistent connections and enabling stateful interactions, this change delivers dramatic performance improvements specifically for use cases involving multiple tool calls. For WordPress developers, this opens opportunities to build more sophisticated AI-powered features without the bandwidth and latency penalties that previously made complex agent workflows impractical. The open-source nature of the responses standard suggests this will become an industry-wide improvement rather than an OpenAI-specific advantage. The broader implication is that we're still early enough in the AI infrastructure evolution to rethink fundamental assumptions about how these systems should work, creating opportunities for significant improvements across the entire stack. ## --- ## Full Transcript [00:00] People often ask me, "If I could do any [00:02] video and not worry about the algorithm, [00:04] what would that video be?" To be honest, [00:06] most of the videos I want to do, I want [00:08] to do some amount because I like the [00:09] topic, but also some amount because of [00:11] the reach. But this is one I kind of [00:12] expect to bomb. Why am I doing it then? [00:15] Because I really care about this topic. [00:17] This is a huge change and it touches on [00:18] a ton of the nerdy weird details that I [00:21] love to dig into and hopefully it will [00:23] perform well. OpenAI just made a huge [00:26] change to their APIs. They are no longer [00:28] just doing everything through REST calls [00:29] like you know everyone is and they're [00:31] moving over to websockets. This is a [00:33] massive shift even though it might not [00:35] seem that way. We're talking about [00:36] reducing bandwidth by 90 plus% and speed [00:40] improvements by 20 to 30%. Which sounds [00:43] crazy just going from SSE to websocket, [00:45] right? Well, yeah. If you don't know how [00:48] AI models work, believe it or not, this [00:50] video is going to be a deep dive both [00:52] into networking technology and into the [00:55] weird characteristics of how AI models [00:57] and agents in particular actually do the [00:59] work they do. If you don't yet [01:01] understand the relationship between the [01:03] thing you're running in your terminal or [01:04] your editor and the actual servers that [01:06] are generating the outputs that go [01:08] there, this is going to be a very useful [01:09] video. And if you do already understand [01:11] that, this will probably be even more [01:12] useful and will be a good resource for [01:14] you to send to your co-workers who [01:15] almost certainly don't get any of this. [01:17] I am regularly surprised by just how [01:19] many developers don't understand the way [01:21] context works and how requests work back [01:23] and forth and any of that at all. And I [01:25] hope this will help you guys understand [01:27] it a lot better. But there's one other [01:28] thing I need you all to understand [01:30] first. How great today's sponsor is. How [01:32] many times have you had a change you [01:33] were ready to ship, but you were blocked [01:34] waiting for CI to build just for it to [01:36] finish and give you a random error you [01:38] had to go fix and then you're waiting [01:39] again for sometimes hours just to get [01:41] small changes out. That's because the CI [01:43] tools we're using suck. Unless you're [01:45] using today's sponsor, Blacksmith, they [01:47] make your CI comically faster. Even for [01:50] my simple TypeScript code bases, build [01:52] times are often cut in half from over 3 [01:54] minutes to under a minute for so many [01:56] projects I work on. And if you happen to [01:58] be using Docker, the 40 times faster [02:00] Docker builds are a godsend. All of your [02:03] Docker layers are cached on NVMe drives [02:05] on the same computer. So you end up with [02:07] way faster caching, downloading, and the [02:09] whole build process with Docker. They [02:11] have really good examples on the site, [02:12] including the Android GitHub actions, [02:14] which used to take 9 minutes to run on [02:15] official GitHub CI, but with a oneline [02:18] change, it's now 3 minutes, and it costs [02:20] a fourth as much. We've been using them [02:22] for all of our CI at ping, and the team [02:24] is hooked. Whenever a new repo appears [02:26] and we start adding CI, we realize we [02:28] forgot to add Blacksmith. We make the [02:29] one line of code change and suddenly our [02:31] builds take under a minute again. It's [02:32] great. And you'll see here, even for [02:34] some of our bigger projects, our builds [02:36] take consistently 50ish seconds. I would [02:39] go in depth on how good the [02:40] observability is, but I'll just sit here [02:41] and talk for hours. It's so much better [02:44] than anything GitHub has published in [02:45] decades. It's insane. If you actually [02:48] want to be able to debug your actions, [02:49] you need to move to Blacksmith. If you [02:51] want to pay half as much for your [02:52] actions, you should use Blacksmith. If [02:53] you want your CI to be way faster, you [02:55] should use Blacksmith. If you want your [02:56] Docker bills to be way faster, you [02:58] should use Blacksmith. Generally [02:59] speaking, you should probably check them [03:00] out at soy.link/blacksmith. [03:03] So, why am I so excited about this [03:05] websocket move? The first thing I want [03:07] to make clear is that we're not yet [03:08] using this for T3 chat. And honestly, [03:10] for the T3 chat use case, it doesn't [03:11] make a ton of sense. But there are other [03:14] places where this makes a absurd amount [03:17] of sense. First, I want to make sure we [03:18] understand how context management and [03:21] requests work with an AI model. So, we [03:23] have the system prompt up top here. We [03:25] then have the user message. Let's do a [03:29] traditional one like, I want to improve [03:31] the SEO of my application. Let's write a [03:34] thorough plan on how we're going to make [03:36] this app have better SEO. Classic [03:38] standard user message. And we'll have, I [03:41] don't know, red be the response from the [03:45] agent. And the agent says, "Okay, first [03:48] let's build a better understanding [03:52] of this codebase." So user makes a [03:55] prompt. Agent decides it needs to go [03:57] understand the codebase better. So what [03:59] does it do? It calls tools. I don't know [04:03] uh lsa or something where it's going to [04:05] list all of the stuff in a given [04:08] directory. What does that tool call? [04:10] Then it says interesting. The important [04:13] code is probably in source slash. So, [04:17] let's scan that next. You get the idea. [04:21] And after a bunch of those happen, we [04:24] end up with the final answer. Your SEO [04:26] has been improved. I edited D. You get [04:30] the idea. We've all built with agents. [04:31] We've all seen this. This is your [04:33] context in that moment. And obviously, [04:36] as things go, the context builds. You [04:38] have the system prompt at the top, then [04:40] the user message, then an agent message. [04:42] The agent message ends with a tool call. [04:44] The tool call fires, gets results, the [04:46] agent then responds, makes the next set [04:49] of tool calls, and then eventually you [04:51] get the actual result. You get the idea. [04:53] So, what happens when you want to do a [04:56] follow-up message? Like, I don't know, [04:59] let's say at the end here, you say that [05:01] looks good, but I want to change the [05:04] description or something like that. What [05:06] request gets sent to the OpenAI API when [05:11] you send this follow-up message? What is [05:14] contained within that? You've been [05:16] around for a bit, you know. And if we [05:18] look at our chat, you see it. All of it. [05:21] Everything. Everything. Yep, you guys [05:23] got it. Follow-up request isn't just you [05:26] sending your new message. It's you [05:28] sending the whole thing. But that's not [05:31] even where the pain point is. If that [05:33] was all, it wouldn't be too big a deal. [05:35] Sending all of this text once per user [05:37] message would be fine. What if I told [05:39] you it was sent significantly more than [05:41] that, though? What if I told you it got [05:44] sent here when you sent the user [05:46] message, here when the first tool call [05:48] completed, here when the second tool [05:50] call completed, and then again when you [05:51] sent the new prompt. Every single tool [05:54] call sends the whole history back. And [05:56] it makes sense if you think about it [05:58] because here we have the agent call. [06:01] Okay, first let's build a better [06:03] understanding of the codebase. and it [06:04] outputs a tool call. The agent is just [06:07] autocomplete. It's just generating [06:09] words. It doesn't know what's happening. [06:11] It doesn't know what system it is. It [06:12] doesn't know how long it will take. [06:14] People seem to think that when the tool [06:16] call happens, a pause occurs, but that [06:18] is not what's happening. A tool call [06:21] means the generation finished. The AI [06:23] isn't running when the tool call [06:25] happens. The AI is paused effectively. [06:28] It's dead. It's killed. It's off. And [06:30] when the tool call is done, everything [06:32] from here up gets sent back to the AI, [06:36] sent back over the API call to whatever [06:38] service is running it. And now it has a [06:40] new history that it is using to continue [06:42] generation from. Somebody asked, is this [06:44] to verify the tool call worked? No, it's [06:46] to have the information from the tool [06:47] call. The tool call was list stuff. It [06:50] was an ls call. So now there's an [06:52] output, but it can't do anything until [06:54] it has that output. So it just stops. So [06:57] every time a tool call happens, the [06:58] model needs to know what the tool call [07:00] result is. So once the tool call is [07:02] done, it then sends everything back [07:04] because it needs everything. It's almost [07:05] like a Groundhog Day type thing where [07:07] every time the model wakes up, there's [07:09] nothing left in its brain. It's [07:10] stateless. It has nothing going on. So [07:12] all of the state has to be shipped back [07:14] every single time. So if you like watch [07:16] your network traffic when you're doing a [07:19] long agent run, every one of these tool [07:21] calls is sending the whole history back [07:24] up to the anthropic API to the OpenAI [07:26] API to whatever else. And then whatever [07:28] GPU is available takes in all of that [07:31] context and then loads it for the next [07:33] response. And I already know what you [07:36] guys are thinking. Well, isn't that what [07:37] caching's for? The cache is just to [07:39] reduce the compute the model has to do, [07:41] which makes it way cheaper and also [07:43] faster to get the first token, but it's [07:45] not changing how much data you send [07:47] because the key for that cache is a hash [07:50] of your history. So if this part's all [07:52] been cached, you're not telling the AI, [07:56] hey, go pull from that cache and then [07:58] add this to it. You're passing the whole [08:00] history and then it hashes it and says, [08:02] oh we have the history up to this [08:04] point cached. The new part is here. So, [08:07] we're going to load the state of things [08:09] up until that new thing. Then, we're [08:12] going to add the new thing. Then, we're [08:13] going to start responding. That's what [08:14] the cache does. The cache does not [08:17] change how much data you send. The cache [08:19] just changes how long it takes to [08:21] process the data. It's effectively a [08:23] hash for that. So, the cache is just a [08:25] compute reduction method. Is not [08:27] changing what traffic has to be sent at [08:28] all. Want to make sure that's very, very [08:30] clear. Someone asked if this is what [08:32] compaction is. And no, compaction is [08:34] very different. Compaction couldn't be [08:36] more different. Compaction is taking the [08:38] entire history and summarizing it so you [08:41] don't need all of the tokens anymore. [08:43] But compaction breaks the cache. It does [08:45] not help the cache. Compaction is [08:47] choosing to forgo the cache because you [08:49] want to have a shorter history. Anyways, [08:51] hopefully you can somewhat see the [08:53] problem. Now, the problem isn't that [08:55] it's too hard for the model to get this [08:57] data. It's very simply that sending all [09:00] of this up every single time a tool call [09:03] happens. A growing amount of data every [09:06] single time just kind of sucks. It's [09:09] really inefficient. It's really [09:10] expensive. It uses a shitload of [09:13] bandwidth. It means running a ton of [09:14] agents in parallel is more expensive [09:16] than it needs to be. It means OpenAI's [09:18] APIs need to be able to handle these [09:20] giant text payloads and the routing and [09:22] all of those things. Wouldn't it be much [09:24] easier if instead of sending all of [09:27] this, it only had to send the new thing, [09:30] the tool call. Wouldn't that be cool? [09:32] But that makes things more complex [09:34] because effectively OpenAI's [09:36] infrastructure doesn't really allow [09:38] that. So OpenAI has a bunch of GPUs, but [09:40] you're not accessing the GPUs. You're [09:42] not sending this request straight to a [09:44] GPU, obviously, because then we have to [09:46] format everything entirely differently. [09:47] It make no sense. And also that GPU [09:49] might not have your specific cache on [09:51] it. So effectively all of these requests [09:54] have to go through an orchestration [09:55] layer, an API if you will. So in front [09:58] of OpenAI's GPUs, they have this API. [10:01] And of course, they don't have one GPU, [10:02] they have thousands of them, tens of [10:04] thousands of them across lots of [10:06] different places. So your request goes [10:08] to the API. Once your request goes to [10:10] the API, it checks things. It checks [10:12] like, do we have a cached value for [10:14] where we were before here? Do we have a [10:16] GPU open that we can route this to? Does [10:18] this user have permission to do this [10:20] thing? And then it routes whatever it [10:22] concludes here over to a GPU. And if [10:25] that goes well, it responds with those [10:27] tokens. And those tokens then get sent [10:29] back to you, which ends up being the [10:31] agent response. But also remember the [10:34] API is not as simple as it may seem [10:36] here. This is also lots of separate API [10:39] boxes that all look for random available [10:42] GPUs and route to those. So your first [10:45] request might go to the first API box. [10:47] Your second request might go to the [10:48] fourth API box and then this one might [10:50] go to a different GPU entirely. You see [10:54] the problem here? Even if the servers [10:57] were to maintain this state, making sure [10:59] the right one has the right context is [11:01] obnoxious. And I'm pretty sure the [11:03] caching layer is a whole separate third [11:06] layer thing that has to manage all of [11:07] this. So when you send the request, it [11:09] goes to a server that OpenAI runs that [11:11] is a relatively dumb server that they [11:13] have tons of that has to go check which [11:16] cache might have the cached value for [11:18] your tokens and then load that state [11:21] into a GPU, load your prompt into the [11:23] GPU, generate tokens, and then respond. [11:27] And get this, it has to do all of that [11:29] for generations that are sometimes only [11:31] 10 tokens. Since this API layer doesn't [11:34] have your history, doesn't know what you [11:36] sent before because every additional [11:38] request gets routed somewhere else, you [11:40] have to resend the whole thing because [11:41] otherwise it loses it. They could try to [11:43] like attach an ID the whole time and [11:46] keep everything cached always, but they [11:47] would never know how long they have to [11:49] keep it around. The latency problems [11:51] there of making that globally scaled [11:52] across it, like not viable, not viable [11:55] at all. The architecture for that would [11:57] have to be an external data store that [11:59] all the APIs connect to that every [12:02] single request has to be cached in. Not [12:05] viable. Not viable at all. Not for that [12:07] amount of data. And not for things where [12:09] half the time it doesn't even matter. [12:10] And remember, sometimes these responses [12:12] are going to be really short like maybe [12:14] it's wrong directory trying. [12:18] This is a real thing. The agent might [12:20] respond. The agent might ingest a [12:22] 100,000 tokens that you sent over API. [12:25] It might literally be sent two megabytes [12:28] of text and then respond with eight [12:31] words. That's a real thing that happens [12:34] to us every single day. You are [12:36] processing so much data for just a few [12:39] tokens to be generated. That's so [12:41] wasteful. And there's basically nothing [12:42] you can do at this layer that keeps it [12:45] scaled as well as it is that also [12:47] persists the state externally. So it [12:50] doesn't matter which you hit throughout. [12:52] There is a solution though. What if you [12:55] could guarantee that every single [12:57] request hits the same box? What if you [13:00] could be sure that tool call 2 goes the [13:03] same box that tool call one did? What if [13:05] you could be sure that all of these [13:07] follow-ups, all of these things [13:09] happening during this session are [13:10] hitting the same box? Then you wouldn't [13:12] have to load all of this from storage. [13:14] You wouldn't have to check what's cached [13:15] you already know. You wouldn't even have [13:17] to send up all of the context. You could [13:19] just send the new message or just the [13:21] new tool call. You wouldn't have to send [13:22] much of anything. But it would have to [13:24] be stateful. The way that OpenAI's API [13:27] worked previously was effectively [13:29] stateless. These boxes didn't know [13:31] anything. They had to go look up data. [13:34] They had to go check the database to see [13:36] if you had a cache entity or not. They [13:37] had to go check the O server to make [13:39] sure you were allowed to do the thing. [13:40] And every single tool call had to [13:42] doublech checkck every single thing. [13:45] Every request had to check everything. [13:47] But if the first request had everything [13:49] covered, does the second request really [13:50] need all of that? Like what if it just [13:52] knew after the first request who you [13:54] were, what you were doing, what cache [13:55] you could hit. This is why we're talking [13:57] about websockets today. The value of [13:59] websockets here isn't that it's a better [14:01] protocol. It's not that it is magically [14:03] faster cuz websockets are this gift from [14:05] the heavens. It is very simple. [14:07] Websockets can guarantee that you're [14:09] hitting the same box. Since you're [14:10] maintaining the connection to that API [14:13] server through the whole generation, you [14:15] don't have to worry about it losing its [14:16] state. You don't have to worry about [14:18] rechecking off. You don't have to worry [14:20] about knowing what's cached or not. The [14:22] websocket is less a protocol here, more [14:25] a guarantee. The guarantee is that as [14:28] this generation is running, you can keep [14:30] hitting the same server box that is now [14:32] capable of keeping track of what you've [14:34] done so far. You don't have to recheck [14:36] your O. You don't have to resend the [14:38] whole context. You just send what's new [14:40] and then it can immediately connect to a [14:42] GPU, feed everything over there, and [14:43] start responding. This reduces how much [14:45] data you send. This reduces how much [14:47] data has to be processed. This reduces [14:49] how many checks have to be done, how [14:50] much cache has to be touched, how many [14:52] things have to occur from when the tool [14:54] call completes on your machine to when [14:56] the API is aware of what's going on. [14:59] That is awesome. And it is insane it [15:01] took this long for something like this [15:02] to happen. And I am very, very, very [15:04] thankful OpenAI are the ones to do it [15:06] because OpenAI's implementation for [15:08] these things tends to become the [15:10] standard. Not only are OpenAI standards [15:12] cloned by pretty much everyone, they [15:14] actually fully open sourced it. Open [15:17] responses is a standard that is inspired [15:19] by the OpenAI responses API. Binds a [15:21] shared request response model, streaming [15:22] semantics, and tool invocation pattern. [15:24] So clients and providers can exchange [15:26] structured inputs and outputs in a [15:27] consistent shape. You can use this shape [15:30] for calling the APIs for everything from [15:32] anthropic to Gemini. And now it's an [15:34] open standard. And sadly, no, the [15:37] standard does not have the websocket [15:39] piece implemented yet. I would expect [15:42] this to be implemented very, very soon. [15:46] Yeah, I'm very thankful who is involved [15:48] with all of this. This being implemented [15:50] the way it was, the place it was, [15:52] massively increases the chance that this [15:53] does not just become standard number 72 [15:56] and instead becomes a new default that [15:59] these agentic tools can use. And to be [16:01] very very clear, just to be [16:02] extraordinarily clear, the benefit here [16:04] is not huge in like the chat app use [16:08] case because persisting that connection [16:10] is not worth it if you're not following [16:12] up immediately. So the second user [16:15] message here is probably not going to go [16:17] over the same websocket and it's [16:18] probably not going to hit the same API [16:19] box. And that's fine. Reloading all of [16:21] that context once per user message is [16:24] totally reasonable. Doing it once per [16:26] tool call where sometimes one user [16:28] message could spawn hundreds of tool [16:30] calls that is much less okay. And if [16:33] you're curious about the results, this [16:34] is what OpenAI had to say. And as [16:36] somebody who was lucky enough to test [16:37] this early, I can confirm it might even [16:38] be crazier than what they're sharing [16:40] here. Websockets keep a persistent [16:41] connection to the responses API, [16:43] allowing you to send only new inputs [16:44] instead of sending round trips for the [16:47] entire context on every turn. By [16:49] maintaining inmemory state across [16:50] interactions, it avoids repeated work [16:52] and speeds up agentic runs with 20 plus [16:54] tool calls by 20 to 40%. And you can see [16:57] what this looks like in their little [16:58] demo video. [17:08] Do you understand how massive this is? [17:15] There are so many things that matter in [17:17] this space that aren't just how quick [17:19] can the model generate tokens. When we [17:21] say we're early to an extent, this is [17:23] the stuff we're talking about. On one [17:25] hand, this is so cool. Like, yeah, we [17:27] now have a way to keep things in state [17:29] and massively improve the performance. [17:31] On the other hand, how crazy early are [17:34] we that we were just okay with sending [17:36] the whole context every single time this [17:39] type of thing was generated because we [17:40] just stapled on tool calling to existing [17:43] APIs and standards. Like there's so much [17:45] opportunity here. I hope this gets you [17:47] excited to build. There's so much space [17:50] to make useful improvements across the [17:52] entire stack. So many things we just do [17:55] a certain way because that's how it [17:56] started. So many tools we still rely on [17:58] because they're just what we're used to. [18:00] Everything in our stack from how we [18:02] network to how we send requests to how [18:04] we store our code in Git is all up for [18:06] change right now. This is really fun. To [18:10] those thinking that deep tech planning [18:12] and thought and process are dead as a [18:14] result of AI, you're wrong. It matters [18:16] more than ever. We get to rethink [18:18] everything from first principles right [18:20] now. And it's fun. It's genuinely fun. I [18:22] was so hyped when OpenAI told me about [18:24] the changes they were making here. I was [18:25] really lucky to get to play with it a [18:27] bit early. And I'm so thankful that I [18:29] get to share this all with you guys now [18:30] because as nerdy as this is and as much [18:32] as I expect this video to bomb, I really [18:35] hope it does well because this stuff is [18:36] so much cooler than just whatever the [18:38] new model is. I am genuinely more [18:39] excited about this than Codex Spark than [18:41] about Sonnet 46 than about basically any [18:44] of the openweight models that have come [18:45] out recently. I'm personally way more [18:46] hyped about the small API change. Yeah. [18:49] Thank you all for nerding out with me. I [18:51] hope you enjoy this as much as I do. [18:52] Until next time, peace nerds.  📄 **Full text:** ![[Full Document Contents/Articles/I Can't Believe Nobody's Done This Before...|I Can't Believe Nobody's Done This Before..., YouTube]] ## Highlights - result is. So once the tool call is done, it then sends everything back because it needs everything. It's almost like a Groundhog Day type thing where every time the model wakes up, there's nothing left in its brain. It's stateless. It has nothing going on. So all of the state has to be shipped back every single time. So if you like watch your network traffic when you're doing a long agent run, every one of these tool calls is sending the whole history back up to the anthropic API to the OpenAI API to whatever else. And then whatever GPU is available takes in all of that ([View Highlight](https://read.readwise.io/read/01kjwygxchcyterb6fny8f13p7)) - Tags: 👻 ai highlighted - context and then loads it for the next response. And I already know what you guys are thinking. Well, isn't that what caching's for? The cache is just to reduce the compute the model has to do, which makes it way cheaper and also faster to get the first token, but it's not changing how much data you send because the key for that cache is a hash of your history. So if this part's all been cached, you're not telling the AI, hey, go pull from that cache and then add this to it. You're passing the whole ([View Highlight](https://read.readwise.io/read/01kjwygxdcwm9n0z6762rx2g5j)) - Tags: 👻 ai highlighted - history and then it hashes it and says, oh we have the history up to this point cached. The new part is here. So, we're going to load the state of things up until that new thing. Then, we're going to add the new thing. Then, we're going to start responding. That's what the cache does. The cache does not change how much data you send. The cache just changes how long it takes to process the data. It's effectively a hash for that. So, the cache is just a compute reduction method. Is not changing what traffic has to be sent at all. Want to make sure that's very, very ([View Highlight](https://read.readwise.io/read/01kjwygxecerm3fn3gmf4fm5bv)) - Tags: 👻 ai highlighted - anything. They had to go look up data. They had to go check the database to see if you had a cache entity or not. They had to go check the O server to make sure you were allowed to do the thing. And every single tool call had to doublech checkck every single thing. Every request had to check everything. But if the first request had everything covered, does the second request really need all of that? Like what if it just knew after the first request who you were, what you were doing, what cache you could hit. This is why we're talking about websockets today. The value of websockets here isn't that it's a better ([View Highlight](https://read.readwise.io/read/01kjwygxgsvx58geb9s72exzfq)) - Tags: 👻 ai highlighted