It's not just you (gpt-5 got dumber) 🔗 **Source:** [It's not just you (gpt-5 got dumber)](https://www.youtube.com/watch?v=05YgaYY2V2M) > [!note]+ Document Notes > tags: [ai-models, openai, gpt, performance-regression, technical-investigation] > author: Theo - t3․gg > source: YouTube > reference: https://www.youtube.com/watch?v=05YgaYY2V2M > date created: 2025-01-01 00:00 > > # 🏷️ Title: It's not just you (gpt-5 got dumber) > <!-- deep-read:sections start --> ## 🧰 Action Inbox - [H1] Test feedback command in CLI — use /feedback for bug reports #openai - [H2] Minimize MCP tools setup — reduce context bloat for better performance #optimization - [H3] Start new conversations frequently — avoid compaction degradation #workflow ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ### 🔗 References - [OpenAI Codex Performance Investigation](https://www.youtube.com/watch?v=05YgaYY2V2M) - [Depot Build Optimization](https://depot.dev) ## 📝 Summary - OpenAI investigated user reports that GPT-5 Codex appeared to be performing worse after September 15th launch - Investigation found multiple small issues rather than one major regression: hardware differences, compaction problems, timeout issues, and bugs - OpenAI created dedicated team, dogfooded their own product, audited 60+ feature flags, and implemented fixes - They refunded credits, reset rate limits, and established permanent team to monitor real-world performance - Demonstrates transparent approach to user feedback and technical debt resolution ## 📖 Main Content OpenAI's Codex team conducted a comprehensive investigation into user reports of degraded performance following the GPT-5 Codex launch. Rather than dismissing user feedback, they assembled a dedicated squad to investigate systematically. The investigation revealed no single major issue but instead found multiple contributing factors: older hardware performing differently, compaction (context summarization) degrading with repeated use, timeout escalation inefficiencies, and subtle bugs in constrained sampling that caused language switching mid-response. Key findings included hardware differences where "slight differences in how a processor handles a bit of data might skew the way tokens are being resolved," compaction issues where performance degrades with multiple auto-compactions in long sessions, and infrastructure optimizations that inadvertently created edge cases. The team's response demonstrates industry best practices: dogfooding by using the same setup as external users, systematic hypothesis testing, transparent communication, and financial accountability through refunds and credit resets. ## 🗒️ Notes #### ⏱️ [00:00](https://www.youtube.com/watch?v=05YgaYY2V2M?t=0) — [H1] GPT-5 Performance Complaints Surface - Key idea: Multiple users reported GPT-5 Codex feeling "dumber" after September 15th launch - Evidence: "A lot of people have been reporting this and the OpenAI team is taking it really seriously" #### ⏱️ [03:21](https://www.youtube.com/watch?v=05YgaYY2V2M?t=201) — [H2] OpenAI's Investigation Plan - Key idea: OpenAI created systematic investigation plan despite no internal metrics showing issues - Evidence: "Despite no immediate evidence from our own usage or topline product metrics, we felt that it was something worth the full-time attention of some of our best engineers" #### ⏱️ [05:15](https://www.youtube.com/watch?v=05YgaYY2V2M?t=315) — [H2] Dogfooding Implementation - Key idea: All employees switched to same setup as external users - Evidence: "All employees, not just the Codex team, will go through the exact same setup as all of our external traffic" #### ⏱️ [06:42](https://www.youtube.com/watch?v=05YgaYY2V2M?t=402) — [H2] Feature Flag Audit Results - Key idea: Team removed massive number of feature flags to reduce complexity - Evidence: "We started by auditing and removing over 60 feature flags. We're in the process of removing 80 more" #### ⏱️ [07:33](https://www.youtube.com/watch?v=05YgaYY2V2M?t=453) — [H1] Hardware Performance Differences - Key idea: Different hardware configurations caused varying model performance - Evidence: "We trained a predictive model to look at relationships between feedback reports and serving cluster hardware" #### ⏱️ [08:45](https://www.youtube.com/watch?v=05YgaYY2V2M?t=525) — [H2] Compaction Performance Issues - Key idea: Context summarization degraded with repeated use in long conversations - Evidence: "Evals confirmed that the performance degrades with the number of compactions used within a single session" #### ⏱️ [10:12](https://www.youtube.com/watch?v=05YgaYY2V2M?t=612) — [H3] Apply Patch Tool Problems - Key idea: Model resorted to deleting and recreating files when patch application failed - Evidence: "We found some reports of the model resorting to deleting and recreating files when it fails to apply a patch" #### ⏱️ [11:30](https://www.youtube.com/watch?v=05YgaYY2V2M?t=690) — [H2] Tool-Influenced Model Training - Key idea: Future models being trained specifically to work better with existing tools - Evidence: "It really feels like we are shifting to a world where we're letting the tool influence each other" #### ⏱️ [12:45](https://www.youtube.com/watch?v=05YgaYY2V2M?t=765) — [H3] Timeout and Latency Issues - Key idea: Model was retrying tasks with escalating timeouts inefficiently - Evidence: "We did notice that the model was retrying tasks with longer timeouts" #### ⏱️ [14:20](https://www.youtube.com/watch?v=05YgaYY2V2M?t=860) — [H2] Constrained Sampling Bug - Key idea: Subtle bug caused token sequences to become out of distribution - Evidence: "We found a subtle bug that led token sequences to become out of distribution for the model" #### ⏱️ [15:15](https://www.youtube.com/watch?v=05YgaYY2V2M?t=915) — [H3] Language Switching Bug - Key idea: Constrained sampling bug caused mid-sentence language switches - Evidence: "Reported issues where the model would switch languages mid-sentence could be attributed to this bug" #### ⏱️ [17:50](https://www.youtube.com/watch?v=05YgaYY2V2M?t=1070) — [H2] MCP Tool Complexity Warning - Key idea: Users setting up too many MCP tools causing context bloat - Evidence: "We continue to recommend using minimalist setups and keeping conversations small and targeted" #### ⏱️ [19:25](https://www.youtube.com/watch?v=05YgaYY2V2M?t=1165) — [H1] Permanent Performance Team - Key idea: OpenAI establishing dedicated team for ongoing performance monitoring - Evidence: "Carrying momentum into a permanently staffed team that will obsess every day about real world performance" #### ⏱️ [20:10](https://www.youtube.com/watch?v=05YgaYY2V2M?t=1210) — [H2] Credits and Refunds - Key idea: OpenAI refunded usage and reset rate limits after finding billing bugs - Evidence: "They refunded all Codex credit usage and reset rate limits for everyone" #### ⏱️ [21:30](https://www.youtube.com/watch?v=05YgaYY2V2M?t=1290) — [H3] User Behavior Evolution - Key idea: Users pushed tool to harder tasks, creating perception of degraded performance - Evidence: "Codex is so good that people kept trying to use it for harder tasks and it didn't do those tasks as well" ## 📚 References - [OpenAI Codex Internal Investigation Document](https://www.youtube.com/watch?v=05YgaYY2V2M) - [Depot Performance Optimization Platform](https://depot.dev) - [OpenAI Feedback Command Documentation](https://openai.com) ## 🗣️ Paraphrase - OpenAI took user complaints about GPT-5 Codex performance seriously and launched comprehensive investigation despite internal metrics showing no issues - Investigation revealed multiple small problems rather than single regression: hardware variations, context compaction degradation, timeout inefficiencies, and sampling bugs - Company implemented systematic fixes, established permanent monitoring team, and provided financial remediation to affected users ## 📰 Newsletter OpenAI's response to user reports of degraded GPT-5 Codex performance offers valuable lessons for WordPress developers and agencies managing client expectations around AI tools. When users reported that Codex felt "dumber" after its September launch, OpenAI didn't dismiss the feedback despite their internal metrics showing no problems. Instead, they assembled a dedicated investigation team and systematically tested every component. **Key findings relevant to WordPress developers:** - **Hardware matters**: Even slight processor differences affected model outputs - similar to how WordPress performance varies across hosting environments - **Context accumulation hurts**: Long conversations with multiple "compactions" degraded performance - paralleling how WordPress sites slow with accumulated plugins and customizations - **Tool complexity creates problems**: Users with many MCP tools experienced worse performance due to context bloat **Practical applications:** - Monitor your AI tools' performance over time, not just initial impressions - Keep AI conversations focused and start fresh threads for new tasks - Minimize unnecessary integrations and tools that bloat context The investigation demonstrates the importance of user feedback channels and systematic debugging - principles equally valuable for WordPress site optimization and client communication. Resources: [OpenAI Feedback Documentation](https://openai.com), [WordPress Performance Guide](https://learn.wordpress.org/), [AI Tool Integration Best Practices](https://developer.wordpress.org/) ## ✅ Action Items #### DevRel - Developer advocacy: Create content around AI tool performance monitoring and debugging methodologies - Education: Workshop on systematic investigation techniques for performance regressions - Developer experience: Implement better feedback mechanisms in developer tools - Developer marketing: Case study on transparent incident response and user communication - Community management: Establish channels for collecting and triaging performance feedback #### OSPO - Policy & Governance: Develop guidelines for AI model performance monitoring and user feedback integration - License Compliance: Review AI tool dependencies and performance obligations in service agreements - Security Management: Assess AI model inference security across different hardware configurations - Software Inventory: Document AI model versions and hardware dependencies in SBOM processes - Developer Support: Training on AI model debugging and performance optimization techniques - Community Engagement: Share performance investigation methodologies with upstream AI projects - Strategic Alignment: KPIs for AI tool reliability and user satisfaction metrics #### Community - Maintainers: Establish systematic approaches to investigating user-reported performance issues - Contributors: Create frameworks for reproducing and debugging non-deterministic performance problems - Sponsors: Transparent reporting on tool performance and infrastructure investments - Ecosystem: Coordinate performance monitoring across related AI tool integrations #### WordPress - Core: Apply systematic debugging approaches to performance regression investigations - Core Performance: Implement monitoring for AI-assisted development tool performance - Core AI: Establish feedback mechanisms for WordPress AI feature performance - Community: Create resources for debugging AI tool performance in WordPress development - Training: Lesson plans on systematic troubleshooting and user feedback integration - Documentation: Document best practices for AI tool performance monitoring in WordPress workflows ## 🧪 Hypotheses - AI model performance varies significantly based on hardware configuration even with identical software stacks - Context accumulation through summarization creates degradation that compounds over long sessions - User behavior evolves to push tools beyond optimal use cases, creating perception of performance regression - Systematic dogfooding and user feedback integration prevents prolonged performance issues ### 📊 Methodology - Controlled hardware comparison testing across different inference configurations - Longitudinal analysis of session length and compaction frequency vs performance metrics - User behavior analysis comparing task complexity over time periods - A/B testing of internal vs external user experience parity ### ❓ Questions - How can organizations detect performance regressions in non-deterministic AI systems before user complaints? - What infrastructure monitoring is needed to identify subtle hardware-dependent AI performance variations? - How should AI tools balance context retention with performance degradation in long sessions? ### 🔮 Future Research - Development of real-time AI model performance monitoring across diverse hardware configurations - Investigation into optimal context management strategies that balance continuity with performance - Analysis of user behavior patterns that lead to suboptimal AI tool utilization ### 💡 Implications - Practice: Implement systematic feedback collection and performance monitoring for AI development tools - Policy: Establish guidelines for AI model performance consistency across infrastructure variations - Security: Consider hardware-dependent AI model behavior as potential security consideration - Education: Train development teams on AI tool optimization and performance debugging techniques - Funding: Budget for dedicated AI tool performance monitoring and systematic investigation processes ### 🧠 Learning Styles Assessment Reading level estimate: 12th grade (technical content with industry jargon) #### 👀 Visual learners - Present: Screenshots of internal investigation document, performance charts - Improve: Hardware architecture diagrams, performance regression timelines, visual debugging workflows #### 🔊 Aural learners - Present: Full video with narration and technical explanation - Improve: Podcast-style summary focusing on investigation methodology #### 🗣️ Verbal learners - Present: Clear technical definitions, analogies to familiar concepts - Improve: Glossary of AI inference terms, comparison with traditional software debugging #### 🧩 Kinesthetic learners - Present: Step-by-step investigation process, actionable debugging checklist - Improve: Interactive debugging exercises, hands-on AI performance testing labs #### 🧠 Logical learners - Present: Systematic investigation methodology, cause-and-effect analysis - Improve: Decision trees for performance debugging, quantitative impact assessments #### 🧑‍🤝‍🧑 Social learners - Present: Discussion prompts about user feedback integration - Improve: Team exercises for systematic incident response, collaborative debugging scenarios #### 🧘‍♀️ Solitary learners - Present: Self-directed analysis of investigation methodology - Improve: Individual reflection questions on performance monitoring practices #### 🌿 Naturalistic learners - Present: Ecosystem thinking about AI tool performance interdependencies - Improve: Analysis of AI tool performance in broader technical environment context ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Clear heading structure, descriptive timestamps - Improve: Alt text for any visual elements, detailed video descriptions #### Keyboard navigation - Present: Standard video controls, logical tab order - Improve: Keyboard shortcuts for video sections, accessible transcript navigation #### Contrast and readability - Present: Good text contrast in video player - Improve: High contrast mode support, adjustable text size in transcript #### Text resizing and reflow - Present: Responsive video player design - Improve: Scalable transcript text, flexible layout at zoom levels #### Media and captions - Present: Full transcript provided - Improve: Time-synced captions, downloadable transcript formats #### Forms and interactions - Present: Standard video playback controls - Improve: Accessible feedback forms referenced in content ### 🌍 DEIB Assessment #### Cultural breadth - Present: Technical content accessible globally - Improve: Multi-language examples, international infrastructure considerations #### Inclusive language - Present: Professional technical communication - Improve: Define industry jargon, avoid assumptions about technical background #### Neurodiversity support - Present: Structured content with clear sections - Improve: Content warnings for long technical discussions, alternative summary formats #### Representation - Present: Technical focus without identity barriers - Improve: <!-- deep-read:sections end --> ## 📺 Video  --- # 📄 Full Transcript *Auto-generated from YouTube captions · timestamps roughly every 30 seconds · [watch on YouTube](https://www.youtube.com/watch?v=05YgaYY2V2M)* **[00:00]** Have you noticed that GPT5 feels a bit dumber lately? If so, you're not alone. A lot of people have been reporting this and the OpenAI team is taking it really seriously. If this feels like déja vu, I'm sorry. We've covered things like this in the past. Like when I first tried GBT5, thought it was great and then everybody questioned me. Then they came around and realized it was actually pretty good when they tried it in the right ways and in the right places. And then there was also the Claude models having weird degraded performance too. Both Opus and Sonnet and Enthropic kind of gaslighting us about it for a while until eventually admitting that things were worse than expected. Thankfully, that has not been what's going on here. **[00:31]** The Open teams have been going really hard trying to find anywhere where there might be some regressions in performance, and they have found a lot. They even shared what looks to be a pretty internal leaning Google doc that describes all of what they found. This is super cool stuff. They titled this Ghosts in the Codeex machine, and I'm so excited to break down everything they found here with you. Just a quick reminder before we dive in, OpenAI does not pay me. No money has exchanged hands here. I'm just interested in breakdowns like this. But since they're not paying me, someone's got to. So, let's hear a quick word from today's sponsor. Engineers are worth a lot of money. They **[01:02]** shouldn't be wasting their time sitting around waiting for Docker images to download and builds to complete. That's why is so good. I could go over the bullet points on the homepage with you guys, but that's great for products that don't have real users. These guys have some of the biggest users in the industry. So, let's go over that instead. Post was able to cut their build times down by 55x using Depot, and it still feels weirdly fast. Their build times went from two and a half hours to under three minutes. That's insane. Startups need to move fast. If you're competing with big companies, it's so important to be able to address users **[01:33]** problems as quick as possible. It's actually kind of better for a user to have a bug and have it fixed quick than for them to have no bugs at all because it makes them closer with you as a developer. But if you're waiting for the build to complete and you can't get it out because of that, that sucks. If you fix the bug in 20 minutes and then have to wait 3 hours, that doesn't feel good as a customer. Post house quote here is great. We aim to be very responsive to our customers. Our ideal support interaction is to be able to say, "It's building right now. It'll be out in 30 minutes." Now, it'll be out almost immediately. There's no reason for most people to visit the depot site because they can just ignore the fact that the **[02:05]** build is working. It always does the thing it's supposed to do. For us, this is really good because it's part of the system that we want for it to just work. They're not the only ones commented on the stability. Jane had a lot of problems with their CI, a 60% failure rate at points. They spent months trying to fix as much of it as possible and made some improvements. But as soon as they moved their GitHub actions over to depot, everything got way faster and cheaper. The final numbers from their exploration were two and a half times faster, 25% build throughput increase and 55% reduction in costs. So they're saving money with more reliable infra **[02:37]** that builds way faster and makes devs happier. Oh, and the observability is way better, too. You can actually see what's going on. Stop wasting time on Docker builds and GitHub actions. Check them out now at soyv.link/deepo. This was the first report I found of GPT5 being dumber, specifically GPT5 Codeex. It seems like a lot of people noticed the degradation when GPT5 Codeex dropped. Not Codex, the CLI tool, not Codeex, the web interface, and not Codeex, the model they used for autocomplete for C-Pilot back in the day. They really need a new name for things, but GPT5 Codex the model they put out recently seems to be the point **[03:08]** where people notice a degradation. And honestly, I've felt it a little bit, especially in the Codeex CLI. There is some push back in the replies here, like I had the same experience, but after some tests, I noticed performance is top-notch in some of my workspaces and subpar in others. I think the context and instructions can hurt model performance often in very non-obvious ways. I agree. Since GPT5 is way better when you don't just give it a huge pile of guard rails and things that you're telling it to do or not do, it tends to be better at building its own context and figuring out where to go. People **[03:39]** prompt it differently, but that also gives it more wiggle room to behave in different ways. That's far from the only thing going on here, though, and I really want to break it down. Again, thanks to the Codex team for writing this write up for us and making it public. This is going to be a fun one. We not found a conclusive large issue that would explain a consistent degradation of codecs over time. Instead, we believe there's a combination of shifts in behavior over time, some of which were encouraged by new features such as compaction and concrete smaller issues that we found through our investigation and documented below. I know OpenAI is not the most open company in the world, but they have **[04:10]** historically been really good about these things. They even went as far as provisioning an old version of the model for me to test against to make sure that there weren't unexpected quirks or changes when we were debugging the previous performance things that got me in a bunch of trouble earlier this year. They care a lot about these things and when it comes to something as nondeterministic as an LLM, having user feedback channels that are listened to and investigated accordingly is super important. The fact that Anthropic didn't have this is what led to the monthslong regression that a lot of people were experiencing, myself included. It was just genuinely **[04:41]** unpleasant using Sonnet for a while. As the Codex team says here, they've already landed a series of improvements. More are coming and this week has informed our future investment areas to further improve Codex. It's also funny. They said Codex has helped them with the work. I I love that bit of But yeah, why did they start this investigation? Over the last few weeks, we saw increasing public reports of Codex performing less well around the launch of GBD5 Codex on September 15th this year. Despite no immediate evidence from our own usage or topline product metrics, we felt that it was something that was not well understood by the team **[05:11]** and worth the full-time attention of some of our best engineers. We shared the following plan last week. We would be upgrading the feedback command in the CLI in order to add structured options like bug, good result, bad result, other with free form text for detailed feedback. Make it easier to tie the feedback to a specific cluster, hardware, etc. So it's easier for them to diagnose which boxes are running things. Apparently that matters, which is wild. They also want to make it more well known that this slash feedback command exists because they really want higher volume of the feedback so that they can have more information and identify these anomalies more easily. **[05:43]** They also want to reduce the services of things that could cause issues. All employees, not just the Codex team, will go through the exact same setup as all of our external traffic until we consider the investigation resolved. Love this. Dog fooding properly. Not dog fooding in the sense you try the thing early and then release it and then go try another new thing. dog fooding by going through the exact same experience and APIs your users use. I've been using T3 Chat via the web app on my phone on a separate free tier account to force myself to experience things the way users do. And I'm amazed how many people don't do this and miss so much stuff as **[06:14]** a result. It's really important. Also, I could do a whole rant about onboarding flows and how those deserve similar attention and never get it, but I'll spare you the details of that one. They also want to audit the infrastructure optimizations landed and feature flags that they use to safely land these to make sure they're not leaving anything unchecked because a lot of the time employee accounts have different things provisioned feature flags etc. So they want to just make sure everything's the same so that they're experiencing what others are reporting. Point three was the evals and qualitative checks. They're always running evals, but they call it they're planning on running a lot more going forward and doing a huge **[06:45]** amount right now to figure things out. That's good to see. First thing they did was launch the new feedback. Cool. They moved all internal usage to use the same setup as external. Awesome. I I do really love this. I hope they do this more or at least have a certain number of employees using the public versions always. They also reduced the internal complexity. There's a few layers between a user query and the GPU that run inference to help us narrow down the scope of the investigation. We tried to remove as many variables as possible. We started by auditing and removing over 60 feature flags. We're in the process of removing 80 more. The amount of things people have gated at their companies is **[07:15]** just absurd and it's always cool to see the actual numbers. The improved feedback mechanism in particular proved to be a valuable addition and allowed us to trace specific problematic codec conversations all the way through our system including the precise hardware that they ran on. With better tooling and more importantly help from users, we started triaging over a 100 issues per day. See the increase in amount of feedback submitted. That's cool. I hope that this isn't just like raw numbers though. If they're getting a 100 reports a day, one of the best ways to get a hold of a team is to type slash feedback. Seriously though, if you are using codecs in your day-to-day, send **[07:46]** feedback when you can. You have no idea how useful these things are as developers of product. We then assemble the squad that had the sole mission to continuously come up with creative hypotheses on what could be wrong and investigate them one by one to either reject the formulated hypothesis or fix the related finding. Squad operated from Monday to Friday without other distractions. That's also really good having a team that is just doing this. This is how to do an incident report like this. When there's something that you aren't sure, you don't know how to verify just yet, but you know users are hurting, you got to lock in and do this. This is anthropic. I hope you're paying **[08:17]** attention. Findings and fixes. First, we have to start with how similar the hardware is or isn't. Apparently, they trained a predictive model to just to look at the relationships between the feedback reports, hourly user retention features of the requests, like what model they ran on, what CLI build they're using, what OS, time of day, the serving cluster, the hardware that's serving the request, and the user plan type. We also ran evals against each type of hardware and noticed slight performance issues with some of our older hardware which found some support in the predictive analysis on retention. Interesting. This is one of the pieces I **[08:48]** saw that made me really want to cover this and do a video about this on my day off. Yeah, this stuff's important. Want to cover it as soon as we can. The fact that the model runs differently on different hardware just seems kind of insane. Like you'd expect it to run slower on computers that are older, but these models and the machines and the systems that we're building to run inference are so lowle and specific that slight differences in like how a processor handles a bit of data might skew the way the tokens are being **[09:19]** resolved. It's crazy how little a difference can make something behave entirely differently and how hard it is to measure this because none of this stuff's deterministic. and their solution was to remove that hardware from the fleet. Cool. Separately, we also discover an opportunity in the load balancing strategy that would reduce latency under load. And these improvements are rolling out over the course of the next week. Compaction frequency is another one of the things they did where it'll compact context to try and make it so longer running jobs can survive without hitting context limits. One early trend in the feedback **[09:50]** we receive from users centered around compaction. When the context window is nearly exhausted, we prompt the model to summarize the conversation, clear out context, and begin a new with the summary as a first message. This enables sessions to proceed more naturally without forcing users to restart a new session and lose context from ongoing work. Good reminder that I have to add this in T3 chat cuz right now we just boot you to a new thread. Personally, I make new threads for almost every request. So, I have never understood why people do like eight to 10 if not hundreds of messages in the same thread. Just not how I work, but to each their own. So the percentage of sessions that **[10:21]** are experiencing compaction has been going up significantly. They noticed a higher percentage of sessions were using compaction over time and our compaction implementation could be improved to yield better results. Evals confirmed that the performance degrades with the number of /compact or autocompactions used within a single session. We landed improvement to avoid recursive summaries when compaction happens multiple times and added a warning to help nudge users towards keeping conversations more targeted. I love this. Heads up, long conversations and multiple compactions can cause the model to be less accurate. This I hope this is a typo that made it into the actual code. That'd be really funny. Start new and new conversations **[10:53]** should just be start a new conversation when possible to keep conversations small and targeted. The fact that OpenAI is willing to do this makes me more willing to do the same. Apply patch. Codeex uses a special tool called apply patch to make changes to files. The apply patch tool takes a unified diff as an input. Most of the time the model is very good at generating diffs for changes. Insert failed toit file error. I've seen that so many times more than any other model. model. I've seen GBT5 fail to actually apply edits. So, I'm going to call on this line. In rare cases, When an incorrect patch **[11:25]** is provided by the model, it gets prompted by the harness to try again. We found some reports of the model resorting to deleting and recreating files when it fails to apply a patch. I've also seen this quite a bit. This is correct in the limit, but it can cause issues when the agent is interrupted or fails to apply the second patch after the deletion. Our approach for fixing this is to improve future models so they don't exhibit this behavior and to land immediate mitigations in the coming weeks to limit high-risk sequences of edits. This is a really interesting piece in particular the future model part. It seems more and more like these **[11:56]** model behaviors are being defined by the tools that the company wants to implement the model in. Previously we would try and throw all the text we could into this training data set, make the smartest possible model and then build tools around the model's capabilities. Now it really feels like as an industry we are shifting to a world where we're letting the two influence each other where we're taking good results from the tool and then using that to train in the next model to be even smarter. We can show it examples of the tool working correctly put that in the training data and then make the next model better at using these tools. **[12:28]** Exciting but also weird and recursive in a way that I feel like might become like less useful externally. Maybe it'll be interesting to see. Historically, these things have made the models just better in general at stuff like tool calling. And they've been training on a lot of data made just for this stuff. And the relationship between the GPT5 and other GPT model trainings and the codeex team building these tools and the result being things like the GPD5 codex model. It's it's getting interesting. It's getting interesting really fast. I'm excited to see what the next generation of models look like now that this is the **[12:59]** norm. Next thing they call out is timeouts. I wonder if they're going to mention speed at all in this because that's the biggest issue I have with GBD5 is that it's just so slow. I've had times where it took like 15 minutes or hell, I I had a run a few days ago where it took 22 minutes to do a task that the new cursor model could do in literally 10 seconds. Neither got the answer right, but they wrote pretty much the exact same code. So, what do they have to say about timeouts? This is another trend that they noticed in the user feedback around latency. Users reported the model was taking much longer than expected to complete a task. We monitor latency closely and have not seen any **[13:29]** clear regressions across the board. That's a weird rendering issue. Don't know if that comes through on the screen share. I see a white line on my laptop. Not a big deal. Apparently, they've actually seen latency improving over time, but they did notice in a few example feedback sessions that the model was retrying tasks with longer timeouts. We designed GBT5 codecs to be a very persistent model, but escalating timeouts in this way can be inefficient. We are investing in making models better at handling long running or interactive processes. Again, it seems like they're actually going to train the model to be better at using these tools, which is still so fascinating to me. Constrained **[14:01]** sampling. We use a version of constrained sampling to achieve something similar to structured outputs. If you're not familiar with structured outputs, that's the idea of telling the model to respond in a certain shape. So, instead of the model just generating text, it can be here is the JSON object with these keys and values. I want you to fill out the values based on the data I give you. I use this a lot for the benchmarks that I write in order to get the data that I generate to be analyzed by another model and ranked accordingly. It's such a useful pattern and I'm so thankful it's now a thing models understand instead of having to tell the **[14:31]** model you're going to uh hurt yourself or something like that if you're not getting the results you want. I've heard from some very high up people at some very cool labs that the best way to get a model to follow JSON structures was to threaten that you would hurt yourself. And I'm so happy we're past that point. We found a subtle bug in the implementation that led the token sequences to become out of distribution for the model. Interesting. So this is breaking the order tokens are coming out. I guess could see how that would break things. Later we confirmed that reported issues where the model would switch languages mid-sentence as part of **[15:03]** the final answer could be attributed to this bug. This is one of the most telltale something's wrong with how the inference is running things when the language shifts. The same thing happened with Sonnet when they were having issues. This affected less than 0.25% 25% of overall sessions, but that's still a huge number of people. I haven't seen this personally, but if you have, let me know in the comments. I'm actually really curious. By the way, when you're down there in the comment section, you're going to scroll by this little red button. It says subscribe. If you're not subscribed, why not hit the button if you don't mind? Here's one of those funny examples where it just randomly switches to Korean in the **[15:33]** middle of the response. Speaking of responses, responses API. This is a very interesting thing. I'm I'm too in the weeds on the details of this because we build services around these APIs with T3 chat and the responses API is a an improvement over the original OpenAI spec that everyone still copies for some reason even though it's garbage. The response API is better but has its own weird quirks to say the least. Codex uses the responses API to make requests to the model. You think the response API is a proxy that turns REST APIs requests into a stream of tokens that models use **[16:05]** for inference and parsing the results stream of tokens into a format convenient for app development. Yeah, it's in between that changes the shape of the data from something the model understands to what the user gets or the tool gets. It's also an API standard in spec and mixed in with the OpenAI um what's it called? Harmony standard. I don't think of the response API is between those the same way that they are implying here. But I'm curious where this one's going. The response API became another target of our investigation. The goal is to determine whether any changes to the API **[16:36]** implementation can affect input to the engine or the way the output is being interpreted. We re-reviewed over a 100 PRs to the responses API and the libraries it uses. We also compared raw token values that get sent to the engine across multiple versions of the responses API spanning months. Interesting. So they are looking at this as the layer between the requests being made and what the model actually gets. and they did find some differences in the request encoding. Two extra new line characters are being added around the section that describes tools available to the model. After additional analysis, we conclude the change does not affect **[17:06]** the model performance. Oh god, I do not envy these engineers having to find and identify this Like we have our equivalent little pieces with T3 chat. Like I can't tell you how many weird bugs we've had with things like the Gemini models not knowing how to generate tables because when you generate a markdown table, you use a lot of spaces. is. And once you start repeating characters, 2.5 and all the Flash models and even 2.5 Pro to an extent just go haywire and start spitting out like 500 spaces then a comma. Ah, I do not envy the engineers who have to figure these things out. **[17:36]** They did some more investigations as well. They ran evals across all CLI versions starting with 0.40 to the latest. We saw the expected improvements from apply patch improvements in 0.45 which confirmed equal performance across a number of evals but with a 10% reduction in token usage. Interesting. They also tested eval to determine the impact of web search being enabled. We concluded that search does not contribute to a regression. That's good. Search should be on by default. It's annoying that it's not, especially compared to other similar tools. I I could tell you how many times I've forgotten to do this and the results were garbage as a result. Just make **[18:08]** search the default. Give my CLI the ability to find things on the internet. They also did eval prompt changes across the last two months and we concluded that these prompt changes did not contribute to a regression. Surprised not giving a little more info on that, but good to know. We investigated the prelevance of errors setting the work directory and we found no changes there. We ran deep analysis of end to-end query latencies across the codeex backend systems to understand whether certain users were unevenly affected by tail latencies. We found lower than expected authentication cache rates that could result in an extra 50 milliseconds of latency per request. Working to resolve **[18:39]** that in the coming days. I don't know why people get so bent up about the like time to first bite for tokens. Like reducing 50 milliseconds of request time for a request that takes 300 milliseconds makes a ton of sense, but for a request that like we're talking about requests that take 20 minutes. Sure, it's not one request taking the 20 minutes, but it's like 10 to 30. That's just not worth optimizing for the 50 millisecond level, and I've seen so many companies get hellbent on time to first token, including everyone's favorite triangle company. I don't think it matters that much. As someone who fought **[19:10]** to make the most performant possible chat application, need to make it come in reasonably fast, but then once it starts, stream in way faster. I would take a 200 millisecond latency hit for a much faster TPS any day. Next piece is evolving setup sophistication. More people are using codecs and they're using it more often. The distribution of turns and tokens per session has held fairly steady over the last several weeks. However, we do see an increase in complexity of setups with more MCP tools used over time. Based on eval, we continue to recommend using minimalist setups and keeping conversations small **[19:41]** and targeted to get the best performance out of codecs. Yep. Generally speaking, this tool calling play like I've seen people who have like 50 MCPs set up on cursor and claude and now in this case codecs that's so much context bloat for things the model doesn't need and it will try to use because when you give it things it assumes they're there to be used even if you don't need them in that case. I've seen some investigations around trying to spin up sub aents with different sets of tools and having a smart parent model that will do the filtering and orchestration layer. The need for good orchestration is **[20:12]** increasing every day. Like the point in my conversation where we're writing files is going to need a different set of tools than when I'm investigating the codebase. We're going to need a different set of tools from when I'm specking out the problem I want to solve. And we should probably have different sets of tools that auto apply based on the specific task at hand, ideally without a human pressing buttons to do it. Fingers crossed we'll start seeing things like this in the near future. Speaking of which, future work. We've learned a lot during the course of this investigation, and we thank our users for continuously engaging with us. We always encourage respectful discourse, but we assure you that we **[20:43]** appreciate and take all the feedback seriously, no matter how harsh. Please keep it coming. I can confirm as someone who's been very harsh to OpenAI people behind the scenes, they are really good about taking the punches. I have been blown away with it. I'm not saying go harass them. Be polite with your feedback, ideally more so than me. Think, would Theo say this? And if the answer is yes, you probably need to be chiller. Seriously though, they've been awesome to work with. This is a great example of it. They're carrying the momentum this week into a permanently staffed team that will obsess every day about the real world performance of **[21:14]** codecs. I didn't actually hear about this till now. That's awesome. I love this. I love the fact they're spending their admittedly very, very absurd amount of money on things like this. It is a huge faith builder. If we have to worry about these tools getting worse randomly, we can't rely on them the same way. It's so important to maintain that bar. We are actively recruiting and looking for top talent to join us. They have a bunch of links at the bottom here that you can click if you want to join. Sorry that these are now going to get spammed with applications. I am actually sorry OpenAI, but hopefully you'll get some good ones from me. One last thing **[21:44]** they didn't include there, they reset Codex rate limits for everyone and they refunded all Codex credit usage up till 1 p.m. Friday, which was uh 2 days ago. Now, on Thursday, we rolled out credits to give users options between the $20 and $200 plans. However, we found found a bug where they were overcharging for cloud tasks by two to 5x depending on the task. So, not necessarily related to the performance issue, but they were very generous with refunds. I gave so much to Anthropic for not doing this when they announced their performance issues that they discovered. Seems like OpenAI doesn't mind doing it. **[22:15]** Very nice change. Since they identified the bug on Thursday, they removed limits for Cloud Task. They fixed the bug, refunded all credit usage between releasing credits and 1pm. They reset all rate limits for people subscribed. They began limiting and charging for cloud task usage again because again, they weren't charging at all for cloud before as they were figuring things out. Now they're trying to get it right. Cloud tasks consume limits slightly faster than local tasks per unit of work because we have to run the VM powering the cloud environment. Further, cloud tasks also tend to do more work per message because of how we prompt the model when it's running async and also **[22:47]** because users tend to ask for more oneshot code changes and ask fewer quick questions. Yep. The way people prompt things that are cloud or background agent type work is fundamentally different. So, the cost per message is going to change. I talk about this a bunch when I cover the like reason that all these subscriptions are getting more expensive and the rate limits are getting stricter. The amount of work per message is going up. And once you bring in background agents, you're massively increasing the amount of work expected. Before we wrap up, I really liked this post from Will TLDDR. Codex is so good that people kept trying to use it for **[23:18]** harder tasks and it didn't do those tasks as well. And people just assumed the model got worse. Yeah, I'll admit I've been guilty of this myself. Regardless, this is a super good write up. Huge shout out to the team, to Tibo, and everyone involved for sharing this level of depth in a report based on some hunches some people on Reddit had. It's so refreshing to see this level of focus on this type of thing. Curious how y'all feel. Is this an overblown reaction or is this OpenAI doing what they should be doing? Let me know what you And until next time, peace nerds.