GPT-5.2 is the best model ever made - CourtneyR.dev's 2nd Brain

GPT-5.2 is the best model ever made 🔗 **Source:** [GPT-5.2 is the best model ever made](https://www.youtube.com/watch?v=CtMk0GuQ7cc) > [!note]+ Document Notes > tags: [ai, gpt, llm-evaluation, model-benchmarks] > author: Theo - t3․gg > source: YouTube > reference: https://www.youtube.com/watch?v=CtMk0GuQ7cc > date created: 2025-01-12 12:00 > > # 🏷️ Title: GPT-5.2 is the best model ever made >  ## 🧰 Action Inbox - [H1] Test GPT 5.2 Pro for deep reasoning tasks — most capable model available #ai-testing - [H2] Compare instruction following between GPT 5.2 vs Opus 4.5 — GPT better at following exact instructions #model-comparison - [H3] Evaluate 5.2 instant mode for faster responses — no reasoning version performs much better #performance-optimization ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ### 🔗 References - [T3 Chat](https://t3.chat) - [ARC AGI Benchmark](https://arcprize.org/) ## 📝 Summary - GPT 5.2 shows massive improvements on most benchmarks but significant regressions on 3D spatial reasoning - Major price increase to $17.5 per million input tokens, first in recent memory - Speed remains the biggest limitation, with Pro models taking 30-50 minutes for complex tasks - Instruction following is significantly better than competing models like Opus 4.5 - New ARC AGI scores of 90.5% represent a 390x efficiency improvement over last year ## 📖 Main Content GPT 5.2 represents OpenAI's latest flagship model with mixed but largely positive results across benchmarks. The model shows exceptional performance on coding tasks (55.6% on SWEBench), mathematical reasoning (100% on AME without tools), and the challenging ARC AGI benchmark (90.5% at $116 per task vs $4,500 a year ago). However, the model exhibits concerning regressions in 3D spatial reasoning. Theo's custom "Skate Bench" - which tests spatial understanding by having models identify skateboard tricks from descriptions - shows GPT 5.2 scoring only 79% compared to GPT 5's 97%, despite using 3x more tokens. The pricing structure introduces complexity with multiple tiers: GPT 5.2 thinking, GPT 5.2 Pro ($210 per million input tokens), and GPT 5.2 instant (no reasoning mode). Speed remains problematic, with Pro models sometimes requiring 30-50 minutes for responses. ## 🗒️ Notes #### ⏱️ [00:00](https://youtu.be/CtMk0GuQ7cc?t=0) — [H1] GPT 5.2 Release and Benchmark Confusion - Key idea: New model shows mixed results with major regression on spatial reasoning - Evidence: "Skatebench shows it as a huge regression. What's going on?" #### ⏱️ [02:45](https://youtu.be/CtMk0GuQ7cc?t=165) — [H2] Skateboard Benchmark Results - Key idea: Custom 3D spatial reasoning benchmark shows significant performance drop - Evidence: "GPT5 would get 97% on this benchmark. When I first ran this benchmark with the new GPT 5.2 model, I got this 4% number" #### ⏱️ [05:12](https://youtu.be/CtMk0GuQ7cc?t=312) — [H2] Pricing Analysis and Performance Cost - Key idea: Extra high reasoning mode costs 5x more but delivers 20% worse performance - Evidence: "GPD5 default, it was about 6 cents per request. And GBT 5.2x high was about 25 cents per request... And the reward for that is a 20% regression in performance" #### ⏱️ [07:30](https://youtu.be/CtMk0GuQ7cc?t=450) — [H1] ARC AGI Breakthrough Results - Key idea: Massive efficiency gains on challenging reasoning benchmark - Evidence: "90.5% and it's only $116 per task. That's a 390x efficiency improvement in one year" #### ⏱️ [09:45](https://youtu.be/CtMk0GuQ7cc?t=585) — [H2] Timeout and Speed Issues - Key idea: Models are extremely slow, sometimes taking 10 minutes per simple request - Evidence: "It took like up to 10 minutes per request at times for naming a skate trick that I can do in 5 seconds" #### ⏱️ [12:20](https://youtu.be/CtMk0GuQ7cc?t=740) — [H1] GDP Val Professional Performance - Key idea: Model matches or beats professional experts on 70.9% of knowledge work tasks - Evidence: "5.2 thinking beats or ties the top industry professionals on 70.9% of the comparisons" #### ⏱️ [15:35](https://youtu.be/CtMk0GuQ7cc?t=935) — [H2] Code Generation Testing - Key idea: Direct comparison shows GPT 5.2 follows instructions better than Opus - Evidence: "GPT5 just did it first shot. That's the difference. I had to do two follow-ups with Opus to get it to behave" #### ⏱️ [18:50](https://youtu.be/CtMk0GuQ7cc?t=1130) — [H2] Frontend and UI Generation Quality - Key idea: Significant improvements in UI generation with consistent gradient patterns - Evidence: "Pink top left, blue bottom right. That's the new gradient pattern all of these AI things love to use" #### ⏱️ [21:15](https://youtu.be/CtMk0GuQ7cc?t=1275) — [H1] Price Increase Details - Key idea: First price increase in recent memory, up from $12.5 to $17.5 per million tokens - Evidence: "It was 125 in and 10 out for GBT 5 and 5.1... Now it's up a bit to 175 in and 14 out" #### ⏱️ [23:40](https://youtu.be/CtMk0GuQ7cc?t=1420) — [H2] GPT 5.2 Pro Extreme Pricing - Key idea: Pro model pricing reaches new highs at $210 per million input tokens - Evidence: "$210 per mill in and 168 per mill out. A new groundbreakingly high price for a model" #### ⏱️ [25:20](https://youtu.be/CtMk0GuQ7cc?t=1520) — [H2] Hallucination Improvements - Key idea: Significant reduction in hallucinations compared to competitors - Evidence: "Going back to a GPT model and feeling the difference of how much less likely it is to hallucinate is insane" #### ⏱️ [27:45](https://youtu.be/CtMk0GuQ7cc?t=1665) — [H1] Matt Schumer's Professional Assessment - Key idea: Expert reviewer calls it "undoubtedly the world's best model" despite speed issues - Evidence: "Undoubtedly the world's best model. I can't live without it... After using Pro for 2 weeks, I can't live without it" ## 📚 References - [T3 Chat Platform](https://t3.chat) - [ARC AGI Prize](https://arcprize.org/) - [SWEBench Evaluation](https://swebench.com/) - [Matt Schumer's Model Reviews](https://twitter.com/mattschumer_) ## 🗣️ Paraphrase - GPT 5.2 achieves breakthrough performance on most benchmarks including 90.5% on ARC AGI with 390x better cost efficiency than previous versions, but shows concerning regressions in 3D spatial reasoning tasks - The model significantly outperforms competitors at instruction following and produces higher quality code and UI generations, though it requires substantially more time and cost for complex reasoning tasks - Despite being the most capable model available, speed limitations and high pricing (especially for Pro tier at $210/million tokens) create practical barriers for everyday use ## 📰 Newsletter **GPT 5.2: The Most Capable Yet Controversial AI Model Release** The latest GPT 5.2 brings unprecedented capabilities with a 90.5% score on the challenging ARC AGI benchmark - representing a massive 390x efficiency improvement over last year's attempts. For developers, this means more reliable code generation, better instruction following, and significantly improved UI design capabilities. However, the release comes with important trade-offs. While excelling at most tasks, the model shows surprising regressions in 3D spatial reasoning. More critically for practical use, response times can stretch to 30-50 minutes for complex tasks, and pricing has increased substantially to $17.5 per million input tokens. The standout feature remains instruction adherence - GPT 5.2 consistently follows directions more precisely than competitors like Opus 4.5, requiring fewer follow-up prompts to achieve desired results. **Key Takeaways for Developers:** - Consider GPT 5.2 instant mode for faster responses on simpler tasks - Budget for higher costs but expect better first-attempt success rates - Pro tier offers exceptional capability but requires patience for complex reasoning Resources: [T3 Chat for Model Access](https://t3.chat), [ARC AGI Benchmark Details](https://arcprize.org/), [SWEBench Code Evaluation](https://swebench.com/) ## ✅ Action Items #### DevRel - Developer experience: Document GPT 5.2 integration patterns, API timeout handling - Developer marketing: Create cost-benefit analysis content comparing model tiers - Community management: Gather feedback on speed vs capability trade-offs #### Community - Contributors: Share benchmark methodologies for spatial reasoning evaluation - Ecosystem: Coordinate model comparison frameworks across projects #### WordPress - Core: Evaluate GPT 5.2 for automated testing and code generation workflows - Documentation: Test model for technical writing and API documentation generation - Meta: Assess cost implications for AI-assisted moderation and support ## 🧪 Hypotheses - GPT 5.2's spatial reasoning regression indicates over-optimization for 2D pattern recognition at expense of 3D understanding - Instruction following improvements stem from better training on precise task completion rather than creative interpretation - Price increases reflect true computational costs rather than market positioning, given performance gains on complex benchmarks - Speed issues will limit adoption for real-time applications despite capability advantages ### 📊 Methodology - Comparative benchmarking: Run identical prompts across GPT 5.2, Opus 4.5, and Gemini 3 Pro measuring accuracy, speed, and cost - Spatial reasoning evaluation: Develop standardized 3D understanding tests beyond skateboard trick identification - Cost-effectiveness analysis: Track token efficiency and success rates across different complexity levels ### ❓ Questions - Will OpenAI address spatial reasoning regressions in future updates or consider them acceptable trade-offs? - How will competing models respond to GPT 5.2's benchmark leadership, particularly in ARC AGI performance? - Can the extreme response times be optimized without sacrificing the model's deep reasoning capabilities? ### 🔮 Future Research - Long-term performance tracking across diverse reasoning tasks to identify consistent capability gaps - Speed optimization techniques for production deployment of high-reasoning models - Cost modeling frameworks for enterprise AI adoption with premium-tier models ### 💡 Implications - Practice: Developers must choose between speed and capability, requiring workflow redesign around longer response times - Policy: Pricing increases may limit access to advanced AI capabilities for smaller organizations - Education: Training materials need updating to reflect new instruction-following paradigms and cost considerations - Funding: Budget planning must account for significantly higher inference costs for complex reasoning tasks ### 🧠 Learning Styles Assessment Reading level estimate: 12th grade #### 👀 Visual learners - Present: Benchmark charts, UI generation examples, gradient pattern demonstrations - Improve: Architecture diagrams showing reasoning pathways, cost comparison visualizations #### 🔊 Aural learners - Present: 28-minute detailed video walkthrough with live demonstrations - Improve: Audio summary highlighting key performance metrics and trade-offs #### 🗣️ Verbal learners - Present: Clear explanations of technical concepts, model comparisons, practical analogies - Improve: Glossary of benchmark terminology, step-by-step reasoning explanations #### 🧩 Kinesthetic learners - Present: Live coding demonstrations, real-time benchmark running, hands-on model testing - Improve: Interactive cost calculators, benchmark reproduction guides #### 🧠 Logical learners - Present: Performance metrics, cost analyses, systematic comparisons across models - Improve: Decision trees for model selection, ROI calculation frameworks #### 🧑‍🤝‍🧑 Social learners - Present: Community feedback integration, expert reviewer quotes, shared experiences - Improve: Discussion prompts for model selection decisions, peer comparison frameworks #### 🧘‍♀️ Solitary learners - Present: Detailed individual testing methodology, personal benchmark creation - Improve: Self-assessment rubrics for model evaluation, reflection questions on use cases #### 🌿 Naturalistic learners - Present: Ecosystem thinking about model relationships and evolution - Improve: Connections to broader AI development patterns, sustainability considerations ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Clear heading hierarchy, descriptive timestamps, alt text for benchmark charts - Improve: More detailed descriptions of visual UI generation examples #### Keyboard navigation - Present: Video controls accessible, timestamp links functional - Improve: Keyboard shortcuts for section navigation #### Contrast and readability - Present: High contrast in video interface, clear text overlays - Improve: Ensure benchmark chart text meets 4.5:1 contrast ratio #### Text resizing and reflow - Present: Responsive video player, scalable text content - Improve: Test transcript readability at 200% zoom #### Media and captions - Present: Full transcript provided with technical detail - Improve: Captions with speaker identification, technical term highlighting #### Forms and interactions - Present: Simple timestamp navigation - Improve: Accessible controls for playback speed adjustment ### 🌍 DEIB Assessment #### Cultural breadth - Present: Focus on universal technical concepts, global AI development impact - Improve: Examples from diverse geographic regions, multi-currency cost analyses #### Inclusive language - Present: Technical terminology clearly defined, gender-neutral references - Improve: Plain language summaries, avoid technical jargon overuse #### Neurodiversity support - Present: Well-structured content with clear sections, predictable formatting - Improve: Content warnings for rapid technical demonstrations, processing time considerations #### Representation - Present: Diverse perspectives through multiple reviewer quotes - Improve: Include voices from different organizational sizes, use case contexts #### Psychological safety - Present: Acknowledges model limitations, presents balanced viewpoints - Improve: More explicit discussion of cost barriers, accessibility implications ### ✅ Logical Validity #### Strong reasoning - Deductive strength: Solid benchmark methodology leading to specific performance conclusions - Inductive strength: Multiple test cases supporting general claims about instruction following - Causal reasoning: Clear connections between design choices and performance outcomes - Analogical strength: Appropriate skateboard benchmark analogy for spatial reasoning #### Fallacies - Cherry picking: Acknowledges benchmark limitations and potential bias in custom testing - Hasty generalization: Appropriately qualified claims about model capabilities with specific contexts #### Tone of voice - Overall tone: Technical but accessible, balanced criticism with enthusiasm, practical focus on real-world implications ### 🛡️ DARVO Analysis - Deny: No significant denials identified - acknowledges both strengths and weaknesses - Attack: No attacks on critics - presents balanced view of --- ## Full Transcript [00:00] Where's the guy with the counter? [00:01] Because there's a new best new model. [00:02] Yes, really. GPT 5.2 just dropped and it [00:05] is really, really good. Which is why it [00:08] is uh Wait, what? Skatebench shows it as [00:12] a huge regression. What's going on? I [00:15] thought this was the best new model. [00:16] Well, in many ways it is, but in a [00:18] handful of important ones, it isn't. And [00:20] I haven't seen many covering this in [00:22] detail. Sure, it's better at code and [00:24] tool calls, and yeah, it's crushing ARC [00:26] AGI, but there is a depth to using these [00:28] models that isn't really being shown in [00:30] a lot of the coverage I'm seeing. And I [00:32] want to break down what it's actually [00:33] like to use because I've been lucky [00:34] enough to be using it for the last week [00:36] or so. Very thankful to OpenAI for [00:37] giving me early access. That said, no [00:39] money has exchanged hands. The only [00:41] person paying me is today's sponsor. We [00:43] need to be realistic about how much [00:44] easier our jobs are now. AI has made [00:46] everything from feature additions to bug [00:48] fixes simpler than it's ever been. [00:49] Navigating your codebase, finding the [00:51] right file, and making the changes you [00:52] need is really easy, but there are a few [00:54] things that are still hard and also [00:56] really scary if you get them wrong. The [00:58] two that I think of the most are [00:59] authentication and authorization of [01:00] users and payment processing. Getting [01:03] these things wrong can be disastrous. [01:05] And even if you think you got them [01:06] right, something might prop up later on [01:08] that makes you regret building it [01:09] yourself entirely. Not only have I done [01:11] this myself incorrectly, I've even [01:13] written detailed documentation on how to [01:15] do it right, that barely feels necessary [01:18] anymore because of today's sponsor, [01:19] Clerk. If I was building a new app [01:21] today, this is what I would choose both [01:23] for authentication and authorization of [01:25] my users and for payment processing [01:27] because it is the best and simplest way [01:29] to do both. Period. And I've used every [01:31] single option. They aren't paying me to [01:33] say that. They're paying me to mention [01:34] them. I'm telling you, as a person who's [01:36] went through all of the options, [01:37] especially on the payment side, Clerk [01:39] has found this perfect integration of [01:41] things as well as developer experience [01:43] that makes it easy to build a secure, [01:45] reliable application for everything from [01:47] signing in to signing up. There's [01:48] something beautiful about seeing a [01:50] component like this for something as [01:52] annoying as payment processing and [01:54] managing what users access to what [01:56] features. You want to protect a feature [01:58] so that you only have access if you're [01:59] paying for the team plan. It's this [02:01] easy. protect feature equals team [02:04] access. And now, as long as you have [02:05] that set on the server side when you [02:07] define these things in their dashboard, [02:09] you're good to go. You could even check [02:10] if they have a thing like the bronze [02:13] plan. And if they don't, you can return [02:15] an error. Do you understand how annoying [02:17] these things are to do traditionally and [02:19] how hilariously easy they are to do with [02:21] Clerk? They even have a pricing table [02:23] component. you know, the fancy like [02:25] comparison of the different tiers that [02:27] is fully integrated, has the Stripe pop [02:29] overview built into it, and once they [02:32] subscribe, it's immediately linked to [02:33] that user's account. It's so much easier [02:36] than the other solutions. I genuinely [02:38] wish this existed when I started T3 [02:39] Chat. It would have made our lives so [02:41] much easier. Like, I'm talking not [02:43] weeks, but months of work. If you're [02:45] ready to have real users and make real [02:46] money, look no further. Check them out [02:48] now at soyb.link/clerk. [02:50] There's a lot to talk about with this [02:51] new model, but I want to resolve the [02:53] click baiting I just did with that whole [02:55] benchmark thing. There are a lot of [02:57] layers to this that we'll get to later, [02:58] but I want to just show the raw numbers [03:00] that I just ran. And I've spent a lot of [03:02] money running this test over and over [03:04] again to be sure of my results. I even [03:05] went back and forth with some of the [03:06] people on the research side trying to [03:08] figure out what caused this regression. [03:10] My current theory is that the new model [03:12] is not as good at three-dimensional [03:13] reasoning because my benchmark is about [03:15] skateboard tricks. I give a description [03:17] of a trick and I expect the model to [03:19] tell me what the name of that trick is. [03:21] And previously the highest score I ever [03:22] got was with GBT5. Not even high. The [03:25] default setting would get 97% on this [03:27] benchmark. 03 Pro would get 96%. So like [03:31] GBT5 was incredible at the spatial [03:33] reasoning necessary to describe a [03:35] skateboard trick accurately. When I [03:37] first ran this benchmark with the new [03:38] GPT 5.2 model, I got this 4% number and [03:41] I was like, "What the [ __ ] went wrong [03:43] here?" I did a lot of back and forth [03:45] with the team and we realized that my [03:46] harness due to some changes in how the [03:48] GPT 5.2 model defaults, which is it [03:51] defaults to no reasoning, the results [03:52] ended up being lower. So like, okay, [03:54] cool. I'll run it on high and on the new [03:56] extra high reasoning options, which [03:58] ended up being comically more expensive [04:01] to run this bench on. Again, compared to [04:03] GPD5 default, it was about6 cents per [04:06] request. And GBT 5.2x high was about 2.5 [04:10] cents per request. Yes, it was about [04:12] five times more expensive. And the [04:14] reward for that is a 20% regression in [04:17] performance. Is naming skateboard tricks [04:19] the best use case for LLMs? Probably [04:21] not. But I was so pumped to see GPT5 [04:24] understand spatial reasoning well enough [04:26] to do this. And this is a massive [04:29] regression. Like absurdly so. It's now [04:31] tied with models like Grock 4. What what [04:35] went wrong there? But it seems like my [04:37] benchmark is novel in this case because [04:39] almost every other bench is showing a [04:41] very different story like GBT 5.2 [04:44] thinking getting a 70.9% on GDP val [04:47] versus 5.1 thinking and five getting a [04:50] 38.8%. [04:52] SW Bench saw a big bump at 55.6%. SWB [04:56] verified saw a first time score for the [04:58] OpenAI team of an 80%. GBQA Diamond was [05:02] a nice boost. AME was now 100% with no [05:04] tools which is really good. It means it [05:06] can solve math problems without using [05:07] math tools. Impressive, but there's a [05:09] lot more to these stories. That all [05:11] said, the Arc AGI scores are actually [05:14] absurd. From ARC prize themselves, a [05:17] year ago, we verified a preview of an [05:19] unreleased version of OpenAI's 03 high [05:21] that scored an 88% on ARC AGI1 at an [05:24] estimated $4.5,000 [05:26] per task. Pretty absurd how expensive it [05:29] was for them to run that special version [05:30] of 03 for this, but uh it was able to [05:33] get a high score. And now the extra high [05:35] option on GBT 5.2 Pro, which is also [05:38] notably not GBT 5.2, scores a 90.5% and [05:42] it's only $1164 [05:44] per task. That's a 390x efficiency [05:47] improvement in one year. ArcGI is a wild [05:50] benchmark. I have covered it many [05:51] different times. If you're not familiar, [05:52] go to their website and try it yourself [05:55] because it's a thing that's easy for [05:56] humans and nearly impossible for LLMs. [05:58] It is no longer nearly impossible for [06:00] these LLMs. It's actually really, really [06:01] impressive. They made a V2 of this [06:03] leaderboard that was meant to be [06:05] actually impossible for LLMs to solve, [06:07] thinking it would be like the final [06:08] benchmark ever. Yet, here we are with [06:10] more and more models getting high scores [06:13] on it. We just had Gemini 3 Pro have a [06:15] groundbreaking 30% and then the Gemini 3 [06:18] Pro deep research, whatever the hell [06:21] they call their like heavy version [06:23] getting all the way into the 40s. And [06:24] now GBD 5.2 Pro High is scoring way, way [06:28] higher here. 54.2% 2% for $15.72 [06:33] per task. But then my favorite part, due [06:35] to API timeouts, we were unable to [06:37] reliably verify GBT 5.2 Pro XH high on [06:41] RKGI2. Turns out these models are still [06:44] timing out all over the place. It's kind [06:46] of insane how long 5.2 Pro will run for. [06:49] I didn't even put it in my most recent [06:50] run of Skate Bench because it took like [06:52] up to 10 minutes per request at times [06:55] for naming a skate trick that I can do [06:56] in 5 seconds, not even. It's kind of [06:59] absurd how hard the new models will go. [07:01] And even 5.2 extra high was able to take [07:03] 240 seconds per request. That's 4 [07:07] minutes per request sometimes for naming [07:09] a skate trick. The first example they [07:11] open with is the economically valuable [07:12] tasks section which includes GDP val [07:15] which is a benchmark they made for [07:16] evaluating how well different models [07:18] behave against known knowledge work [07:21] tasks that you have to have like a [07:22] degree or something for. They take [07:24] actual examples of work and ask the [07:26] model to perform against it and see if [07:28] it performs at or above a human expert [07:30] level. And 5.2 thinking beats or ties [07:33] the top industry professionals on 70.9% [07:35] of the comparisons on the GDP file [07:38] knowledge work tasks according to expert [07:39] human judges. That's the other [07:41] interesting piece here is this benchmark [07:42] needs to be judged by humans. So I can't [07:44] just run it on my laptop. And it's a [07:46] massive jump from GPT5 which was a 38.8% [07:49] to 70.9 for 5.2 thinking and 74.1 for [07:52] 5.2. 2 Pro. And there's no chart crimes [07:55] happening here either, which is nice. [07:56] Don't worry, we'll have plenty of chart [07:58] crimes later. While reviewing one [07:59] especially good output, one GDP Val [08:01] judge commented, "It is an exciting and [08:03] noticeable leap in output quality. It [08:06] appears to have been done by a [08:07] professional company with staff and has [08:09] a surprisingly well-designed layout and [08:11] advice on both deliverables, though with [08:13] one we still have some minor errors to [08:15] correct. Yeah, apparently very [08:18] impressive. But on the topic of design, [08:20] we should see how it handles design [08:22] work. I have a stock Dex.js project that [08:25] I haven't made any changes to. We're [08:26] gonna do my favorite, the image gen [08:27] bench where I tell it to make an image [08:29] generation studio mock and uh apparently [08:32] cursor is breaking and I am now past my [08:36] usage. I'm on the $20 plan so the fact [08:38] that I'm just hitting that now is really [08:40] cool. Obvious bias. I am an investor in [08:42] cursor so account for that. This is also [08:44] one of my first times using 5.2 incursor [08:46] because I didn't do the thing where I [08:47] set up the open AI like API manually [08:50] incursor. I have told this to the team [08:52] many times and I will tell them again [08:54] right now. Hi friends at cursor. I [08:56] should be able to set a custom OpenAI [08:58] API endpoint without having it break [09:00] every other model in your app. It's very [09:02] annoying that if I go set an OpenAI API [09:04] endpoint that I can no longer use Opus [09:06] or Composer or any of the other models [09:08] that I like to use in Cursor. It's quite [09:10] obnoxious. I know I'm unique that I get [09:12] early access and I have a need to test [09:13] these things. But it basically makes the [09:15] feature of the checkbox to put in a [09:17] custom API endpoint entirely useless to [09:19] me. The majority of that was uncut, but [09:21] I'll let you know roughly how much time [09:23] this generation takes. It isn't very [09:25] fast. These models are still not very [09:27] fast. It's the one big pain point I have [09:29] had with the GPT5 series now that I'm [09:31] experiencing Opus 4.5 and especially the [09:34] composer model and cursor way more. It [09:36] is obnoxious how slow the GPT5 series [09:40] is, especially if you take advantage of [09:42] some of the fancy pro model stuff, which [09:44] like I've had the Pro models take 30 to [09:47] 50 minutes to respond to things. It's [09:49] awesome that they can go out and do work [09:50] and come back with a correct answer so [09:52] reliably, but god damn, those models are [09:55] slow. I thought this was going to be [09:56] fast enough that I could talk over it [09:58] and give you guys like a realistic [09:59] feeling for how long it would take. I [10:01] overestimated just how fast it could be. [10:04] So, we'll come back to GBT 5.2 and [10:06] cursor in a moment. Almost forgot to [10:07] mention, as y'all probably expect, we [10:09] threw all of the new models in T3 chat. [10:11] So, if you want access to all of the [10:12] versions, including the new reasoning [10:14] one that's a little restricted on the [10:15] free tier on chat GPT, for eight bucks a [10:17] month, you can use all these here. If [10:19] you want your first month for only $1, [10:21] use code 5.2 at checkout. Anyways, back [10:23] to whatever the heck I was just talking [10:25] about. GBD 5.2 Thinking sets a new [10:27] state-of-the-art of 55.6% on SWEBench [10:30] Pro, a rigorous evaluation of real world [10:33] software engineering. Also note, this is [10:35] GPT 5.2 thinking, not GPT 5.2 to codeex [10:39] which will probably come in the near [10:40] future. It does seem like they're [10:42] finally realizing how bad the [10:43] terminology for the models is and they [10:45] might not do the whole codeex thing [10:48] again. I I hope that they drop the [10:49] overuse of that word. It just they [10:51] should have called codec cli GP [10:53] terminal. They should have had the model [10:55] be the code edition or something, but [10:57] just calling everything codeex is [10:58] obnoxious. Regardless, this SWB verified [11:01] bench only tests Python, and SBench Pro [11:03] tests four languages and aims to be more [11:05] contamination resistant, challenging, [11:07] diverse, and industrially relevant. In [11:09] the Pro version, they got the new [11:11] highest score on the extra high version [11:13] of 56%. But very interestingly, the 5.1 [11:17] Codeex version when given max extra [11:19] high, so like allowed to just use as [11:21] much context as it wants, it actually [11:22] performed slightly worse than the normal [11:25] high version does. the the way rein are [11:27] being used is increasingly weird, [11:30] especially with the new defaults and the [11:32] behaviors we're seeing here. Like GPT5 [11:34] on my benchmark got a 97% doing 600 [11:38] tokens per request average. GBD5.2 extra [11:41] high got a 79% an 18 point regression [11:45] while using over three times the number [11:47] of tokens. [11:49] Interesting. For everyday professional [11:51] use, this translates into a model that [11:52] can more reliably debug production code, [11:54] implement feature requests, refactor [11:56] large code bases, and ship fixes end to [11:57] end with less manual intervention. They [11:59] also said it's better at front end. So, [12:01] we'll look at that in a sec cuz the [12:02] generation did finish. But, I want to [12:04] first talk a little bit about my [12:05] experience using this to generate my own [12:08] tests. I maintain a handful of different [12:11] benchmarks and evaluate [12:14] bench in particular. had some changes I [12:16] wanted to make around how the caching [12:18] worked because some of these new models [12:20] would hit errors and I would need to [12:22] rerun it, but the cached errors would [12:24] keep the model from rerunning. That plus [12:26] all the weird things about usage and I [12:28] really really wanted to start tracking [12:29] token utilization so I could include [12:31] that in my coverage here as well as [12:33] these awful run times. So I had to make [12:35] some changes for that. So I wrote a [12:36] prompt make the following changes to [12:39] this project. One, cache should also [12:41] include token counts and durations. Two, [12:43] errors should not be cached. So a rerun [12:45] should re-trigger any jobs that errored [12:47] out. And three, you should show the [12:49] average token usage in the table in the [12:51] CLI view. Think this is a pretty clear [12:54] set of things to do. I had this come [12:56] through with composer, with Opus, and [12:58] with GPT 5.2. The composer version came [13:01] back almost instantaneously with changes [13:03] that were mostly good. I didn't like how [13:05] it was doing the cost calculation and [13:07] the token calculation though. It was [13:08] relying on a weird sub field instead of [13:09] just using what the SDK gave me. So I [13:12] rejected that. I don't know why it has a [13:13] thumb there cuz I picked opus, not [13:14] composer. The opus version seemed like [13:16] it was really good and it was half as [13:18] much code as the others until I looked [13:19] at it more closely and realized that it [13:22] wasn't actually using the cache for the [13:24] token counts or for the times like it [13:26] was expected to. So that was incredibly [13:28] frustrating. So I told it to start [13:30] caching these things and it did, but it [13:31] didn't use the cache results for [ __ ] [13:33] anything. But I had to remind it later [13:35] on with a follow-up message because I I [13:38] didn't notice this mistake until I [13:39] reran. was like, "Wait, you just screwed [13:41] this up." Also, it was tracing input [13:43] tokens, which is the same for every [13:45] single run, so it doesn't matter. So, I [13:46] told it to drop that, but it still [13:47] entirely forgot to restore durations [13:49] when loading from cache. So, I had to do [13:51] two follow-up prompts with Opus 4.5 to [13:53] get it working how I expected. That all [13:55] said, I have not actually tried the [13:57] chatbt 3.2 version. I gave the code a [13:59] quick look and it seemed fine, but [14:01] that's not like me. I want to actually [14:03] test it. Let's do that. Copy work path. [14:06] Hey uh friends at cursor who are [14:08] probably watching this. At no point did [14:11] I do anything that would result in these [14:13] changes from happening. In fact, the [14:15] thing that is being deleted here is the [14:18] correct version that works. And the [14:19] thing that's here, the green, the [14:21] proposal, is entirely wrong and [14:22] incorrect. There's no review anything [14:24] here. I have no idea why this is here at [14:26] all. It's wrong and bad and broken. It's [14:29] a reversion of something that hasn't [14:30] existed in this codebase forever, if [14:32] ever. Why is this here? What the [ __ ] [14:34] wrong with the UI? I don't know what's [14:36] went wrong with the review mode, but [14:37] it's getting really, really egregious [14:39] lately. But now that I have fixed the [14:41] package JSON, to be very clear, this is [14:44] not broken because of GBT 5.2. This is [14:45] broken because of cursor. [14:47] Let's run the test. And look at that. [14:50] The one from GPT 5.2 did everything [14:53] right first shot. So the the reason I [14:55] just tested all of that is I really [14:56] wanted to emphasize the difference in I [14:59] guess vibe when I'm using GPT models, [15:01] especially 5.2, [15:03] Compared to models like Opus and [15:04] Composer, GPT5 is just the series that [15:07] follows instructions the best. That's [15:09] the best I can put it. The Opus models [15:11] will roughly finish the task you give [15:13] them, and they're very smart. Their [15:14] ability to figure out what's wrong, [15:16] debug, and push is incredible. Opus [15:19] models can turn through insane tasks for [15:21] long amounts of time and actually [15:22] generate something that works. But the [15:24] GPT models will do what you [ __ ] tell [15:25] them to do. I had to do two follow-ups [15:28] with Opus to get it to behave. It wrote [15:30] slightly better code that is closer to [15:32] what I would have written, but I had to [15:34] tell it what to do with multiple [15:35] follow-ups. GPT5 just did it first shot. [15:40] That's the difference. I have had a much [15:42] better time with the GPT models for [15:43] that. But what's extra funny is GPD 5.2 [15:46] still took longer than me running Opus, [15:49] testing the results, realizing there [15:50] were things wrong, reprompting it, and [15:52] getting a new result. So, if you want a [15:54] model that does what you say and you're [15:56] willing to wait, 5.2 is incredible. If [15:58] you want a model that is really really [16:00] smart, possibly even smarter than you [16:01] for the things that you do, but loves to [16:04] just go off on its own little tangents [16:05] that you have to like grab your like [16:07] harness and pull it back in. Opus is [16:10] great for that. And even then, Opus 4.5 [16:12] is a significant improvement in [16:13] instruction following in my opinion from [16:15] how I've experienced previous enthropic [16:17] models. They seem too happy to go change [16:20] code and not happy enough to do what you [16:21] tell them to. Enough of all this. I want [16:23] to see my image generation studio. So, [16:25] let's see how this one came out. [16:29] Not bad at all. They've tuned the [16:31] gradient stuff a little bit more, which [16:33] is cool to see. It looks really solid. I [16:36] have been impressed with Opus' frontend [16:38] abilities. Gemini 3 Pro is still also [16:40] really, really good at it. I know a lot [16:42] of people that hate the Gemini 3 Pro [16:43] model for everything other than [16:45] Tailwind. Funny enough, one of those [16:47] people is my channel manager, Ben Davis. [16:48] He wrote his thoughts on 5.2. Since it [16:51] reasons less, it feels way faster than [16:53] GPD 5 and 5.1 did. So, if the speed of [16:55] five and 5.1 is bad for you, but not [16:57] terrible, it's worth giving 5.2 a shot [17:00] because the difference here simply in [17:01] how much fewer reasoning tokens it uses [17:03] might be enough of a jump. Here's [17:05] another example somebody posted of a 5.2 [17:08] UI gen, and it looks really good. I love [17:10] how it did the gradient for this section [17:12] here. It really loves this grid pattern [17:14] like that it puts behind everything. But [17:16] honestly, this is a great UI for a model [17:19] to [ __ ] out. I'm impressed. [17:21] The models all generate UI looking the [17:23] same thing is mostly over. They use [17:25] gradients in a similarish way now, but [17:27] man, the quality of the UIs these models [17:29] are generating compared to even just [17:30] like 6 months ago is hilarious. So yeah, [17:34] pretty cool to see. They also claim it's [17:36] way better at 3D visualization stuff. So [17:39] like if you're using React 3 Fiber or [17:41] 3JS, even stuff like Phaser, they they [17:44] have suggested it'll handle those things [17:45] better. Apparently it generated this and [17:47] it does look really really nice. Like [17:49] this is a cool little visualizer they [17:51] made. Had it make a holiday card builder [17:53] with way too much animation. And you can [17:54] see that gradient pattern, the pink in [17:56] the top left and the blue in the bottom [17:58] right. Pink top left, blue bottom right. [18:01] That's the new gradient pattern all of [18:03] these AI things love to use. This looks [18:07] solid as well. Typeer. [18:14] This is kind of fun. Look at that. It [18:16] made an actual decent game. Who would [18:18] have thought? [18:20] Yeah, I still personally have found for [18:22] 3D stuff in particular, Gemini 3 Pro [18:24] feels a decent bit ahead, which on my [18:27] bench here, Gemini 3 Pro placed [18:30] relatively high at 86%. [18:32] Which is worse than GPT5 did, around the [18:36] same as 5.1 and worse than 5.2. I don't [18:40] know what happened where it's a [18:40] regression in 3D stuff, but I'm not the [18:42] only one saying it. I've talked to a few [18:43] friends who are using it heavily for 3D [18:45] like 3JS type stuff and they've also [18:47] noticed the regression. But if it's [18:50] doing 3D in a more 2D way, like a really [18:53] really fancy version of the hexagon ball [18:55] test, it looks beautiful. Its ability to [18:58] make good-looking things in 3D space [19:00] seems on point, but its ability to [19:01] understand 3D space for my experience [19:03] has not been that great. It almost seems [19:06] like they overindexed on 2D space [19:07] because of ArcGI and through doing that [19:10] kind of broke their 3D understanding. [19:12] Even though again, Flavio disagrees and [19:14] says it's surprisingly good at 3D and [19:16] physics. I just realized I almost forgot [19:18] about one of the most important changes [19:19] with GBT 5.2. They up the price. This is [19:22] the first price increase we've had for [19:23] models in a minute because they had [19:25] dropped the price recently. It was 125 [19:27] in and 10 out for GBT 5 and 5.1, which [19:30] really surprised me. Now it's up a bit [19:32] to 175 in and 14 out. The theory I'm [19:35] seeing many say is that they were [19:37] intentionally releasing smaller [19:38] distilled versions of the model before, [19:39] and 5.2 to is their like coming back [19:42] home type move where they're finally [19:44] putting out the full size real version [19:46] which again I have proven is very [19:48] unlikely with skate bench as silly as [19:50] this bench is these types of gaps mean [19:52] something it means that this model is [19:54] actually worse at some things than the [19:56] previous versions were which is a very [19:57] hard thing to do if GBD5 is a [19:59] distillation of this version it is also [20:01] worth noting that for many tasks the [20:03] model ends up being cheaper because it's [20:04] so much more efficient with its [20:05] reasoning tokens according to openai on [20:08] multiple aentic evals we found that [20:09] despite GPD 5.2's greater cost per [20:11] token. The cost of attaining a given [20:13] level of quality ended up less expensive [20:15] due to GBT 5.2's greater token [20:17] efficiency. To be fair, it says cost of [20:19] attaining a given level of quality. So [20:21] that means to get a certain score, it's [20:23] cheaper, but if you want the best score, [20:25] it's still more expensive. Yeah. And [20:28] then there's the 5.2 Pro pricing, which [20:30] makes me feel sick. $21 per mill in and [20:33] 168 per mill out. A new groundbreakingly [20:36] high price for a model. bit more of the [20:39] facts from OpenAI. It's way better at [20:41] not hallucinating, which is if you've [20:43] been using Gemini 3 Pro, you're back in [20:45] like the 2024 era of hallucinations. I'm [20:48] going to do a whole video about the [20:49] weird quirks of Gemini 3 Pro soon, but [20:52] going back to a GPT model and feeling [20:54] the difference of how much less likely [20:55] it is to hallucinate is insane. It's [20:57] just it doesn't lie anywhere near as [20:59] badly. And it's also way better at [21:00] handling really long context reasoning [21:02] when you're doing long benchmarks and [21:04] finding things in the like the needle [21:06] and the haystack tests. This is an [21:08] insane benchmark. To still have really [21:10] high recall, 98% accuracy at 256k tokens [21:15] is a massive achievement. Massive [21:17] achievement. Gro 4 on this same [21:19] benchmark and 4.1 fast as well was in [21:22] the like 30 percentile. Huge, huge win. [21:26] With eight needles, it still drops down [21:27] to like 70%. But that's way better than [21:30] before in the 30s. They've also had [21:32] massive improvements on vision, massive [21:34] improvements on tool calling. Okay, not [21:35] that massive from 5.1, but it's getting [21:37] higher and higher, hitting like the 98 [21:39] 99% range for a lot of the different [21:42] benches we have for how accurately you [21:44] can call tools. And I haven't seen it do [21:45] a single malformed tool call in cursor [21:47] yet, which is a a huge win considering [21:49] how aggressively it used to do that. [21:51] Better at science and math. It crushed [21:53] dark AGI. And one other important piece [21:56] because like the model range is getting [21:58] complex where we have 5.2 thinking that [22:00] has different reasoning levels between [22:02] minimal, low, medium, high, and the new [22:04] extra high as well as 5.2 Pro [22:06] separately. And then 5.2 instant. 5.2 [22:09] instant isn't actually a separate model. [22:12] It is the no reasoning version of 5.2 [22:14] thinking where you set reasoning to [22:15] none. And it actually is much much [22:17] better. They've been pushing this [22:18] weirdly hard. It seems like they're [22:20] really hyped on 5.2 without reasoning. [22:23] And a couple of my friends, including of [22:24] course Ben, have been saying it's a huge [22:26] improvement as well. Early testers [22:27] particularly noted clear explanations [22:29] that surface key information up front. [22:31] Cool. I haven't experienced that yet, [22:32] but it sounds likely. They continue [22:33] working on the mental health problems [22:35] that these models can cause. Good to see [22:37] improvements here. Awesome. And I have [22:39] two last pieces I want to dive in on. [22:42] Matt Schumer posts awesome reviews of [22:44] the new models. He's similar to me where [22:45] he gets early access. We're in a lot of [22:47] those early calls and groups together [22:50] and his thoughts are really good. Both [22:51] of these will be linked in the [22:52] description, but I just want to go over [22:53] the TLDDRs because it's very aligned [22:55] with my experience. By which way to [22:57] thinking is a meaningful step forward in [22:58] instruction following and willingness to [23:00] attempt hard tasks. Code generation is a [23:02] lot better than 5.1. It's more capable, [23:04] more autonomous, more careful, and [23:05] willing to write a lot more code. Yes, [23:08] vision and long context are much [23:09] improved, especially understanding [23:11] position and images and working with [23:12] huge code bases. Haven't played with [23:15] this enough yet, but I trust him. Speed [23:17] is the main downside. Yes, yes, yes. [23:19] Couldn't agree more. In my experience, [23:21] the thinking mode is very slow for most [23:23] questions, though other testers reported [23:24] mixed results. I almost never use [23:26] instant. Yeah. And 5.2 Pro is insanely [23:29] better for deep reasoning, but it's even [23:31] slower. And every so often it will think [23:33] forever and still fail. Had that a [23:35] couple times, too, where it's like 30 [23:36] minutes in it just stops. It It feels [23:38] like a bug. I don't know if it's going [23:39] to do that over the API or not, cuz they [23:40] finally give us pro over API. According [23:42] to Matt, in the CODC CLI, 5.2 Pro is the [23:45] closest thing he's felt to a proquality [23:46] coding model in a CLI. But the extra [23:48] high reasoning mode that gets it there [23:50] makes it take forever. He's been pushing [23:51] over there to get the pro models into [23:53] his editors for a while. So, I'm very [23:55] happy for him. He bitched about this a [23:57] lot with his 5.1 Pro review, and I feel [23:59] him because the Pro models are [24:01] exceptional. They're just really, really [24:03] slow. So, what does he have to say about [24:04] 5.2 Pro? Undoubtedly the world's best [24:07] model. I can't live without it. That's a [24:09] lot nicer than what he said about 5.2. [24:11] The most capable model available today, [24:12] but it's slow, so it's not for [24:14] everything. It only exists inside of [24:15] chat GPT, not codecs nor the API, which [24:17] is so frustrating. It is now on the API, [24:20] so thank them for that. Expect extremely [24:22] long thinking times on hard tasks. It's [24:24] willing to think for longer than [24:25] previous models, which makes a huge [24:27] difference for the hardest ones. And it [24:28] improves reliability for everything [24:30] else. It has an uncanny ability to infer [24:32] missing context that I didn't provide in [24:34] the prompt. Not just obvious things, but [24:36] constraints I hadn't even realized were [24:38] important myself until it pointed them [24:40] out. I have even felt this a bit with [24:41] 5.1 Pro. So for him to call it out here, [24:44] that's real. Every so often, it will [24:45] think for a long time and still make a [24:47] big mistake, wasting a lot of my time. [24:48] Yeah, that that's because it takes 30 [24:50] minutes to an hour. When it does fail, [24:51] it feels so much worse. Prompting [24:54] matters more than ever, so be explicit. [24:55] Add constraints and refine prompts [24:57] before you send them. After using Pro [24:59] for 2 weeks, I can't live without it. [25:00] It's my go-to for everything I do that [25:02] requires deep thinking, research, or [25:04] coding, or almost any prompt I run that [25:06] doesn't require an instant answer. Yeah. [25:08] And this is somebody who was just [25:10] really, really hyped on Opus 4.5 a few [25:13] weeks ago. So again, I trust Matt [25:15] dearly. Very good reviews. Highly [25:17] recommend reading those in detail if you [25:18] want more information. But I think I've [25:20] said all I have to say here. Seems like [25:21] a very good model. I want to play with [25:22] it more myself. I'm curious how y'all [25:24] feel though. Is 5.2 overhyped or is it [25:26] too slow or is it actually incredible? [25:28] I'm curious how y'all feel. And until [25:30] next time, peace nerds.  ## 📺 Video ![](https://www.youtube.com/watch?v=CtMk0GuQ7cc) --- # 📝 Full Notes (Deep-Read Analysis) # 🏷️ Title: GPT-5.2 is the best model ever made ## 🧰 Action Inbox - [H1] Test GPT 5.2 Pro for deep reasoning tasks — most capable model available #ai-testing - [H2] Compare instruction following between GPT 5.2 vs Opus 4.5 — GPT better at following exact instructions #model-comparison - [H3] Evaluate 5.2 instant mode for faster responses — no reasoning version performs much better #performance-optimization ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ### 🔗 References - [T3 Chat](https://t3.chat) - [ARC AGI Benchmark](https://arcprize.org/) ## 📝 Summary - GPT 5.2 shows massive improvements on most benchmarks but significant regressions on 3D spatial reasoning - Major price increase to $17.5 per million input tokens, first in recent memory - Speed remains the biggest limitation, with Pro models taking 30-50 minutes for complex tasks - Instruction following is significantly better than competing models like Opus 4.5 - New ARC AGI scores of 90.5% represent a 390x efficiency improvement over last year ## 📖 Main Content GPT 5.2 represents OpenAI's latest flagship model with mixed but largely positive results across benchmarks. The model shows exceptional performance on coding tasks (55.6% on SWEBench), mathematical reasoning (100% on AME without tools), and the challenging ARC AGI benchmark (90.5% at $116 per task vs $4,500 a year ago). However, the model exhibits concerning regressions in 3D spatial reasoning. Theo's custom "Skate Bench" - which tests spatial understanding by having models identify skateboard tricks from descriptions - shows GPT 5.2 scoring only 79% compared to GPT 5's 97%, despite using 3x more tokens. The pricing structure introduces complexity with multiple tiers: GPT 5.2 thinking, GPT 5.2 Pro ($210 per million input tokens), and GPT 5.2 instant (no reasoning mode). Speed remains problematic, with Pro models sometimes requiring 30-50 minutes for responses. ## 🗒️ Notes #### ⏱️ [00:00](https://youtu.be/CtMk0GuQ7cc?t=0) — [H1] GPT 5.2 Release and Benchmark Confusion - Key idea: New model shows mixed results with major regression on spatial reasoning - Evidence: "Skatebench shows it as a huge regression. What's going on?" #### ⏱️ [02:45](https://youtu.be/CtMk0GuQ7cc?t=165) — [H2] Skateboard Benchmark Results - Key idea: Custom 3D spatial reasoning benchmark shows significant performance drop - Evidence: "GPT5 would get 97% on this benchmark. When I first ran this benchmark with the new GPT 5.2 model, I got this 4% number" #### ⏱️ [05:12](https://youtu.be/CtMk0GuQ7cc?t=312) — [H2] Pricing Analysis and Performance Cost - Key idea: Extra high reasoning mode costs 5x more but delivers 20% worse performance - Evidence: "GPD5 default, it was about 6 cents per request. And GBT 5.2x high was about 25 cents per request... And the reward for that is a 20% regression in performance" #### ⏱️ [07:30](https://youtu.be/CtMk0GuQ7cc?t=450) — [H1] ARC AGI Breakthrough Results - Key idea: Massive efficiency gains on challenging reasoning benchmark - Evidence: "90.5% and it's only $116 per task. That's a 390x efficiency improvement in one year" #### ⏱️ [09:45](https://youtu.be/CtMk0GuQ7cc?t=585) — [H2] Timeout and Speed Issues - Key idea: Models are extremely slow, sometimes taking 10 minutes per simple request - Evidence: "It took like up to 10 minutes per request at times for naming a skate trick that I can do in 5 seconds" #### ⏱️ [12:20](https://youtu.be/CtMk0GuQ7cc?t=740) — [H1] GDP Val Professional Performance - Key idea: Model matches or beats professional experts on 70.9% of knowledge work tasks - Evidence: "5.2 thinking beats or ties the top industry professionals on 70.9% of the comparisons" #### ⏱️ [15:35](https://youtu.be/CtMk0GuQ7cc?t=935) — [H2] Code Generation Testing - Key idea: Direct comparison shows GPT 5.2 follows instructions better than Opus - Evidence: "GPT5 just did it first shot. That's the difference. I had to do two follow-ups with Opus to get it to behave" #### ⏱️ [18:50](https://youtu.be/CtMk0GuQ7cc?t=1130) — [H2] Frontend and UI Generation Quality - Key idea: Significant improvements in UI generation with consistent gradient patterns - Evidence: "Pink top left, blue bottom right. That's the new gradient pattern all of these AI things love to use" #### ⏱️ [21:15](https://youtu.be/CtMk0GuQ7cc?t=1275) — [H1] Price Increase Details - Key idea: First price increase in recent memory, up from $12.5 to $17.5 per million tokens - Evidence: "It was 125 in and 10 out for GBT 5 and 5.1... Now it's up a bit to 175 in and 14 out" #### ⏱️ [23:40](https://youtu.be/CtMk0GuQ7cc?t=1420) — [H2] GPT 5.2 Pro Extreme Pricing - Key idea: Pro model pricing reaches new highs at $210 per million input tokens - Evidence: "$210 per mill in and 168 per mill out. A new groundbreakingly high price for a model" #### ⏱️ [25:20](https://youtu.be/CtMk0GuQ7cc?t=1520) — [H2] Hallucination Improvements - Key idea: Significant reduction in hallucinations compared to competitors - Evidence: "Going back to a GPT model and feeling the difference of how much less likely it is to hallucinate is insane" #### ⏱️ [27:45](https://youtu.be/CtMk0GuQ7cc?t=1665) — [H1] Matt Schumer's Professional Assessment - Key idea: Expert reviewer calls it "undoubtedly the world's best model" despite speed issues - Evidence: "Undoubtedly the world's best model. I can't live without it... After using Pro for 2 weeks, I can't live without it" ## 📚 References - [T3 Chat Platform](https://t3.chat) - [ARC AGI Prize](https://arcprize.org/) - [SWEBench Evaluation](https://swebench.com/) - [Matt Schumer's Model Reviews](https://twitter.com/mattschumer_) ## 🗣️ Paraphrase - GPT 5.2 achieves breakthrough performance on most benchmarks including 90.5% on ARC AGI with 390x better cost efficiency than previous versions, but shows concerning regressions in 3D spatial reasoning tasks - The model significantly outperforms competitors at instruction following and produces higher quality code and UI generations, though it requires substantially more time and cost for complex reasoning tasks - Despite being the most capable model available, speed limitations and high pricing (especially for Pro tier at $210/million tokens) create practical barriers for everyday use ## 📰 Newsletter **GPT 5.2: The Most Capable Yet Controversial AI Model Release** The latest GPT 5.2 brings unprecedented capabilities with a 90.5% score on the challenging ARC AGI benchmark - representing a massive 390x efficiency improvement over last year's attempts. For developers, this means more reliable code generation, better instruction following, and significantly improved UI design capabilities. However, the release comes with important trade-offs. While excelling at most tasks, the model shows surprising regressions in 3D spatial reasoning. More critically for practical use, response times can stretch to 30-50 minutes for complex tasks, and pricing has increased substantially to $17.5 per million input tokens. The standout feature remains instruction adherence - GPT 5.2 consistently follows directions more precisely than competitors like Opus 4.5, requiring fewer follow-up prompts to achieve desired results. **Key Takeaways for Developers:** - Consider GPT 5.2 instant mode for faster responses on simpler tasks - Budget for higher costs but expect better first-attempt success rates - Pro tier offers exceptional capability but requires patience for complex reasoning Resources: [T3 Chat for Model Access](https://t3.chat), [ARC AGI Benchmark Details](https://arcprize.org/), [SWEBench Code Evaluation](https://swebench.com/) ## ✅ Action Items #### DevRel - Developer experience: Document GPT 5.2 integration patterns, API timeout handling - Developer marketing: Create cost-benefit analysis content comparing model tiers - Community management: Gather feedback on speed vs capability trade-offs #### Community - Contributors: Share benchmark methodologies for spatial reasoning evaluation - Ecosystem: Coordinate model comparison frameworks across projects #### WordPress - Core: Evaluate GPT 5.2 for automated testing and code generation workflows - Documentation: Test model for technical writing and API documentation generation - Meta: Assess cost implications for AI-assisted moderation and support ## 🧪 Hypotheses - GPT 5.2's spatial reasoning regression indicates over-optimization for 2D pattern recognition at expense of 3D understanding - Instruction following improvements stem from better training on precise task completion rather than creative interpretation - Price increases reflect true computational costs rather than market positioning, given performance gains on complex benchmarks - Speed issues will limit adoption for real-time applications despite capability advantages ### 📊 Methodology - Comparative benchmarking: Run identical prompts across GPT 5.2, Opus 4.5, and Gemini 3 Pro measuring accuracy, speed, and cost - Spatial reasoning evaluation: Develop standardized 3D understanding tests beyond skateboard trick identification - Cost-effectiveness analysis: Track token efficiency and success rates across different complexity levels ### ❓ Questions - Will OpenAI address spatial reasoning regressions in future updates or consider them acceptable trade-offs? - How will competing models respond to GPT 5.2's benchmark leadership, particularly in ARC AGI performance? - Can the extreme response times be optimized without sacrificing the model's deep reasoning capabilities? ### 🔮 Future Research - Long-term performance tracking across diverse reasoning tasks to identify consistent capability gaps - Speed optimization techniques for production deployment of high-reasoning models - Cost modeling frameworks for enterprise AI adoption with premium-tier models ### 💡 Implications - Practice: Developers must choose between speed and capability, requiring workflow redesign around longer response times - Policy: Pricing increases may limit access to advanced AI capabilities for smaller organizations - Education: Training materials need updating to reflect new instruction-following paradigms and cost considerations - Funding: Budget planning must account for significantly higher inference costs for complex reasoning tasks ### 🧠 Learning Styles Assessment Reading level estimate: 12th grade #### 👀 Visual learners - Present: Benchmark charts, UI generation examples, gradient pattern demonstrations - Improve: Architecture diagrams showing reasoning pathways, cost comparison visualizations #### 🔊 Aural learners - Present: 28-minute detailed video walkthrough with live demonstrations - Improve: Audio summary highlighting key performance metrics and trade-offs #### 🗣️ Verbal learners - Present: Clear explanations of technical concepts, model comparisons, practical analogies - Improve: Glossary of benchmark terminology, step-by-step reasoning explanations #### 🧩 Kinesthetic learners - Present: Live coding demonstrations, real-time benchmark running, hands-on model testing - Improve: Interactive cost calculators, benchmark reproduction guides #### 🧠 Logical learners - Present: Performance metrics, cost analyses, systematic comparisons across models - Improve: Decision trees for model selection, ROI calculation frameworks #### 🧑‍🤝‍🧑 Social learners - Present: Community feedback integration, expert reviewer quotes, shared experiences - Improve: Discussion prompts for model selection decisions, peer comparison frameworks #### 🧘‍♀️ Solitary learners - Present: Detailed individual testing methodology, personal benchmark creation - Improve: Self-assessment rubrics for model evaluation, reflection questions on use cases #### 🌿 Naturalistic learners - Present: Ecosystem thinking about model relationships and evolution - Improve: Connections to broader AI development patterns, sustainability considerations ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Clear heading hierarchy, descriptive timestamps, alt text for benchmark charts - Improve: More detailed descriptions of visual UI generation examples #### Keyboard navigation - Present: Video controls accessible, timestamp links functional - Improve: Keyboard shortcuts for section navigation #### Contrast and readability - Present: High contrast in video interface, clear text overlays - Improve: Ensure benchmark chart text meets 4.5:1 contrast ratio #### Text resizing and reflow - Present: Responsive video player, scalable text content - Improve: Test transcript readability at 200% zoom #### Media and captions - Present: Full transcript provided with technical detail - Improve: Captions with speaker identification, technical term highlighting #### Forms and interactions - Present: Simple timestamp navigation - Improve: Accessible controls for playback speed adjustment ### 🌍 DEIB Assessment #### Cultural breadth - Present: Focus on universal technical concepts, global AI development impact - Improve: Examples from diverse geographic regions, multi-currency cost analyses #### Inclusive language - Present: Technical terminology clearly defined, gender-neutral references - Improve: Plain language summaries, avoid technical jargon overuse #### Neurodiversity support - Present: Well-structured content with clear sections, predictable formatting - Improve: Content warnings for rapid technical demonstrations, processing time considerations #### Representation - Present: Diverse perspectives through multiple reviewer quotes - Improve: Include voices from different organizational sizes, use case contexts #### Psychological safety - Present: Acknowledges model limitations, presents balanced viewpoints - Improve: More explicit discussion of cost barriers, accessibility implications ### ✅ Logical Validity #### Strong reasoning - Deductive strength: Solid benchmark methodology leading to specific performance conclusions - Inductive strength: Multiple test cases supporting general claims about instruction following - Causal reasoning: Clear connections between design choices and performance outcomes - Analogical strength: Appropriate skateboard benchmark analogy for spatial reasoning #### Fallacies - Cherry picking: Acknowledges benchmark limitations and potential bias in custom testing - Hasty generalization: Appropriately qualified claims about model capabilities with specific contexts #### Tone of voice - Overall tone: Technical but accessible, balanced criticism with enthusiasm, practical focus on real-world implications ### 🛡️ DARVO Analysis - Deny: No significant denials identified - acknowledges both strengths and weaknesses - Attack: No attacks on critics - presents balanced view of --- # 📄 Full Transcript *Auto-generated from YouTube captions · timestamps roughly every 30 seconds · [watch on YouTube](https://www.youtube.com/watch?v=CtMk0GuQ7cc)* **[00:00]** Where's the guy with the counter? Because there's a new best new model. Yes, really. GPT 5.2 just dropped and it is really, really good. Which is why it is uh Wait, what? Skatebench shows it as a huge regression. What's going on? I thought this was the best new model. Well, in many ways it is, but in a handful of important ones, it isn't. And I haven't seen many covering this in detail. Sure, it's better at code and tool calls, and yeah, it's crushing ARC AGI, but there is a depth to using these models that isn't really being shown in **[00:30]** a lot of the coverage I'm seeing. And I want to break down what it's actually like to use because I've been lucky enough to be using it for the last week or so. Very thankful to OpenAI for giving me early access. That said, no money has exchanged hands. The only person paying me is today's sponsor. We need to be realistic about how much easier our jobs are now. AI has made everything from feature additions to bug fixes simpler than it's ever been. Navigating your codebase, finding the right file, and making the changes you need is really easy, but there are a few things that are still hard and also really scary if you get them wrong. The two that I think of the most are authentication and authorization of **[01:00]** users and payment processing. Getting these things wrong can be disastrous. And even if you think you got them right, something might prop up later on that makes you regret building it yourself entirely. Not only have I done this myself incorrectly, I've even written detailed documentation on how to do it right, that barely feels necessary anymore because of today's sponsor, Clerk. If I was building a new app today, this is what I would choose both for authentication and authorization of my users and for payment processing because it is the best and simplest way to do both. Period. And I've used every **[01:31]** single option. They aren't paying me to say that. They're paying me to mention them. I'm telling you, as a person who's went through all of the options, especially on the payment side, Clerk has found this perfect integration of things as well as developer experience that makes it easy to build a secure, reliable application for everything from signing in to signing up. There's something beautiful about seeing a component like this for something as annoying as payment processing and managing what users access to what features. You want to protect a feature so that you only have access if you're paying for the team plan. It's this easy. protect feature equals team **[02:04]** access. And now, as long as you have that set on the server side when you define these things in their dashboard, you're good to go. You could even check if they have a thing like the bronze plan. And if they don't, you can return an error. Do you understand how annoying these things are to do traditionally and how hilariously easy they are to do with Clerk? They even have a pricing table component. you know, the fancy like comparison of the different tiers that is fully integrated, has the Stripe pop overview built into it, and once they subscribe, it's immediately linked to that user's account. It's so much easier **[02:36]** than the other solutions. I genuinely wish this existed when I started T3 Chat. It would have made our lives so much easier. Like, I'm talking not weeks, but months of work. If you're ready to have real users and make real money, look no further. Check them out now at soyb.link/clerk. There's a lot to talk about with this new model, but I want to resolve the click baiting I just did with that whole benchmark thing. There are a lot of layers to this that we'll get to later, but I want to just show the raw numbers that I just ran. And I've spent a lot of money running this test over and over again to be sure of my results. I even went back and forth with some of the **[03:06]** people on the research side trying to figure out what caused this regression. My current theory is that the new model is not as good at three-dimensional reasoning because my benchmark is about skateboard tricks. I give a description of a trick and I expect the model to tell me what the name of that trick is. And previously the highest score I ever got was with GBT5. Not even high. The default setting would get 97% on this benchmark. 03 Pro would get 96%. So like GBT5 was incredible at the spatial reasoning necessary to describe a skateboard trick accurately. When I **[03:37]** first ran this benchmark with the new GPT 5.2 model, I got this 4% number and I was like, "What the [ __ ] went wrong here?" I did a lot of back and forth with the team and we realized that my harness due to some changes in how the GPT 5.2 model defaults, which is it defaults to no reasoning, the results ended up being lower. So like, okay, cool. I'll run it on high and on the new extra high reasoning options, which ended up being comically more expensive to run this bench on. Again, compared to GPD5 default, it was about6 cents per request. And GBT 5.2x high was about 2.5 **[04:10]** cents per request. Yes, it was about five times more expensive. And the reward for that is a 20% regression in performance. Is naming skateboard tricks the best use case for LLMs? Probably not. But I was so pumped to see GPT5 understand spatial reasoning well enough to do this. And this is a massive regression. Like absurdly so. It's now tied with models like Grock 4. What what went wrong there? But it seems like my benchmark is novel in this case because almost every other bench is showing a **[04:41]** very different story like GBT 5.2 thinking getting a 70.9% on GDP val versus 5.1 thinking and five getting a 38.8%. SW Bench saw a big bump at 55.6%. SWB verified saw a first time score for the OpenAI team of an 80%. GBQA Diamond was a nice boost. AME was now 100% with no tools which is really good. It means it can solve math problems without using math tools. Impressive, but there's a lot more to these stories. That all **[05:11]** said, the Arc AGI scores are actually absurd. From ARC prize themselves, a year ago, we verified a preview of an unreleased version of OpenAI's 03 high that scored an 88% on ARC AGI1 at an estimated $4.5,000 per task. Pretty absurd how expensive it was for them to run that special version of 03 for this, but uh it was able to get a high score. And now the extra high option on GBT 5.2 Pro, which is also notably not GBT 5.2, scores a 90.5% and **[05:42]** it's only $1164 per task. That's a 390x efficiency improvement in one year. ArcGI is a wild benchmark. I have covered it many different times. If you're not familiar, go to their website and try it yourself because it's a thing that's easy for humans and nearly impossible for LLMs. It is no longer nearly impossible for these LLMs. It's actually really, really impressive. They made a V2 of this leaderboard that was meant to be actually impossible for LLMs to solve, thinking it would be like the final benchmark ever. Yet, here we are with more and more models getting high scores **[06:13]** on it. We just had Gemini 3 Pro have a groundbreaking 30% and then the Gemini 3 Pro deep research, whatever the hell they call their like heavy version getting all the way into the 40s. And now GBD 5.2 Pro High is scoring way, way higher here. 54.2% 2% for $15.72 per task. But then my favorite part, due to API timeouts, we were unable to reliably verify GBT 5.2 Pro XH high on RKGI2. Turns out these models are still **[06:44]** timing out all over the place. It's kind of insane how long 5.2 Pro will run for. I didn't even put it in my most recent run of Skate Bench because it took like up to 10 minutes per request at times for naming a skate trick that I can do in 5 seconds, not even. It's kind of absurd how hard the new models will go. And even 5.2 extra high was able to take 240 seconds per request. That's 4 minutes per request sometimes for naming a skate trick. The first example they open with is the economically valuable tasks section which includes GDP val **[07:15]** which is a benchmark they made for evaluating how well different models behave against known knowledge work tasks that you have to have like a degree or something for. They take actual examples of work and ask the model to perform against it and see if it performs at or above a human expert level. And 5.2 thinking beats or ties the top industry professionals on 70.9% of the comparisons on the GDP file knowledge work tasks according to expert human judges. That's the other interesting piece here is this benchmark needs to be judged by humans. So I can't just run it on my laptop. And it's a **[07:46]** massive jump from GPT5 which was a 38.8% to 70.9 for 5.2 thinking and 74.1 for 5.2. 2 Pro. And there's no chart crimes happening here either, which is nice. Don't worry, we'll have plenty of chart crimes later. While reviewing one especially good output, one GDP Val judge commented, "It is an exciting and noticeable leap in output quality. It appears to have been done by a professional company with staff and has a surprisingly well-designed layout and advice on both deliverables, though with one we still have some minor errors to correct. Yeah, apparently very **[08:18]** impressive. But on the topic of design, we should see how it handles design work. I have a stock Dex.js project that I haven't made any changes to. We're gonna do my favorite, the image gen bench where I tell it to make an image generation studio mock and uh apparently cursor is breaking and I am now past my usage. I'm on the $20 plan so the fact that I'm just hitting that now is really cool. Obvious bias. I am an investor in cursor so account for that. This is also one of my first times using 5.2 incursor because I didn't do the thing where I set up the open AI like API manually **[08:50]** incursor. I have told this to the team many times and I will tell them again right now. Hi friends at cursor. I should be able to set a custom OpenAI API endpoint without having it break every other model in your app. It's very annoying that if I go set an OpenAI API endpoint that I can no longer use Opus or Composer or any of the other models that I like to use in Cursor. It's quite obnoxious. I know I'm unique that I get early access and I have a need to test these things. But it basically makes the feature of the checkbox to put in a custom API endpoint entirely useless to me. The majority of that was uncut, but **[09:21]** I'll let you know roughly how much time this generation takes. It isn't very fast. These models are still not very fast. It's the one big pain point I have had with the GPT5 series now that I'm experiencing Opus 4.5 and especially the composer model and cursor way more. It is obnoxious how slow the GPT5 series is, especially if you take advantage of some of the fancy pro model stuff, which like I've had the Pro models take 30 to 50 minutes to respond to things. It's awesome that they can go out and do work and come back with a correct answer so **[09:52]** reliably, but god damn, those models are slow. I thought this was going to be fast enough that I could talk over it and give you guys like a realistic feeling for how long it would take. I overestimated just how fast it could be. So, we'll come back to GBT 5.2 and cursor in a moment. Almost forgot to mention, as y'all probably expect, we threw all of the new models in T3 chat. So, if you want access to all of the versions, including the new reasoning one that's a little restricted on the free tier on chat GPT, for eight bucks a month, you can use all these here. If you want your first month for only $1, use code 5.2 at checkout. Anyways, back **[10:23]** to whatever the heck I was just talking about. GBD 5.2 Thinking sets a new state-of-the-art of 55.6% on SWEBench Pro, a rigorous evaluation of real world software engineering. Also note, this is GPT 5.2 thinking, not GPT 5.2 to codeex which will probably come in the near future. It does seem like they're finally realizing how bad the terminology for the models is and they might not do the whole codeex thing again. I I hope that they drop the overuse of that word. It just they should have called codec cli GP terminal. They should have had the model **[10:55]** be the code edition or something, but just calling everything codeex is obnoxious. Regardless, this SWB verified bench only tests Python, and SBench Pro tests four languages and aims to be more contamination resistant, challenging, diverse, and industrially relevant. In the Pro version, they got the new highest score on the extra high version of 56%. But very interestingly, the 5.1 Codeex version when given max extra high, so like allowed to just use as much context as it wants, it actually performed slightly worse than the normal high version does. the the way rein are **[11:27]** being used is increasingly weird, especially with the new defaults and the behaviors we're seeing here. Like GPT5 on my benchmark got a 97% doing 600 tokens per request average. GBD5.2 extra high got a 79% an 18 point regression while using over three times the number of tokens. Interesting. For everyday professional use, this translates into a model that can more reliably debug production code, implement feature requests, refactor large code bases, and ship fixes end to **[11:57]** end with less manual intervention. They also said it's better at front end. So, we'll look at that in a sec cuz the generation did finish. But, I want to first talk a little bit about my experience using this to generate my own tests. I maintain a handful of different benchmarks and evaluate bench in particular. had some changes I wanted to make around how the caching worked because some of these new models would hit errors and I would need to rerun it, but the cached errors would keep the model from rerunning. That plus all the weird things about usage and I **[12:28]** really really wanted to start tracking token utilization so I could include that in my coverage here as well as these awful run times. So I had to make some changes for that. So I wrote a prompt make the following changes to this project. One, cache should also include token counts and durations. Two, errors should not be cached. So a rerun should re-trigger any jobs that errored out. And three, you should show the average token usage in the table in the CLI view. Think this is a pretty clear set of things to do. I had this come through with composer, with Opus, and **[12:58]** with GPT 5.2. The composer version came back almost instantaneously with changes that were mostly good. I didn't like how it was doing the cost calculation and the token calculation though. It was relying on a weird sub field instead of just using what the SDK gave me. So I rejected that. I don't know why it has a thumb there cuz I picked opus, not composer. The opus version seemed like it was really good and it was half as much code as the others until I looked at it more closely and realized that it wasn't actually using the cache for the token counts or for the times like it was expected to. So that was incredibly **[13:28]** frustrating. So I told it to start caching these things and it did, but it didn't use the cache results for [ __ ] anything. But I had to remind it later on with a follow-up message because I I didn't notice this mistake until I reran. was like, "Wait, you just screwed this up." Also, it was tracing input tokens, which is the same for every single run, so it doesn't matter. So, I told it to drop that, but it still entirely forgot to restore durations when loading from cache. So, I had to do two follow-up prompts with Opus 4.5 to get it working how I expected. That all said, I have not actually tried the chatbt 3.2 version. I gave the code a **[13:59]** quick look and it seemed fine, but that's not like me. I want to actually test it. Let's do that. Copy work path. Hey uh friends at cursor who are probably watching this. At no point did I do anything that would result in these changes from happening. In fact, the thing that is being deleted here is the correct version that works. And the thing that's here, the green, the proposal, is entirely wrong and incorrect. There's no review anything here. I have no idea why this is here at all. It's wrong and bad and broken. It's a reversion of something that hasn't **[14:30]** existed in this codebase forever, if ever. Why is this here? What the [ __ ] wrong with the UI? I don't know what's went wrong with the review mode, but it's getting really, really egregious lately. But now that I have fixed the package JSON, to be very clear, this is not broken because of GBT 5.2. This is broken because of cursor. Let's run the test. And look at that. The one from GPT 5.2 did everything right first shot. So the the reason I just tested all of that is I really wanted to emphasize the difference in I guess vibe when I'm using GPT models, **[15:01]** especially 5.2, Compared to models like Opus and Composer, GPT5 is just the series that follows instructions the best. That's the best I can put it. The Opus models will roughly finish the task you give them, and they're very smart. Their ability to figure out what's wrong, debug, and push is incredible. Opus models can turn through insane tasks for long amounts of time and actually generate something that works. But the GPT models will do what you [ __ ] tell them to do. I had to do two follow-ups with Opus to get it to behave. It wrote slightly better code that is closer to **[15:32]** what I would have written, but I had to tell it what to do with multiple follow-ups. GPT5 just did it first shot. That's the difference. I have had a much better time with the GPT models for that. But what's extra funny is GPD 5.2 still took longer than me running Opus, testing the results, realizing there were things wrong, reprompting it, and getting a new result. So, if you want a model that does what you say and you're willing to wait, 5.2 is incredible. If you want a model that is really really smart, possibly even smarter than you for the things that you do, but loves to **[16:04]** just go off on its own little tangents that you have to like grab your like harness and pull it back in. Opus is great for that. And even then, Opus 4.5 is a significant improvement in instruction following in my opinion from how I've experienced previous enthropic models. They seem too happy to go change code and not happy enough to do what you tell them to. Enough of all this. I want to see my image generation studio. So, let's see how this one came out. Not bad at all. They've tuned the gradient stuff a little bit more, which is cool to see. It looks really solid. I **[16:36]** have been impressed with Opus' frontend abilities. Gemini 3 Pro is still also really, really good at it. I know a lot of people that hate the Gemini 3 Pro model for everything other than Tailwind. Funny enough, one of those people is my channel manager, Ben Davis. He wrote his thoughts on 5.2. Since it reasons less, it feels way faster than GPD 5 and 5.1 did. So, if the speed of five and 5.1 is bad for you, but not terrible, it's worth giving 5.2 a shot because the difference here simply in how much fewer reasoning tokens it uses might be enough of a jump. Here's another example somebody posted of a 5.2 **[17:08]** UI gen, and it looks really good. I love how it did the gradient for this section here. It really loves this grid pattern like that it puts behind everything. But honestly, this is a great UI for a model to [ __ ] out. I'm impressed. The models all generate UI looking the same thing is mostly over. They use gradients in a similarish way now, but man, the quality of the UIs these models are generating compared to even just like 6 months ago is hilarious. So yeah, pretty cool to see. They also claim it's way better at 3D visualization stuff. So **[17:39]** like if you're using React 3 Fiber or 3JS, even stuff like Phaser, they they have suggested it'll handle those things better. Apparently it generated this and it does look really really nice. Like this is a cool little visualizer they made. Had it make a holiday card builder with way too much animation. And you can see that gradient pattern, the pink in the top left and the blue in the bottom right. Pink top left, blue bottom right. That's the new gradient pattern all of these AI things love to use. This looks solid as well. Typeer. **[18:14]** This is kind of fun. Look at that. It made an actual decent game. Who would have thought? Yeah, I still personally have found for 3D stuff in particular, Gemini 3 Pro feels a decent bit ahead, which on my bench here, Gemini 3 Pro placed relatively high at 86%. Which is worse than GPT5 did, around the same as 5.1 and worse than 5.2. I don't know what happened where it's a regression in 3D stuff, but I'm not the only one saying it. I've talked to a few friends who are using it heavily for 3D **[18:45]** like 3JS type stuff and they've also noticed the regression. But if it's doing 3D in a more 2D way, like a really really fancy version of the hexagon ball test, it looks beautiful. Its ability to make good-looking things in 3D space seems on point, but its ability to understand 3D space for my experience has not been that great. It almost seems like they overindexed on 2D space because of ArcGI and through doing that kind of broke their 3D understanding. Even though again, Flavio disagrees and says it's surprisingly good at 3D and **[19:16]** physics. I just realized I almost forgot about one of the most important changes with GBT 5.2. They up the price. This is the first price increase we've had for models in a minute because they had dropped the price recently. It was 125 in and 10 out for GBT 5 and 5.1, which really surprised me. Now it's up a bit to 175 in and 14 out. The theory I'm seeing many say is that they were intentionally releasing smaller distilled versions of the model before, and 5.2 to is their like coming back home type move where they're finally putting out the full size real version **[19:46]** which again I have proven is very unlikely with skate bench as silly as this bench is these types of gaps mean something it means that this model is actually worse at some things than the previous versions were which is a very hard thing to do if GBD5 is a distillation of this version it is also worth noting that for many tasks the model ends up being cheaper because it's so much more efficient with its reasoning tokens according to openai on multiple aentic evals we found that despite GPD 5.2's greater cost per token. The cost of attaining a given level of quality ended up less expensive due to GBT 5.2's greater token **[20:17]** efficiency. To be fair, it says cost of attaining a given level of quality. So that means to get a certain score, it's cheaper, but if you want the best score, it's still more expensive. Yeah. And then there's the 5.2 Pro pricing, which makes me feel sick. $21 per mill in and 168 per mill out. A new groundbreakingly high price for a model. bit more of the facts from OpenAI. It's way better at not hallucinating, which is if you've been using Gemini 3 Pro, you're back in like the 2024 era of hallucinations. I'm **[20:48]** going to do a whole video about the weird quirks of Gemini 3 Pro soon, but going back to a GPT model and feeling the difference of how much less likely it is to hallucinate is insane. It's just it doesn't lie anywhere near as badly. And it's also way better at handling really long context reasoning when you're doing long benchmarks and finding things in the like the needle and the haystack tests. This is an insane benchmark. To still have really high recall, 98% accuracy at 256k tokens is a massive achievement. Massive achievement. Gro 4 on this same **[21:19]** benchmark and 4.1 fast as well was in the like 30 percentile. Huge, huge win. With eight needles, it still drops down to like 70%. But that's way better than before in the 30s. They've also had massive improvements on vision, massive improvements on tool calling. Okay, not that massive from 5.1, but it's getting higher and higher, hitting like the 98 99% range for a lot of the different benches we have for how accurately you can call tools. And I haven't seen it do a single malformed tool call in cursor yet, which is a a huge win considering how aggressively it used to do that. **[21:51]** Better at science and math. It crushed dark AGI. And one other important piece because like the model range is getting complex where we have 5.2 thinking that has different reasoning levels between minimal, low, medium, high, and the new extra high as well as 5.2 Pro separately. And then 5.2 instant. 5.2 instant isn't actually a separate model. It is the no reasoning version of 5.2 thinking where you set reasoning to none. And it actually is much much better. They've been pushing this weirdly hard. It seems like they're really hyped on 5.2 without reasoning. **[22:23]** And a couple of my friends, including of course Ben, have been saying it's a huge improvement as well. Early testers particularly noted clear explanations that surface key information up front. Cool. I haven't experienced that yet, but it sounds likely. They continue working on the mental health problems that these models can cause. Good to see improvements here. Awesome. And I have two last pieces I want to dive in on. Matt Schumer posts awesome reviews of the new models. He's similar to me where he gets early access. We're in a lot of those early calls and groups together and his thoughts are really good. Both of these will be linked in the description, but I just want to go over **[22:53]** the TLDDRs because it's very aligned with my experience. By which way to thinking is a meaningful step forward in instruction following and willingness to attempt hard tasks. Code generation is a lot better than 5.1. It's more capable, more autonomous, more careful, and willing to write a lot more code. Yes, vision and long context are much improved, especially understanding position and images and working with huge code bases. Haven't played with this enough yet, but I trust him. Speed is the main downside. Yes, yes, yes. Couldn't agree more. In my experience, the thinking mode is very slow for most questions, though other testers reported **[23:24]** mixed results. I almost never use instant. Yeah. And 5.2 Pro is insanely better for deep reasoning, but it's even slower. And every so often it will think forever and still fail. Had that a couple times, too, where it's like 30 minutes in it just stops. It It feels like a bug. I don't know if it's going to do that over the API or not, cuz they finally give us pro over API. According to Matt, in the CODC CLI, 5.2 Pro is the closest thing he's felt to a proquality coding model in a CLI. But the extra high reasoning mode that gets it there makes it take forever. He's been pushing over there to get the pro models into his editors for a while. So, I'm very **[23:55]** happy for him. He bitched about this a lot with his 5.1 Pro review, and I feel him because the Pro models are exceptional. They're just really, really slow. So, what does he have to say about 5.2 Pro? Undoubtedly the world's best model. I can't live without it. That's a lot nicer than what he said about 5.2. The most capable model available today, but it's slow, so it's not for everything. It only exists inside of chat GPT, not codecs nor the API, which is so frustrating. It is now on the API, so thank them for that. Expect extremely long thinking times on hard tasks. It's willing to think for longer than previous models, which makes a huge **[24:27]** difference for the hardest ones. And it improves reliability for everything else. It has an uncanny ability to infer missing context that I didn't provide in the prompt. Not just obvious things, but constraints I hadn't even realized were important myself until it pointed them out. I have even felt this a bit with 5.1 Pro. So for him to call it out here, that's real. Every so often, it will think for a long time and still make a big mistake, wasting a lot of my time. Yeah, that that's because it takes 30 minutes to an hour. When it does fail, it feels so much worse. Prompting matters more than ever, so be explicit. Add constraints and refine prompts before you send them. After using Pro **[24:59]** for 2 weeks, I can't live without it. It's my go-to for everything I do that requires deep thinking, research, or coding, or almost any prompt I run that doesn't require an instant answer. Yeah. And this is somebody who was just really, really hyped on Opus 4.5 a few weeks ago. So again, I trust Matt dearly. Very good reviews. Highly recommend reading those in detail if you want more information. But I think I've said all I have to say here. Seems like a very good model. I want to play with it more myself. I'm curious how y'all feel though. Is 5.2 overhyped or is it too slow or is it actually incredible? I'm curious how y'all feel. And until **[25:30]** next time, peace nerds.