Cursor just crushed Claude Code - CourtneyR.dev's 2nd Brain

Cursor just crushed Claude Code ![rw-book-cover](https://img.youtube.com/vi/UvUzpSlXKtg/maxresdefault.jpg) 🔗 **Source:** [Cursor just crushed Claude Code](https://www.youtube.com/watch?v=UvUzpSlXKtg) > [!note]+ Document Notes > --- > tags: [cursor-editor, ai-coding, model-training, developer-tools, composer-model, spacex-ai] > author: [MISSING] > source: [MISSING] > reference: https://www.youtube.com/watch?v=UvUzpSlXKtg > date created: {{date:YYYY-MM-DD}} {{time:HH:mm}} > --- > # 🏷️ Title: Cursor just crushed Claude Code > ## 🧭 Table of Contents > - [📝 Summary](#-summary) > - [📖 Main Content](#-main-content) > - [🗒️ Notes](#️-notes) > - [📚 References](#-references) > - [🗣️ Paraphrase](#️-paraphrase) > - [📰 Newsletter](#-newsletter) > - [✅ Action Items](#-action-items) > - [🧪 Hypotheses](#-hypotheses) > - [📊 Methodology](#-methodology) > - [❓ Questions](#-questions) > - [🔮 Future Research](#-future-research) > - [💡 Implications](#-implications) > - [🧠 Learning Styles Assessment](#-learning-styles-assessment) > - [📊 Accessibility Assessment](#-accessibility-assessment) > - [🌍 DEIB Assessment](#-deib-assessment) > - [✅ Logical Validity](#-logical-validity) > - [🛡️ DARVO Analysis](#-darvo-analysis) > - [🎯 Conclusion](#-conclusion) > ## 📝 Summary > - Cursor released Composer 2.5, a coding-focused AI model that significantly outperforms previous versions while being substantially cheaper > - The model achieved 63% on Cursor's internal benchmark, nearly matching GPT-5.5 (64%) and Opus 4.7 (65%) at 10-20x lower cost > - Built on Moonshot's Kimi K25 base model with 10x more compute than the original, trained using SpaceX AI's compute resources > - Introduces novel training techniques like targeted textual feedback and synthetic task generation to improve coding performance > - Only available through Cursor's platform, not via API, creating competitive advantages but limiting external evaluation > ## 📖 Main Content > Cursor has released Composer 2.5, marking a significant advancement in AI coding models that challenges the dominance of major AI labs. The model represents a dramatic improvement in both capability and cost-effectiveness for coding tasks. > ### Performance and Pricing Revolution > Composer 2.5 achieved remarkable results on Cursor's internal benchmark, scoring 63% compared to GPT-5.5's 64% and Opus 4.7's 65%. The critical difference lies in cost: Composer 2.5 operates at roughly 10-20x lower cost than these competing models while delivering comparable performance. > The pricing structure reflects this advantage: > - Composer 2.5: $0.50 per million tokens in, $2.50 out > - Claude Sonnet: $3.00 in, $15.00 out > - GPT-5.5: $5.00 in, $30.00 out > - Opus 4.7: $5.00 in, $25.00 out > ### Technical Innovation > The model builds on Moonshot's Kimi K25 base model but incorporates 10x more compute than the original training. Two key innovations drive its performance: > **Targeted RL with Textual Feedback**: This addresses a fundamental challenge in reinforcement learning where models struggle to identify which specific decisions in long sequences contributed to success or failure. The system provides targeted feedback at problematic points in code generation, using a teacher-student model approach where hints guide the learning process without requiring explicit instructions during inference. > **Synthetic Data Generation**: The team created 25x more synthetic training tasks than Composer 2, including innovative approaches like "feature deletion" where models remove features from codebases and then reconstruct them using existing tests as verification. > ### Strategic Positioning > Cursor's partnership with SpaceX AI provides access to massive compute resources (Colossus 2's million H100 equivalents) while their user data offers unique training advantages. The company collects valuable interaction patterns from developers using their platform, creating a feedback loop that improves model performance. > However, the model remains exclusively available through Cursor's platform, limiting external evaluation and API access. This creates competitive moats but raises questions about independent assessment of capabilities. > ## 🗒️ Notes > #### ⏱️ [00:00:00](https://www.youtube.com/watch?v=UvUzpSlXKtg?t=0) Introduction and Context > - Cursor released Composer 2.5, a major coding model advancement > - Built by company with significant compute resources (SpaceX partnership) > - Represents competitive threat to major AI labs > #### ⏱️ [00:02:30](https://www.youtube.com/watch?v=UvUzpSlXKtg?t=150) Pricing Analysis Deep Dive > - Explains token-based pricing model complexity > - Input tokens cheaper than output tokens > - Three pricing factors: per-token cost, tokens per problem, negotiation leverage > - Labs can subsidize heavily through direct products vs API pricing > #### ⏱️ [00:08:45](https://www.youtube.com/watch?v=UvUzpSlXKtg?t=525) Cursor's Data Advantage > - Cursor collects valuable developer interaction data > - Chat histories with feedback loops extremely valuable for training > - Double-dip strategy: resell models at higher cost while collecting training data > #### ⏱️ [00:12:00](https://www.youtube.com/watch?v=UvUzpSlXKtg?t=720) Composer Evolution > - Composer 1.0: Too expensive, not very good > - Composer 1.5: Better but still overpriced compared to alternatives > - Composer 2.0: 7x cheaper than 1.5, significantly smarter > - Composer 2.5: Same price as 2.0, major intelligence leap > #### ⏱️ [00:15:30](https://www.youtube.com/watch?v=UvUzpSlXKtg?t=930) Benchmark Results > - 63% on Cursor Bench vs GPT-5.5 (64%) and Opus (65%) > - 10-20x cost advantage over competing models > - Based on Kimi K25 with doubled performance through post-training > #### ⏱️ [00:18:00](https://www.youtube.com/watch?v=UvUzpSlXKtg?t=1080) Training Innovations > - Targeted textual feedback for specific behavior correction > - Addresses credit assignment problem in long rollouts > - Teacher-student model approach with contextual hints > #### ⏱️ [00:22:15](https://www.youtube.com/watch?v=UvUzpSlXKtg?t=1335) Synthetic Data Strategy > - 25x more synthetic tasks than Composer 2 > - Feature deletion approach: remove features, then reconstruct using tests > - Reward hacking examples: models finding clever workarounds > #### ⏱️ [00:26:00](https://www.youtube.com/watch?v=UvUzpSlXKtg?t=1560) Practical Testing > - Game reconstruction test with Fish Slop project > - Model demonstrates parallel agent capabilities > - Asset swapping and UI scaling challenges revealed > #### ⏱️ [00:35:00](https://www.youtube.com/watch?v=UvUzpSlXKtg?t=2100) API Access Limitations > - No external API access available > - Only measurable through Cursor's internal benchmarks > - Limits independent evaluation and integration options > #### ⏱️ [00:40:00](https://www.youtube.com/watch?v=UvUzpSlXKtg?t=2400) Future Outlook > - Next model training with 10x more compute (100x vs original Kimi) > - Potential to leapfrog state-of-the-art coding capabilities > - Enterprise adoption advantages over individual lab partnerships > ## 📚 References > - [Cursor](https://cursor.com) > - [Moonshot Kimi K25 model](https://moonshot.ai) > - [SpaceX AI partnership](https://spacex.com) > - [Artificial Analysis benchmarks](https://artificialanalysis.ai) > - [SWE-bench evaluation](https://swe-bench.github.io) > ## 🗣️ Paraphrase > Cursor has achieved a breakthrough in AI coding assistance with Composer 2.5, a model that rivals the performance of leading AI systems while operating at dramatically lower costs. The model demonstrates that focused post-training and domain specialization can create competitive advantages against well-funded general-purpose models from major tech companies. > The technical innovations, particularly targeted textual feedback and synthetic data generation, represent meaningful advances in training methodology. However, the platform-exclusive availability limits broader ecosystem integration and independent verification of capabilities. > ## 📰 Newsletter > 🚀 **Cursor Composer 2.5 Shakes Up AI Coding Scene** > The WordPress development world just got more interesting! Cursor's latest Composer 2.5 model is delivering near-GPT-5.5 performance at 1/20th the cost. For WordPress developers working on complex themes, plugins, or custom solutions, this could be a game-changer. > **What's exciting for WP developers:** > - Lightning-fast code generation perfect for iterative development > - Specialized training on real codebases (hello, better WordPress-specific suggestions!) > - Cost-effective for agencies managing multiple client projects > **Try it out:** If you're already using Cursor for WordPress development, the 2.5 model should be available now. Perfect for those custom block development sessions or complex WooCommerce integrations. > **Community spotlight:** The synthetic training approach (where models learn by breaking and fixing features) mirrors how many of us debug WordPress sites. It's AI learning to think like developers! > ## ✅ Action Items > #### DevRel > - **Developer advocacy**: Create comparison benchmarks between Composer 2.5 and other coding models for WordPress-specific tasks > - **Education**: Develop workshops on effective AI-assisted WordPress development workflows > - **Developer experience**: Test integration capabilities with WordPress CLI and local development environments > - **Developer marketing**: Document cost-benefit analysis for WordPress agencies considering Cursor adoption > - **Community management**: Monitor developer feedback on platform-exclusive model availability concerns > #### WordPress > - **Core**: Evaluate impact of AI coding tools on core contribution workflows and code review processes > - **Core Performance**: Test AI-generated code performance against manual optimizations in WordPress context > - **Documentation**: Update coding standards documentation to address AI-assisted development practices > - **Training**: Create learning paths for developers transitioning to AI-assisted WordPress development > - **Plugins**: Establish guidelines for AI-generated plugin code in directory submissions > #### Community > - **Contributors**: Develop mentorship programs pairing AI tools with human expertise for onboarding > - **Maintainers**: Create templates for AI-assisted code review and testing procedures > - **Ecosystem**: Coordinate with hosting providers on performance implications of AI-generated code > ## 🧪 Hypotheses > - **Cost reduction hypothesis**: Composer 2.5's pricing advantage will drive significant adoption among budget-conscious development teams and individual developers > - **Platform lock-in hypothesis**: Cursor's API exclusivity will create competitive moats but limit broader ecosystem integration and innovation > - **Training data advantage hypothesis**: Cursor's access to real developer interaction data provides sustainable competitive advantages over models trained primarily on static code repositories > - **Specialization effectiveness hypothesis**: Domain-specific post-training (coding focus) can achieve competitive performance against general-purpose models with significantly less compute > - **Enterprise adoption hypothesis**: The combination of cost-effectiveness and performance will accelerate Cursor's enterprise market penetration > - **Innovation catalyst hypothesis**: Platform-exclusive models will pressure major AI labs to improve their own coding-specific offerings > ### 📊 Methodology > **Cost-benefit analysis studies**: Compare development velocity and code quality metrics across different AI coding assistants in controlled WordPress development scenarios. > **Performance benchmarking**: Establish WordPress-specific coding benchmarks measuring plugin development, theme creation, and core contribution tasks. > **Developer experience surveys**: Track satisfaction, productivity metrics, and workflow changes among developers using different AI coding tools. > **Market adoption tracking**: Monitor enterprise adoption rates and developer tool preferences through industry surveys and usage analytics. > ### ❓ Questions > - How does platform exclusivity affect long-term innovation and competition in AI coding tools? > - What are the implications of training AI models on proprietary developer interaction data for the broader development community? > - Can specialized post-training consistently outperform general-purpose models across different coding domains? > - How will major AI labs respond to competitive pressure from specialized, cost-effective coding models? > - What governance and ethical considerations arise from AI models trained on developer workflow data? > ### 🔮 Future Research > - **Comparative effectiveness studies**: Systematic evaluation of coding model performance across different programming languages and frameworks > - **Developer workflow optimization**: Research into optimal human-AI collaboration patterns for different types of coding tasks > - **Training methodology advancement**: Investigation into synthetic data generation techniques for specialized AI model domains > - **Economic impact analysis**: Study of AI coding tools' effects on software development labor markets and productivity > ### 💡 Implications > **Practice implications**: Developers need to evaluate workflow changes required for effective AI collaboration, particularly the shift from batch processing to interactive development sessions. > **Economic implications**: Dramatic cost reductions may democratize access to advanced AI coding assistance, potentially leveling competitive advantages previously available only to well-funded development teams. > **Competitive implications**: Platform-exclusive models create new forms of vendor lock-in that may fragment the developer tools ecosystem while driving innovation through competition. > **Education implications**: Development education must incorporate AI collaboration skills alongside traditional programming knowledge, changing how coding is taught and learned. > ## 🧠 Learning Styles Assessment > **Reading level estimate**: Grade 12-13 (Flesch-Kincaid), appropriate for technical professionals but may challenge general audiences due to specialized AI/ML terminology. > #### 👀 Visual learners > - **Present**: Performance comparison charts, cost analysis tables, technical architecture diagrams > - **Improve**: Add workflow diagrams showing human-AI collaboration patterns, visual timeline of model evolution, before/after code examples > #### 🔊 Aural learners > - **Present**: Video format with detailed verbal explanations and examples > - **Improve**: Provide audio-only version for developers who prefer listening while coding, add podcast-style discussion format > #### 🗣️ Verbal learners > - **Present**: Clear definitions of technical terms, analogies explaining complex concepts, step-by-step explanations > - **Improve**: Add glossary of AI/ML terms, provide plain-language summaries of technical sections > #### 🧩 Kinesthetic learners > - **Present**: Hands-on coding demonstrations, interactive testing scenarios > - **Improve**: Add downloadable test projects, provide step-by-step tutorials for setting up and testing different models > #### 🧠 Logical learners > - **Present**: Systematic cost-benefit analysis, benchmark comparisons, logical progression of technical improvements > - **Improve**: Add decision trees for model selection, quantitative ROI calculations, systematic evaluation frameworks > #### 🧑‍🤝‍🧑 Social learners > - **Present**: Community discussion context, developer experience sharing, enterprise adoption stories > - **Improve**: Add discussion prompts for development teams, collaborative evaluation frameworks > #### 🧘‍♀️ Solitary learners > - **Present**: Comprehensive written analysis, individual testing scenarios, self-assessment questions > - **Improve**: Add personal reflection prompts on workflow changes, individual skill development paths > #### 🌿 Naturalistic learners > - **Present**: Evolution of AI models as ecosystem development, competitive dynamics as natural selection > - **Improve**: Add analogies to biological systems, discuss sustainability of different business models > ## 📊 Accessibility Assessment > #### Screen reader semantics > - **Present**: Clear heading structure, descriptive video content > - **Improve**: Add alt text for performance charts and diagrams, provide transcript with technical term definitions > #### Keyboard navigation > - **Present**: Standard video player controls accessible via keyboard > - **Improve**: Add keyboard shortcuts for technical sections, ensure all interactive elements are keyboard accessible > #### Contrast and readability > - **Present**: Standard video player with good contrast in UI elements > - **Improve**: Provide high-contrast versions of charts and graphs, ensure code examples have sufficient contrast > #### Text resizing and reflow > - **Present**: Responsive video player that adapts to different screen sizes > - **Improve**: Ensure transcript and supplementary materials reflow properly at 200% zoom > #### Media and captions > - **Present**: Video with clear audio and visual demonstrations > - **Improve**: Add professional captions with technical term spelling, provide downloadable transcript > #### Forms and interactions > - **Present**: Basic video player interaction elements > - **Improve**: Add accessible feedback mechanisms for community discussion, ensure error handling for any interactive elements > #### Standards reference > - Map accessibility improvements to WCAG 2.2 AA criteria, particularly around multimedia content and technical documentation > ## 🌍 DEIB Assessment > #### Cultural breadth > - **Present**: Discussion of global AI development competition, enterprise adoption across different markets > - **Improve**: Include perspectives from developers in different geographic regions, discuss accessibility in different economic contexts > #### Inclusive language > - **Present**: Generally professional technical language, avoids most jargon without definition > - **Improve**: Define all technical terms clearly, avoid assumptions about prior AI/ML knowledge > #### Neurodiversity support > - **Present**: Well-structured content with clear sections and logical progression > - **Improve**: Provide content in multiple formats (video, text, audio), add visual organizing elements > #### Representation > - **Present**: Focus on technical capabilities rather than individual personalities > - **Improve**: Include diverse developer perspectives in case studies, highlight contributions from underrepresented groups in AI development > #### Psychological safety > - **Present**: Balanced analysis acknowledging both benefits and limitations > - **Improve**: Address concerns about job displacement sensitively, provide resources for skill development > ## ✅ Logical Validity > #### Strong reasoning > - **Economic analysis**: Well-structured comparison of pricing models with clear mathematical relationships and market dynamics > - **Technical evaluation**: Systematic breakdown of training innovations with specific examples and measurable outcomes > - **Competitive positioning**: Logical analysis of strategic advantages and market positioning based on concrete capabilities > #### Fallacies > - **Hasty generalization**: Drawing broad conclusions about model superiority from limited, internal benchmarks without external validation > - **Appeal to novelty**: Suggesting newer training techniques are necessarily better without comprehensive comparative analysis > - **False dilemma**: Presenting platform exclusivity as purely advantageous without adequately weighing ecosystem fragmentation costs > #### Tone of voice > - **Technical enthusiasm with critical skepticism**: Balances excitement about innovations with realistic assessment of limitations and concerns > ## 🛡️ DARVO Analysis > No significant DARVO patterns detected. The content maintains analytical objectivity while acknowledging both competitive advantages and legitimate concerns about platform exclusivity and evaluation limitations. > ## 🎯 Conclusion > Cursor's Composer 2.5 represents a significant advancement in AI coding assistance, demonstrating that focused domain specialization and innovative training techniques can compete with well-funded general-purpose models. The dramatic cost advantages (10-20x cheaper than comparable models) combined with near-state-of-the-art performance create compelling value propositions for developers and enterprises. > However, the platform-exclusive availability raises important questions about ecosystem fragmentation and independent evaluation capabilities. The success of this approach may pressure major AI labs to improve their coding-specific offerings while potentially creating new forms of vendor lock-in. > For WordPress developers and the broader development community, this represents both opportunity (more accessible advanced AI assistance) and challenge (navigating an increasingly fragmented AI tools landscape). The key practical next step is evaluating how these tools integrate into existing workflows and whether the productivity gains justify potential platform dependencies. > ## 🔄 Refactor Appendix — Cursor just crushed Claude Code > ### 📚 Literature Atoms (H3) > #### Domain-specific post-training can match general models at lower cost > > [!quote] ID:rw_cursor_composer > > Composer 2.5 achieved 63% on Cursor Bench vs GPT-5.5 (64%) and Opus 4.7 (65%) at 10-20x lower cost through specialized coding focus and innovative training techniques > **Context & use**: > - Demonstrates effectiveness of focused domain specialization over general-purpose model scaling > - Evidence for sustainable competitive advantages through training methodology innovation > **Tags**: #ai-training #cost-optimization #competitive-strategy > **Highlight**: [view highlight](https://www.youtube.com/watch?v=UvUzpSlXKtg?t=930) >  > #### Platform-exclusive AI models create competitive moats but limit ecosystem integration > > [!quote] ID:rw_cursor_exclusive > > Unlike every other model, there is no API to hit to test Composer ourselves - the only way to try it is inside Cursor, unless you spend $60 billion to buy them > **Context & use**: > - Highlights new form of competitive strategy in AI model distribution > - Raises concerns about ecosystem fragmentation and independent evaluation > **Tags**: #platform-strategy #api-access #competitive-moats > **Highlight**: [view highlight](https://www.youtube.com/watch?v=UvUzpSlXKtg?t=2100) >  > #### Developer interaction data provides unique training advantages for coding models > > [!quote] ID:rw_cursor_data > > Chat histories where you work with the agent, give it feedback as it generates things, tell it 'you missed this, go change that too' - those are so valuable for training > **Context & use**: > - Explains sustainable competitive advantage from user-generated training data > - Illustrates value of interactive development workflows for model improvement > **Tags**: #training-data #developer-workflows #feedback-loops > **Highlight**: [view highlight](https://www.youtube.com/watch?v=UvUzpSlXKtg?t=525) >  > ### 🧠 Permanent Candidates (H4) > #### Specialized AI training outcompetes generalist scaling > **Claim**: Domain-specific post-training with targeted techniques can achieve competitive performance against general-purpose models using significantly less compute and cost. > **Evidence pointer**: [[#Domain-specific post-training can match general models at lower cost]] > **Scope & conditions**: Holds when domain has sufficient training data and clear success metrics; breaks when tasks require broad knowledge integration > **Links**: > - [[AI model specialization]] — because extends focused training approaches > - [[Competitive AI strategy]] — because contrasts with general-purpose scaling approaches > - [[Developer tool economics]] — because applies-in cost-sensitive professional software markets >  > ### ✏️ Fleeting Queue > - Platform-exclusive models as new competitive strategy - implications for open ecosystem development #fleeting #inbox > - Targeted textual feedback training technique - potential applications beyond coding #fleeting #inbox > - SpaceX AI compute partnership dynamics - precedent for other AI lab collaborations #fleeting #inbox > ### → Things 3 > - [ ] Test Composer 2.5 performance on WordPress-specific coding tasks — compare against current development workflow #things > - [ ] Research Cursor SDK capabilities for potential integration with existing development tools — evaluate API limitations and workarounds #things > --- > ## Full Transcript > [00:00] Last week there was a really cool [00:01] release I'm afraid people are sleeping [00:03] on. It came from a company with a lot of [00:04] compute and it was focused on being a [00:06] small model that is very effective at [00:08] solving code tasks. And no, it wasn't [00:10] Gemini 3.5 Flash, even though it [00:12] probably should have been. It was [00:13] Composer 2.5 by Codium. This is a crazy [00:18] drop. The rate at which Codium has [00:20] caught up to state of the art is pretty [00:23] absurd, but there are catches here that [00:26] we need to talk about as well. If you're [00:28] not familiar with the Composer line of [00:30] models from Codium, this has been their [00:32] attempt to distill good open weight [00:34] models into a great coding model for [00:37] usage within Codium. That said, it's [00:40] only really for use within Codium. So, [00:42] it's hard for us to bench it externally. [00:45] That means we kind of got to vibe our [00:47] way through it, but it also means Codium [00:50] does to an extent as well. They have [00:51] published some awesome numbers including [00:53] their own benchmark, which sadly is not [00:56] particularly public, the Codium bench. [00:58] But the results seem to be incredible, [01:00] and I can tell you from my own use of [01:01] it, this isn't just hype. While I don't [01:04] think this is the greatest model ever [01:06] and you should use it for everything, I [01:07] do think it's worth looking at and [01:09] understanding both because it's a good [01:12] model release from a company that [01:14] understands what we need as devs, but [01:15] also because it shows that the strength [01:18] that the major labs have might be [01:20] starting to crack a little bit under [01:22] pressure. While I am an early investor [01:24] in Codium, the acquisition stuff [01:27] happening with SpaceX kind of destroys [01:29] any incentive I might have had to shill [01:31] it super hard. So, I'm not sharing this [01:34] cuz I'm biased. I'm sharing this cuz [01:35] it's actually really interesting to me. [01:37] And don't worry, I will [ __ ] all over a [01:38] lot of other Codium things throughout [01:40] this, too. All of that said, want to [01:42] make sure it's very clear Codium is not [01:43] paying me at all for this coverage. The [01:45] only people paying me today are the [01:47] sponsor. As powerful as continuous [01:49] integration is, it's full of [01:50] compromises. I like to call this the CI [01:52] compromise triangle. The different [01:54] things you have to to about when you're [01:55] configuring your continuous integration. [01:57] You start with convenience, which is how [01:58] easy is it to implement this? You also [02:00] have performance. How long does it take [02:02] to run? Be nice if the CI runners were [02:03] just a little bit faster. And of course, [02:05] you have cost. Obviously, there's lots [02:07] of different solutions for CI that have [02:09] different compromises across this [02:10] triangle because it would be impossible [02:12] for anything to just be better at all of [02:14] these, right? Like there's no way [02:15] something could be more convenient, [02:16] faster, and cheaper, right? [02:20] Oh, is it a Blacksmith ad? Yeah. [02:22] Blacksmith is 2x faster and 60% cheaper [02:24] for most people when they move from [02:25] GitHub Actions. I thought this was too [02:27] good to be true, as did many of their [02:28] customers like Mercury, Superbase, [02:30] Dscript, Exa, and more. But then you use [02:32] it and you realize you can't live [02:33] without it. If it was just cheaper, it [02:35] would be worth it. If it was just [02:36] faster, it would be worth it. And if it [02:38] was just more reliable, it would still [02:39] be worth it. But then you discover the [02:41] console and how much better it is than [02:43] anything GitHub has shipped in years. [02:45] You get actual metrics about the [02:47] workflows you're running, what ones have [02:49] high error rates, what the errors might [02:50] be, as well as logs that give you really [02:52] useful details on what is going on [02:54] across all of your CI. Every time I [02:56] forget to set up Blacksmith on a new [02:58] project, I regret it immensely and then [03:00] I move over and watch my build times go [03:01] from 3+ minutes to under one. Your devs [03:03] and your agents is our fast CI. Get it [03:05] today at solidity.link/blacksmith. [03:07] So, let's talk about this new, [03:09] intelligent, fast, cheap model that [03:12] Cursor put out. First, I want to talk [03:14] about price. A lot of people don't [03:16] understand how models are priced. I'm [03:18] going to do my best to explain it simply [03:20] here. You have the input and the output [03:23] costs and these are in million token [03:25] groups. A token is just a chunk of text, [03:28] usually three to five characters, kind [03:30] of like a word, that is how models [03:32] interpret text and then generate text as [03:35] well. Input tokens tend to be a lot [03:36] cheaper than output tokens because the [03:38] inputs are the things you're passing the [03:39] model and it's just using that to set up [03:42] what its guesses for next token [03:43] prediction are going to be. The outputs [03:44] are the actual work it's doing, so those [03:46] tend to be much more. The base price for [03:48] a model like Claude for Sonnet has [03:49] historically been $3 per million tokens [03:51] in and $15 per million tokens out. On [03:54] the 1 million token context versions, [03:57] they would sometimes be more expensive. [03:58] Anthropic has since stopped doing that, [04:00] OpenAI has not, and the Opus models tend [04:02] to be much more expensive. Like 4.5 Opus [04:04] is $5 per mil and 25 per mil out. GPT 5 [04:08] to 5.2 were actually very generously [04:10] priced in my opinion at $1.25 per mil in [04:14] and 10 per mil out. 5.2 was a bit of a [04:16] jump to $1.75 in and 14 out. 5.4 was [04:20] another jump to 250 in and 15 out, and [04:22] then 5.5 was a massive jump to $5 in and [04:25] 30 out, doubling what it was in 5.4. It [04:27] seems like the prices are going up [04:29] across the industry, but they aren't [04:31] necessarily. There's two additional [04:33] layers to the pricing that are important [04:35] to consider. The first is the amount of [04:37] tokens that are actually being [04:38] generated. Flashbang warning for [04:40] artificial analysis. Artificial analysis [04:42] measures roughly how intelligent models [04:44] are on their own benchmarks internally [04:47] that they publish to the world, but they [04:49] also measure other pieces. In [04:50] particular, how fast the model solve the [04:52] problem, how many tokens they generate, [04:54] and how much it cost to run. Output [04:56] tokens are one of my favorite things to [04:57] look at here, and you'll notice some [04:59] interesting stuff. The most output [05:01] tokens didn't come from the smartest [05:02] models. It came from DeepSeek V4 Flash, [05:05] GPT 5.4 Mini, and Claude Sonnet 4.6. [05:09] Smaller models often generate way more [05:11] tokens than they need to to solve [05:14] problems. It's also worth noting that [05:16] the lab trying the hardest to reduce [05:18] this overuse of tokens very much appears [05:20] to be OpenAI, where models like 5.5 only [05:23] did 75 mil tokens for the same bench [05:26] that Sonnet did 200 mil for. Almost a [05:29] third the number of tokens and a higher [05:31] score as well. And when you add in [05:33] models like 5.5 low and medium, which [05:35] still score really well in the benches, [05:37] like 5.5 medium scores as well as Opus [05:40] 4.7 roughly, and the low version is [05:43] worse, but it's still much higher than [05:45] 5.4 Mini and Flash and roughly around [05:47] where something like Sonic 46 is. [05:50] It is expensive per token, [05:52] but remember, it's putting out comically [05:55] less 7 mil tokens. That is a tenth as [05:58] many as 55 high and a 30th or less as [06:02] many as Sonic 46. [06:04] And 55 medium is only 22 mil, so very [06:06] token efficient models. [06:08] So that's the second factor of pricing. [06:10] Factor one is per token cost. Factor two [06:13] is number of tokens to solve a problem. [06:15] Factor three is your ability to [06:17] negotiate better deals. Some of these [06:19] labs are up charging as much as like 90% [06:23] for how much it costs to run the model [06:25] because they have lots of other costs [06:26] like the people who trained the model, [06:28] the computer they had to buy to be able [06:29] to run the model, the computer they had [06:30] to spend it to train it as well. They [06:32] have to factor that into for their [06:33] prices, but a lot of these companies [06:35] could hypothetically speaking give you a [06:38] deal for half off or more per token and [06:41] it wouldn't cost them money. It would [06:43] just be opportunity cost. So they are [06:45] pricing based on how much they think [06:47] they can get away with charging for the [06:48] thing cuz they need to make massive [06:51] profits to justify the absurd amount of [06:52] money they spend training and doing all [06:54] that. That's also why they can subsidize [06:55] so hard doing things like, you know, [06:58] giving you $4,000 of usage per month on [07:01] the $200 plans like Claude does for [07:03] example. Those plans subsidize so hard [07:07] that it's affecting cursor because if [07:09] you're subscribed to Codex or Claude [07:11] code, you are now using a competitor to [07:14] cursor. And Anthropic and OpenAI are [07:17] going to subsidize your usage in [07:19] meaningful ways because they want to get [07:21] you locked into their ecosystems. That's [07:23] just the nature of how this all works. [07:25] So if you spend $200 on Claude code, you [07:27] can get 4,000 plus dollars of usage. You [07:30] can probably get as much if not more [07:32] doing the same with Codex. But if you're [07:34] using the APIs directly, you don't get [07:37] that level of subsidization at all. You [07:39] would be lucky to negotiate 30% off. So [07:42] $4,000 to a Claude code sub costs $200. [07:45] $4,000 of Anthropic models to Cursor [07:50] probably costs $3,000 [07:52] at best. And this means a handful of [07:55] things for Cursor. It means that these [07:58] models getting more expensive on the [08:00] books for the API prices massively hurts [08:04] the amount of usage you can get as a [08:06] user of Cursor. It also means they are [08:08] competing in a subsidization war that [08:10] they don't have a player in. Because if [08:14] Anthropic or OpenAI decide they want to [08:16] crush Cursor, which seems like they kind [08:18] of have, they have the compute and the [08:20] models and the wedge to increase how [08:23] much Cursor has to pay and decrease how [08:25] much users have to pay to use something [08:27] else, that puts them in a really hard [08:29] spot. But Cursor does have one thing [08:32] that almost no one else does, including [08:35] the labs to an extent. Cursor has data. [08:38] Because Cursor's the tool that's been [08:39] used by so many developers for so long [08:42] building with AI, and many of them [08:44] didn't turn off the share my data switch [08:46] in the settings, that is data they can [08:48] use to train the models. Not just like [08:50] they read your code base or something. [08:51] So, people get way too interested in [08:53] that part. That's not that interesting. [08:54] What's much more interesting is how you [08:56] work with the agent, where you work on a [08:58] plan with it, and then you give it [09:00] feedback as it generates things. You [09:01] tell it, "Hey, you missed this thing. Go [09:02] change that, too." Those chat histories [09:05] are so valuable. And Cursor kind of gets [09:08] to double dip here because they can use [09:10] the models at admittedly way higher a [09:12] price than other people get them for to [09:15] resell to use the user building with AI [09:18] in their tool, but they also get to keep [09:20] the data if the right switches are hit, [09:22] so they can use that to train a smarter [09:24] model, not as smart as the ones that [09:26] they are distilling from, but close [09:28] enough that they can collect a lot of [09:31] the margins that they are losing to [09:34] there. Because if you choose to use [09:36] Cursor for open AI models instead of [09:38] using Codex for open AI models, you open [09:41] AI makes a lot more money as a result of [09:42] that. And this is a necessary thing for [09:45] cursor to survive because if the prices [09:47] keep going up, the subsidization [09:49] continues, the incentive to use cursor [09:51] is just a massive financial mess, which [09:54] is why when composer first dropped with [09:56] composer one and it was priced at 125 [09:59] per mil in and 10 per mil out and it was [10:02] super token hungry, I was confused. That [10:05] is too expensive for a model that wasn't [10:08] very good. And then one five, which was [10:10] meaningfully better but still not great, [10:12] was more expensive than sonnet. And [10:14] sonnet's already overpriced. $3.50 per [10:17] mil in and 17.50 per mil out. This is [10:20] the point where I kind of wrote off [10:21] composer for a bit. I just didn't see [10:23] the light. That was my mistake though. I [10:25] was betting against Jacob and you should [10:26] never bet against Jacob. If you're not [10:27] familiar, Jacob's the creator of [10:29] Supermaven as well as Tabnine. [10:30] Supermaven got acquired by Cursor and he [10:33] made the tab complete what it is. He [10:35] wanted bigger challenges, which is why [10:36] he started doing the model training [10:38] side. And then composer two hit. It was [10:40] way cheaper. It was seven times cheaper [10:43] than one five was and way smarter too. [10:46] And now we have 2.5, which is still that [10:49] same cheap price of 50 cents per mil in [10:51] and 2.50 per mil out, but the numbers [10:54] are where it shines. On cursor bench, [10:57] and this is cost per task by the way, so [10:59] left is more expensive, right is [11:01] cheaper. Composer 2.5 is scoring between [11:05] 5.5 high and X high, but costs way less [11:08] than either. [11:10] That's a really big deal. Composer two [11:12] was already pretty competitive in this [11:13] sense where it was crushing things like [11:15] 3.5 flash and Kimmy K26, which is the [11:18] model it's based on, which is really [11:19] interesting. It was getting up there [11:21] with like Opus on medium for way [11:23] cheaper. But 2.5 has gotten slightly [11:25] more token efficient, significantly [11:27] smarter, and stayed absurdly fast. Here [11:30] are the numbers of all of the models [11:32] cursor currently publishes runs for with [11:34] what is admittedly their benchmark [11:37] cursor bench. This benchmark doesn't [11:39] have too many details public. It's real [11:41] problems that they've encountered that [11:42] they collected is my understanding but [11:44] they have not published it. There's a [11:46] lot of good reasons for that like SW [11:48] bench Pro is contaminated. It just is. [11:50] If you make the benchmark open source, [11:52] the labs will get the data on how to do [11:54] it better and then that benchmark stops [11:56] being useful. It's sad that that's the [11:57] case but that is the case. Apparently [11:59] both composer 2 and 2.5 are still based [12:01] on Kimmy K25. They didn't switch to K26 [12:03] as the base which makes their numbers [12:05] even crazier. They have effectively [12:06] doubled the score of their base model [12:09] with Kimmy K25 with composer. Now at a [12:11] 63% [12:13] on their bench. GPT 5.5 got a 64. Opus [12:16] 4.7 got a little closer with 65 at 20x [12:19] the cost though. 5.5 was only roughly [12:22] 10x the cost for a point percentage [12:24] point but composer 2.5 being that good [12:27] for that cheap is absurd. Showing that [12:29] you can distill an open weight model to [12:32] be close to that level of intelligence [12:35] in coding task specifically is both [12:38] super cool to see but also kind of shows [12:41] that all of the work for making models [12:42] good at code is post training in RL [12:45] largely. Like they took a model that was [12:47] okay at code and just ran a [ __ ] load of [12:50] compute and a [ __ ] load of post training [12:52] on it to try and force it through RL to [12:54] be better at coding and they succeeded. [12:56] So let's read what they have to say [12:57] about it. 2.5 is now available in [12:58] cursor. It's a substantial improvement [13:00] in intelligence and behavior over [13:02] composer 2. It's better at sustained [13:03] work on long running tasks following [13:05] complex instructions more reliably and [13:07] it's more pleasant to collaborate with. [13:10] They improved composer by scaling [13:11] training generating more complex RL [13:12] environments and introducing new [13:14] learning methods. In addition to [13:15] training composer for 2.5 on more [13:17] difficult tasks, they've improved [13:19] behavioral aspects of the model like [13:21] communication style and effort [13:22] calibration. These dimensions are not [13:24] well captured by existing benchmarks but [13:25] we find that they matter for real world [13:27] usefulness. I agree. A lot of these [13:29] things are just hard to measure. [13:30] Composer 2.5 is built on the same [13:32] open-source checkpoint as Composer 2, [13:34] which was Moonshots Kimmy K25. I like [13:36] this chart. It's a little crazy, but [13:39] shows how this got so good. This is a [13:43] estimated measurement of how much [13:44] compute was used for each stage of [13:46] training. If you take all the compute [13:48] that went into Composer 2.5 and break it [13:50] up based on Kimmy K2, which was the [13:52] original base that was trained by [13:53] Moonshot, then compare that with Kimmy [13:55] K25 where they did another roughly the [13:57] same to refine it into a better, smarter [14:00] model. Composer did another 8x-ish [14:04] compared to that. It was 10x more total [14:06] compute in the end. And they were able [14:08] to do this because of their collab with [14:09] SpaceX AI, which is still a very [14:12] annoying, weird name collab thing. If [14:14] you're not familiar with that, Cursor [14:15] currently has a pending deal with SpaceX [14:18] where either they will be paid $10 [14:20] billion for their collab, which is [14:21] mostly getting their data, and also [14:23] Cursor gets access to their compute to [14:25] use for things, or at the end of the [14:27] year SpaceX can purchase Cursor for 60 [14:30] bill. The current rumor is that that's [14:32] going to happen when they IPO, which is [14:33] currently in process. The S-1 was just [14:35] filed. So, it's not unlikely this is [14:38] going to happen and Cursor will become [14:40] part of SpaceX, but that compute seems [14:42] to have been huge for them. For them to [14:44] have so soon after that collab started [14:46] already put out a model that's in a [14:48] third place on their benches for code is [14:50] nuts. They gave some fun details on how [14:52] they were doing the training here, too. [14:54] First piece they call it is the targeted [14:55] RL with textual feedback. Credit [14:57] assignment during RL is becoming an [14:58] increasingly difficult challenge as [14:59] rollouts can span hundreds of thousands [15:01] of tokens. When a reward is computed [15:03] over an entire rollout, it may be hard [15:05] for the model to tell which specific [15:06] decision helped or hurt the outcome. [15:08] That's especially limiting when we want [15:09] to discourage a localized behavior like [15:11] bad tool calls, confusing explanations, [15:13] or style violations. That is really [15:15] hard. If you're you're measuring with RL [15:16] that the output works, but it did three [15:19] bad things during it, how do you tell it [15:21] failed without having it fail the actual [15:23] output? The final reward can tell us [15:25] something went wrong, but it's a noisy [15:26] signal for where it went wrong. So, they [15:28] trained 25 with targeted textual [15:30] feedback. The idea is to provide [15:32] feedback directly at the point in the [15:34] trajectory where the model could have [15:35] behaved better. For a target model [15:36] message, we construct a short hint [15:38] describing the desired improvement, [15:40] insert that hint into the local context, [15:42] and use the resulting model distribution [15:44] as a teacher. We use the policy with the [15:46] original context as the student and add [15:48] an on-policy distillation KL loss that [15:50] moves the student token probabilities [15:52] towards the teachers. Interesting. So, [15:54] effectively, there is a another model [15:57] that has been steered the right way by [15:59] being given this additional context and [16:01] information, and then it tries to direct [16:04] the thing being trained to behave [16:06] similarly without having that same [16:08] context. So, you tell one model, "Hey, [16:11] this is the thing we want you to do." [16:13] And it tries to steer the other model [16:15] into going that way without actually [16:18] having that additional instruction. [16:20] Because in the end, we shouldn't have to [16:21] tell our models every step along the way [16:23] how we want them to behave. We should [16:24] tell them what we want, and then they [16:26] behave. So, this makes sense to me why [16:28] this would work well. It's really cool [16:29] to hear. It's awesome they're being so [16:30] transparent about it, too. [16:32] A good example here is when you have [16:33] tool errors. For example, if the model [16:36] has a bad tool call or tries calling a [16:38] tool that's unavailable, the model would [16:39] receive a tool not found error, but the [16:41] model would then continue to make [16:42] additional valid tool calls. This [16:44] screams Gemini. I hope my friends over [16:46] at DeepMind and Google are reading this [16:48] because you need to copy everything [16:49] they're talking about here. No model I [16:51] use regularly has as many malformed tool [16:53] calls as Gemini models do. That's [16:56] because they're RL for the end result, [16:58] almost certainly. So, if it has five bad [16:59] calls and eight good ones, it doesn't [17:01] care cuz the result came out correct. [17:03] They're trying to fix those behaviors. [17:06] The fact that it hit one error in the [17:07] process of hundreds of tool calls would [17:09] have a minimal impact on the final [17:10] reward. Yes, but how So, how do you fix [17:12] that? With text feedback, we can target [17:14] the specific mistake by inserting a hint [17:16] in the context of the problematic turn, [17:19] such as reminder, available tools are [17:21] with a list of available tools. And [17:23] remember, this is all just [17:24] probabilities, where based on the things [17:26] in the context and how that's affected [17:28] the model, it is more likely to answer [17:30] in certain ways. So, if they have a [17:32] teacher that has different context, its [17:35] probabilities are better aligned with [17:36] what we want, it will steer the student [17:39] model that is being RL then trained in [17:41] this process to behave more like the [17:44] teacher who that has this information. [17:45] So, by giving it that additional [17:46] context, it lowers the likelihood that [17:48] you have the wrong tool call [17:50] probabilistically, and it increases the [17:51] likelihood that you have a valid tool [17:53] call instead. And for this specific [17:55] turn, we update the student weights [17:58] towards the new probabilities. During [17:59] the Composer 2.5 run, we applied this [18:01] method to a variety of model behaviors, [18:03] from coding styles to model [18:05] communication. Here's an example. It [18:06] tried calling a tool that doesn't exist, [18:09] read lints. Reminder, available tools [18:12] are read, write, shell, and string [18:13] replace. And now the token probabilities [18:16] that violate the hint go down, and the [18:18] model weights are updated to avoid that [18:19] type of error. The next call they have [18:21] is synthetic data. Apparently, they're [18:23] not just using the data they collect [18:24] from us anymore. Composer's coding [18:25] abilities improved substantially to the [18:27] point where it began to get most [18:29] training problems correct. To continue [18:30] increasing intelligence, we both select [18:32] for and create harder tasks dynamically [18:34] throughout the run. Composer 2.5 is [18:36] trained with 25x more synthetic tasks [18:39] than Composer 2 was. We use a range of [18:41] approaches from creating synthetic tasks [18:42] that are grounded in real code bases. [18:44] For example, one synthetic approach is [18:46] feature deletion. For these tasks, the [18:48] agent is given a code base with a large [18:50] set of tasks and asked to delete code [18:52] and files in such a way that the code [18:54] base remains functional while specific [18:56] testable features are removed. The [18:58] synthetic task is to re-implement the [18:59] feature, and the tests are used as [19:01] verifiable rewards. That's a clever way [19:03] to generate this data. A lot of the work [19:05] for making models smarter is these types [19:07] of clever things. I talked about them a [19:09] lot in other videos, but this is a smart [19:11] one. Tell the model to delete a feature [19:14] and then tell it to re-add the feature [19:15] using the old tests. One downstream [19:17] consequence of large-scale synthetic [19:18] task creation is that it can cause [19:20] unexpected reward hacking. As the model [19:22] got more adept, Composer 25 was able to [19:24] find increasingly sophisticated [19:25] workarounds to solve the task at hand. [19:28] In one example, the model found a [19:29] leftover Python type checking cache and [19:31] reverse-engineered the format to find [19:33] the deleted function signature. In [19:35] another, it was able to find and [19:37] decompile Java bytecode to reconstruct a [19:39] third-party API. We were able to find [19:42] and diagnose these problems using [19:43] agentic monitoring tools, but they [19:44] demonstrate the increasing care [19:46] necessary for large-scale RL. [19:48] Yeah. [19:49] I even seen things like this in my own [19:51] stupid tests. Like earlier, I was trying [19:53] to demonstrate a new framework I'm [19:55] working on and I made a bunch of demo [19:57] projects for this built. And in here, [20:00] you will see that the model found a [20:03] to-do app with this already implemented [20:06] on another project that had a similar [20:08] name nearby cuz I did a restart I did a [20:10] Mario demo I did multiple demos and it [20:11] kept finding other projects nearby that [20:14] it could use to cheat. I'm checking [20:16] nearby Lakebed Capsules for updated [20:17] syntax rather than guessing the database [20:19] API. Yeah, it found other folders nearby [20:22] that had examples in it. The models are [20:24] trained to succeed, not to avoid [20:27] cheating and trying to RL without [20:30] letting the models cheat is a very [20:31] difficult thing and it's cool they're [20:32] sharing examples of the model hacking [20:34] that. Enough of this though, I want to [20:36] play. But here's also where I have to [20:38] crash out a little bit because Glass [20:41] frankly still kind of sucks. This is [20:43] their attempt at doing a Conductor Codex [20:45] app T3 code type thing and it is slow, [20:48] clunky, and obnoxious. We will use it [20:50] here cuz it is the best option they [20:51] provide. I do want to call out in T3 [20:53] code that we support Cursor through [20:56] their ECP bindings for the CLI so you [20:59] can use Cursor models here, but to make [21:01] sure we're using everything properly, I [21:03] am going to do this pass using Cursor [21:05] Glass. It's annoying. I just opened [21:07] cursor dot for this directory and it [21:09] brought me to a new thread view for a [21:12] different project. This is like the [21:14] command I use the most, like cursor [21:16] here. [21:17] Okay, the second time it opened it [21:18] right. Again, like the [21:20] I don't know how to put into words how [21:22] far the quality control has fallen. [21:24] I was really hoping glass would be the [21:25] reset we needed. It hasn't been. This [21:28] has become one of my favorite tests to [21:30] do where I take a game that I built [21:31] already that is kind of functioning but [21:34] needs a lot of work and I tell the model [21:36] to re-implement it from scratch in this [21:38] directory. I'd like to re-implement my [21:40] game fish slop from scratch. I want [21:42] better assets and more reliable core [21:43] engine and a path to making it a real [21:44] quality web game. The new directory for [21:46] it is here, use that as a reference and [21:48] build the game in its entirety in this [21:49] directory. Do not stop until you're [21:51] done. Let's see what it does. [21:53] One of the coolest things about it is [21:54] that it's really fast. It's already [21:56] spinning up parallel agents to do [21:57] different things at the same time, which [21:59] is really cool. One of the crazy things [22:00] they did, I haven't seen a company try [22:02] something this crazy before, is that [22:04] they did an internal test last week with [22:06] the model where all cursor chats for [22:08] employees were redirected to composer [22:10] 2.5 for a few days. This employee didn't [22:12] even notice, which is a testament to how [22:14] good the model's gotten. That is super [22:16] cool to hear. And you can see it flying [22:18] through this stuff right now. It is it [22:21] is churning. It did just do like five [22:23] edits in a row here to like the same [22:25] thing. -1 +1 -1 +1 -1 +1 -1 +7 +1 -7. [22:30] Interesting. [22:31] It's actually checking the resulting [22:33] code, which when I tried the same thing [22:34] with 3.5 flash yesterday, it did not do. [22:37] On that note, 3.5 flash, which also came [22:41] out the same time as composer 2.5, got a [22:43] way lower score than 2.5, lower than [22:46] 2.0, and cost $1.94. I just I don't see [22:50] why anyone would use 3.5 flash for [22:52] almost anything. It's just not a good [22:53] option. It's done? Oof. [22:56] Page failed to load. Here's the result. [23:00] So, it is one of the two models that [23:01] failed to make the actual page render, [23:04] which is sad but not super surprising. [23:07] It is a Chinese open weight model that [23:09] was RL'd to all hell. Yeah, let's see [23:11] how this works now. [23:17] I don't have enough shells to unlock [23:18] those. [23:20] Well, unlike the one that Gemini made, [23:23] it actually functions and has the core [23:24] mechanics behaving. [23:26] That is genuinely insane. I I don't love [23:30] the asset it generated here. [23:33] I'll give it an annoying task, actually, [23:35] to fix that. Pull over all the assets [23:38] from the original game. Let's see if [23:41] it's able to swap out the assets for the [23:43] ones from the original game and how [23:45] quickly it's able to do that. PNG assets [23:47] exist in Fish Game Public, but the glob [23:48] tool failed to detect them. Interesting. [23:51] This is a big move, changing the entire [23:53] render pipeline for your game live. You [23:56] can see the game flashing in the [23:57] background as it does all the changes. [23:59] Apparently, it's done. That took under [24:01] 30 seconds. And yeah, I switched over to [24:04] my original assets. [24:07] It made a lot of things too small. Let's [24:09] tell it to fix that. Scale is really off [24:12] and UI is hard to read, low contrast. On [24:17] the topic of UI, I do want to talk about [24:19] the UI it generates cuz it's actually [24:21] kind of impressive. Friend of the [24:22] channel Dara has made an awesome tool [24:25] that lets you compare what models design [24:28] and how they handle design different [24:29] ways. The site itself was designed by [24:31] Composer 2, funny enough. He runs [24:33] different models with the same prompt to [24:35] generate different web pages to see how [24:37] good they are at design. And here are [24:39] the designs that we got out of Composer [24:41] 2.5. This one's actually kind of nice [24:43] and chill. I I like the way it used the [24:46] font. I like the little thing it made [24:47] here. Tasteful grain. Yeah, this is [24:49] fine. This is clearly using the [24:51] front-end design skill. Yeah, cuz these [24:53] are all the the designs you get out of [24:55] the front-end design skill. It screwed [24:56] up the spacing on these buttons a bit, [24:58] like that should be uh aligned to the [25:00] bottom, I think. [25:01] And this doesn't look great either, but [25:03] like it it honored the goal of this [25:05] design, okay. [25:07] This one I just I don't like. I don't [25:08] like this style of like preppy [25:12] [ __ ] [25:13] I hate the retro ones. [25:15] And this is fine. So, here's how it goes [25:17] without the front end design skill from [25:19] Infropik. This looks fine. [25:22] This looks fine, but way too similar. [25:24] This is different. [25:26] It's different in a way that's kind of [25:27] cute. I don't love it, but it it is [25:29] cute. This is awful. [25:31] And this is [25:33] like an old Tailwind slash like [25:35] Bootstrappy site. I've seen a lot of [25:36] sites that look like this. This feels [25:37] like an old era design that you would [25:40] get in the the purple pink gradient [25:43] classic. Not bad. Like like getting this [25:45] type of design out of a model this cheap [25:47] is really cool to see. Interesting bug [25:49] here. We'll resume when background work [25:51] finishes. What is the background work [25:53] that it is finishing? Cool, they called [25:55] me. [25:55] Let's see how this looks now. [25:58] Oh. [25:59] Yeah, it screwed up the scaling a bunch [26:01] for this, sadly. [26:03] It got like the core mechanics right. [26:05] These buttons on the bottom don't work [26:07] for clicking things, though. Like I have [26:08] to press the hot keys. [26:10] It's [26:11] it's far from perfect, but I could see [26:13] how somebody who doesn't have a lot of [26:15] money, does have a cursor sub or like a [26:17] student discount or something, could [26:19] with enough massaging and back and forth [26:21] make something actually decent out of [26:23] this. Like I see the path there. It's [26:25] not a super happy path, but there is [26:27] one. Okay, it actually does have a lot [26:28] of things wrong with the feeding [26:29] mechanic in the game. It's sadly not the [26:31] worst solution I've seen for this, but [26:33] it is not great. This is an admittedly [26:36] very complex task, but yeah, I was [26:39] hoping for a little more there. For my [26:40] like day-to-day random TypeScript [26:42] end-to-end [ __ ] I found it to be [26:44] relatively pleasant, but there is a big [26:46] problem. [26:47] The problem isn't that I don't like [26:50] glass, even though I don't. The problem [26:52] isn't that it's too expensive or too [26:54] slow or too dumb or any of those things. [26:57] The problem's actually over here on [26:59] Cursor Bench. And the problem isn't [27:01] Cursor Bench itself. It's not that it's [27:02] a dishonest test or anything like that. [27:04] It's that we don't have anything [ __ ] [27:06] else to measure against. Because unlike [27:08] every other model that we talk about on [27:10] this channel, there is no API we can hit [27:12] to test this model ourselves. The only [27:15] way to try out Composer, regardless of [27:17] how much money you're willing to spend, [27:19] is to use it inside of Cursor. Unless [27:22] you spend $60 billion to buy them, then [27:23] you can probably use it for whatever. [27:25] Point I'm trying to make is that we [27:27] don't have a real measurement of this [27:30] outside of what Cursor is putting out. [27:33] The Rob did come out and say that they [27:35] are reporting terminal bench and SWE [27:36] bench multilingual, which sadly is a [27:39] polluted test. Enough info on it has [27:42] leaked that models have it in the [27:43] training data and it's kind of [27:45] compromised now. It is what it is. [27:47] Allegedly they are working with [27:48] artificial analysis and others for more [27:49] external evals, which is very good to [27:51] hear. But I understand on one hand why [27:53] they're not going hard on this because [27:56] they made this model just for coding and [27:57] nothing else. It'll probably perform [27:59] really poorly on stuff like SkateBench, [28:01] for example, because it isn't built for [28:04] that use case. It's built just for [28:07] coding. And a lot of these benches, like [28:09] if you were to throw it on the [28:10] artificial analysis intelligence index, [28:12] the core bench they have here, it would [28:14] perform very poorly, possibly as bad as [28:17] Kim iK25 or worse, because a lot of the [28:21] benches things that aren't code. So I [28:23] get why they're not prioritizing this, [28:24] but it does suck to have a model that [28:26] seems really useful for a bunch of [28:28] different [ __ ] that I can only use [28:30] through Cursor's app or through the ACP [28:33] implementation on their CLI that we had [28:35] to like push really hard for them to [28:36] fix. For a while you couldn't even use [28:39] Composer models through the ACP bindings [28:42] because they had a limited hard-coded [28:43] set of models that didn't include the [28:45] new Composer models. Obnoxious. They've [28:47] since fixed that after a lot of back and [28:49] forth with the team. They also just put [28:50] out the Cursor SDK, which allows you to [28:52] build agents using Cursor's harness and [28:55] infrastructure and everything, which [28:57] hypothetically means you can write code [28:59] like this, where you create an agent [29:01] that uses Composer 2 and has access to a [29:03] specific directory, and then you can [29:05] programmatically run things against it. [29:07] This is cool. Especially cuz you can [29:09] point it at cloud repos as well. There's [29:10] a lot of promise here. And as I've [29:12] mentioned many times, I do genuinely [29:14] really, really like the Cursor Cloud [29:16] stuff. cursor.com/agents has [29:18] consistently been my favorite way to use [29:19] Cursor as of late because it has a full [29:22] Linux box in the cloud that it can use [29:24] to spin up whatever your project is, as [29:26] long as it's not like an iOS app, and [29:28] actually test against it with computer [29:30] use. Super cool when combined with fast [29:33] models like Composer 2 and 2.5 and the [29:35] ability to trigger these from my phone [29:37] or from Slack. I like where they're [29:39] going with all of this. I just hate when [29:41] I see a new model with numbers that are [29:43] this good, and I can't it. This is cool, [29:47] especially cuz you can point it at cloud [29:48] repos as well. There's a lot of promise [29:49] here. [29:50] They're the only lab, in quotes, worth [29:53] looking at that doesn't expose their [29:55] stuff over API at all. This is very [29:57] different for companies like Anthropic, [29:59] who subsidize usage through their tool, [30:01] but charge you full price to hit it [30:03] directly. The only thing going on right [30:04] now that's close is Gemini 3.5 Flash, [30:07] where you can pay API prices, but if you [30:10] want the full speed version, that the [30:12] one that runs at 1,000 TPS instead of [30:13] 200, you have to use antigravity IDE or [30:17] CLI to get the full speeds. There are [30:19] very few instances nowadays, thankfully, [30:21] of labs restricting access to their [30:24] models on a fundamental level. Usually, [30:26] they just subsidize. But now we have [30:28] both Google restricting us from using [30:30] the fast version of their model over the [30:32] API. We can only use the slow one. [30:33] Previously, we had Mistral do the same [30:35] thing, where they had a version of the [30:36] model that could do 1,000 TPS that was [30:38] working through their partnership with [30:40] Cerebras at the time. But if you hit it [30:42] over the official Mistral API, you'd be [30:44] around 50 TPS. And now we have Composer, [30:47] which is literally only headdable via [30:49] Cursor surfaces. You cannot do it over [30:51] API at all. Apparently, people have it [30:54] working in tools like Pi by using the [30:56] Agent SDK that Cursor put out. That's [30:58] cool to see, and I don't think they're [31:00] going to be as egregious with limiting [31:02] how people use the SDK as companies like [31:04] you know, the one big A has been about [31:07] that. But it does suck that I feel like [31:09] I can't meaningfully benchmark this or [31:11] integrate it in other tools. Like if I [31:13] just wanted to swap out the API endpoint [31:16] in something like Snitch Bench or inside [31:18] of something like Pi to use what they [31:21] have here, I can't. I have to rebuild [31:24] the application to use their SDK. And [31:26] for what it is worth, I have not heard [31:27] the best things about the SDK. I have [31:29] admittedly not had a chance to try it [31:31] myself, but I have heard that it is [31:33] missing things and is not the most [31:34] reliable stuff yet. That said, if enough [31:36] of you guys build cool things with the [31:38] Cursor SDK, I will have the leverage I [31:40] need to force the team to push it [31:42] further. And that would be really cool. [31:44] So I do still recommend you go build [31:46] cool things with this. Just make sure [31:47] you do it in a way where if you do have [31:49] to swap out, it's not that hard to do. [31:51] So what are my final thoughts on all of [31:53] this? Well, if you're the type of [31:55] developer that is still staring at the [31:57] agent as it runs and are doing very [32:00] collaborative work with it, where you [32:01] ask it to research a thing, it [32:03] researches it, it gives you a result, [32:05] you ask it to then go make a change, and [32:06] then it does that, you look at the [32:07] result, where you're having a back and [32:09] forth that you're engaged with the [32:10] model, having a model this fast and [32:13] smart is really, really powerful. That's [32:15] why I was excited for 3.5 Flash and [32:17] ultimately was disappointed because it [32:18] wasn't good enough at coding to have a [32:21] good experience doing that with. [32:23] Composer 2.5 is a much better balance [32:25] here. It might not be as capable of [32:27] unblocking itself in solving complex [32:29] tasks as things like the latest GPT [32:31] models or the latest Opus models, but it [32:34] is really fast and close enough that you [32:37] can have this back and forth with it in [32:39] a way that reminds me of the old days [32:41] doing a gentle coding with the [32:42] difference being it's way smarter and [32:44] way more capable. So, the people who are [32:46] working at big companies on giant repos [32:48] that just want to like sit in the code [32:51] base with a real IDE and talk back and [32:53] forth with the model about the changes [32:54] they want to make and not have to wait [32:56] 20 minutes for it to go explore [ __ ] [32:58] This is an incredible model. But, if [32:59] you're the type of person that's [33:01] spinning up 20 agents in parallel on the [33:02] same thing all doing different [ __ ] and [33:05] by the time you spin up agent 17, agents [33:07] 1 through 6 are done, this will not be [33:09] as useful to you. That said, I find [33:11] myself steering away from the super [33:13] parallelization stuff lately cuz it's [33:15] just too much context in your head to [33:17] keep and if you're doing anything else, [33:19] it's just impossible to maintain. I [33:21] can't tell you how many cool projects [33:22] are probably sitting on random work [33:24] trees on this computer that I've never [33:25] had a chance to even look at yet. So, if [33:27] you're doing this type of engaged back [33:29] and forth with the model, a fast model [33:31] is really powerful. I'm saying this [33:33] confidently cuz I've experienced it [33:34] myself recently cuz I've been using [33:36] GPT-4-5 on the fast mode for like 2 [33:40] weeks now because I got way too much [33:42] free usage from OpenAI cuz I went to the [33:44] 4-5 event and everybody who applied to [33:46] go got I think a 10x bump. So, despite [33:48] building a literal cloud from scratch [33:51] over the last few days using the these [33:53] models, yeah, I'm building a cloud. [33:54] We'll talk about that another time, [33:55] don't worry. I'm not struggling to even [33:57] take down 1% of my 5-hour limit much [34:00] less my weekly usage limits. I'm trying [34:02] so hard to burn through this and I [34:03] literally can't because they gave me too [34:05] high of a limit as a result of that [34:07] event. Insane. It is what it is, but as [34:10] such, I've gotten addicted to fast [34:11] because it's 2x more expensive and I'm [34:13] still barely able to burn the [ __ ] [34:15] tokens. I think Composer 2.5 fast is a [34:18] much more realistic option for companies [34:21] where they can't have devs doing the [34:22] subsidized token usage that also don't [34:24] have that free 10x bump temporarily. For [34:26] companies with big code bases that have [34:28] developers actually in their editors [34:30] reading the files and looking at the [34:31] changes, like hand-in-hand with the [34:33] model. It's a good workflow, and having [34:36] a model this fast, this cheap, this [34:38] reliable for that type of real-world [34:40] work fits the archetype of Cursor's [34:43] customer very well. Because Cursor is [34:45] killing it in enterprise. When I talk to [34:47] my friends at big companies that aren't [34:49] Amazon, where they built their own [34:50] shitty IDE, almost everyone else I talk [34:53] to is using Cursor. So much of the [34:55] Fortune 500 is on Cursor, it's insane. [34:57] This is a really good win for them that [35:00] will keep them from taking a potential [35:02] deal with OpenAI or with Anthropic that [35:05] I'm sure they're in negotiations with [35:06] all of those companies. This is a [35:08] massive play. And unlike those other [35:10] labs, if you go to Anthropic, for [35:13] example, with a deal, you can only use [35:15] the latest Opus and Sonnet models. If [35:18] OpenAI does something cool, you're [35:19] [ __ ] If Cursor does something cool, [35:21] you're [ __ ] Same deal if you go with [35:22] OpenAI. But if you go with Cursor, you [35:25] at the very least have their awesome [35:27] models with Composer. But if one of the [35:29] other labs leapfrogs at any point, you [35:31] still get access to that, too. So, [35:33] Cursor has continued to be a really [35:34] compelling bet for enterprises, and I [35:36] think these developments help that a [35:39] ton. By having a really good default [35:41] that they can subsidize themselves and [35:43] make really compelling value [35:45] propositions for their users with, they [35:47] are able to stay a player in this race. [35:50] And as silly as it sounds, I have more [35:52] faith in Cursor to sustain the AGI [35:56] bubble or whatever happens than I do a [35:58] company like Microsoft. Because at the [36:00] very least, Cursor's moved far beyond [36:02] the VS Code fork we made fun of them for [36:05] into a real lab focused on real [36:07] developer workflows at a level that [36:09] Microsoft cannot even comprehend. If I [36:12] was to guess where we'll be in 2 years, [36:15] I legitimately think it's more likely [36:16] that we're using xAI Cursor Composer 7 [36:20] than anything by Gemini for our [36:22] day-to-day dev work. Even if it thinks [36:24] things are going on in the background [36:25] when they're not, bugs will happen. I [36:27] have one last thing I want to end on [36:28] here. Cursor posted in the thread where [36:30] they announced the new model, "Together [36:32] with SpaceX AI, we are training a [36:34] significantly larger model from scratch [36:36] using 10x more total compute. That means [36:39] it's going to be 100x more than Kimmy [36:41] was trained on." With Colossus 2's [36:43] million H100 equivalents and our [36:45] combined data and training techniques, [36:48] we expect this to be a major leap in [36:50] model capability. It is legitimately [36:52] possible that Cursor doesn't just catch [36:55] up to state of the art for code. They [36:57] might leapfrog. There's a real chance [37:00] that in just a few months, Cursor will [37:03] have the best model for code. And when [37:05] you look at the trajectory, when you [37:07] look at how fast they have scaled up [37:09] where they are, and combine that with [37:10] the insane amount of compute they have [37:12] and data and all of the new training [37:14] methods they've been figuring out, I [37:16] think there's a real chance they do it. [37:18] At least I hope so so my investment [37:19] becomes something, but we'll see where [37:20] that all goes. I got nothing else on [37:22] this one. If you're still on Cursor, you [37:23] should definitely try it. And if you're [37:24] not on Cursor, I'm curious what you're [37:26] using instead. Let me know in the [37:27] comments and until next time, [37:29] peace nerds. Last week there was a really coolrelease I'm afraid people are sleepingon. It came from a company with a lot ofcompute, and it was focused on being asmall model that is very effective atsolving code tasks. And no, it wasn'tGemini 3.5 Flash, even though itprobably should have been. It wasComposer 2.5 by Codium. This is a crazydrop. The rate at which Codium hascaught up to the state of the art is prettyabsurd, but there are catches here thatwe need to talk about as well. If you'renot familiar with the Composer line of models from Codium, this has been theirattempt to distill good open weightmodels into a great coding model forusage within Codium. That said, it'sonly really for use within Codium. So,it's hard for us to bench it externally.That means we kind of got to vibe ourway through it, but it also means Codiumdoes to an extent as well. They havepublished some awesome numbers includingtheir own benchmark, which sadly is notParticularly public, the Codium bench. But the results seem to be incredible,and I can tell you from my own use ofit, this isn't just hype. While I don'tthink this is the greatest model everand you should use it for everything, Ido think it's worth looking at andunderstanding both because it's a goodmodel release from a company thatunderstands what we need as devs, butalso because it shows that the strengththat the major labs have might bestarting to crack a little bit underpressure. While I am an early investorin Codium, the acquisition stuffhappening with SpaceX kind of destroysany incentive I might have had to shillit super hard. So, I'm not sharing thiscuz I'm biased. I'm sharing this cuzit's actually really interesting to me.And don't worry, I will [ \_\_ ] all over alot of other Codium things throughoutthis, too. All of that said, want tomake sure it's very clear Codium is notpaying me at all for this coverage. Only people paying me today are thesponsor. As powerful as continuousintegration is, it's full ofcompromises. I like to call this the CIcompromise triangle. The differentthings you have to consider when you'reconfiguring your continuous integration. You start with convenience, which is howeasy it is to implement this. You alsohave performance. How long does it taketo run? It would be nice if the CI runners werejust a little bit faster. And of course,you have cost. Obviously, there arelots of different solutions for CI that havedifferent compromises across thistriangle because it would be impossiblefor anything to just be better at all ofthese, right? Like there's no waysomething could be more convenient,faster, and cheaper, right? Oh, is it a Blacksmith ad? Yeah.Blacksmith is 2x faster and 60% cheaperfor most people when they move fromGitHub Actions. I thought this was toogood to be true, as did many of theirCustomers like Mercury, Superbase,Dscript, Exa, and more. But then you useit, and you realize you can't livewithout it. If it was just cheaper, itwould be worth it. If it was justfaster, it would be worth it. And if itwas just more reliable, it would stillbe worth it. But then you discover theconsole and how much better it is thananything GitHub has shipped in years. You get actual metrics about theworkflows you're running, what ones havehigh error rates, what the errors mightbe, as well as logs that give you reallyuseful details on what is going onacross all of your CI. Every time Iforget to set up Blacksmith on a newproject, I regret it immensely, and thenI move over and watch my build times gofrom 3+ minutes to under one. Your devsand your agents have our fast CI. Get ittoday at solidity.link/blacksmith. So, let's talk about this new,intelligent, fast, cheap model thatCursor put out. First, I want to talkAbout price. A lot of people don'tunderstand how models are priced. I'mgoing to do my best to explain it simplyhere. You have the input and the outputcosts, and these are in million tokengroups. A token is just a chunk of text,usually three to five characters, kindof like a word, that is how modelsinterpret text and then generate text aswell. Input tokens tend to be a lotcheaper than output tokens because theinputs are the things you're passing themodel, and it's just using that to set upwhat its guesses for next tokenprediction are going to be. The outputsare the actual work it's doing, so thosetend to be much more. The base price fora model like Claude for Sonnet hashistorically been $3 per million tokensin and $15 per million tokens out. Onthe 1 million token context versions,they would sometimes be more expensive. Anthropic has since stopped doing that,OpenAI has not, and the Opus models tendto be much more expensive. Like 4.5 Opus. Is $5 per mil and 25 per mil out. GPT 5to 5.2 were actually very generouslypriced, in my opinion, at $1.25 per mil inand 10 per mil out. 5.2 was a bit of ajump to $1.75 in and 14 out. 5.4 wasanother jump to 2.50 in and 15 out, andthen 5.5 was a massive jump to $5 in and30 out, doubling what it was in 5.4. Itseems like the prices are going upacross the industry, but they aren’tnecessarily. There are two additionallayers to the pricing that are importantto consider. The first is the amount oftokens that are actually beinggenerated. Warning for artificial analysis. Artificial analysismeasures roughly how intelligent modelsare on their own benchmarks, internallythat they publish to the world, but theyalso measure other pieces. Inparticular, how fast the model solves theproblem, how many tokens they generate,and how much it costs to run. Outputtokens are one of my favorite things tolook at here, and you’ll notice someInteresting stuff. The most outputtokens didn't come from the smartestmodels. It came from DeepSeek V4 Flash,GPT 5.4 Mini, and Claude Sonnet 4.6. Smaller models often generate way moretokens than they need to solveproblems. It's also worth noting thatthe lab trying the hardest to reducethis overuse of tokens very much appearsto be OpenAI, where models like 5.5 onlydid 75 million tokens for the same benchthat Sonnet did 200 million for. Almost athird the number of tokens and a higherscore as well. And when you add inmodels like 5.5 low and medium, whichstill score really well in the benches,like 5.5 medium scores as well as Opus4.7 roughly, and the low version isworse, but it's still much higher than5.4 Mini and Flash and roughly aroundwhere something like Sonic 4.6 is. It is expensive per token,but remember, it's putting out comicallyless 7 million tokens. That is a tenth asmany as 55 high and a thirtieth or less asMany as Sonic 46. And 55 medium is only 22 mil, so verytoken efficient models.So that's the second factor of pricing.Factor one is per token cost. Factor twois number of tokens to solve a problem.Factor three is your ability tonegotiate better deals. Some of theselabs are up charging as much as like 90%for how much it costs to run the model,because they have lots of other costs,like the people who trained the model,the computer they had to buy to be ableto run the model, the computer they hadto spend it to train it as well. Theyhave to factor that into for theirprices, but a lot of these companiescould hypothetically speaking give you adeal for half off or more per token, andit wouldn’t cost them money. It wouldjust be opportunity cost. So, they are pricing based on how much they thinkthey can get away with charging for thething because they need to make massiveprofits to justify the absurd amount ofMoney they spend training and doing allthat. That's also why they can subsidizeso hard doing things like, you know,giving you $4,000 of usage per month onthe $200 plans like Claude does forexample. Those plans subsidize so hardthat it's affecting cursor because ifyou're subscribed to Codex or Claudecode, you are now using a competitor tocursor. And Anthropic and OpenAI aregoing to subsidize your usage inmeaningful ways because they want to getyou locked into their ecosystems. That'sjust the nature of how this all works. So, if you spend $200 on Claude code, youcan get 4,000 plus dollars of usage. Youcan probably get as much, if not more,doing the same with Codex. But if you'reusing the APIs directly, you don't getthat level of subsidization at all. Youwould be lucky to negotiate 30% off. So,$4,000 to a Claude code sub costs $200. $4,000 of Anthropic models to Cursorprobably costs $3,000at best. And this means a handful ofThings for Cursor. It means that thesemodels are getting more expensive on thebooks for the API. Prices massively hurtthe amount of usage you can get as auser of Cursor. It also means they arecompeting in a subsidization war thatthey don't have a player in. Because ifAnthropic or OpenAI decide they want tocrush Cursor, which seems like they kindof have, they have the compute and themodels and the wedge to increase howmuch Cursor has to pay and decrease howmuch users have to pay to use something. Else, that puts them in a really hardspot. But Cursor does have one thingthat almost no one else does, includingthe labs to an extent. Cursor has data.Because Cursor's the tool that's beenused by so many developers for so long.Many of them didn't turn off the share my data switchin the settings. That is data they canuse to train the models. Not just likethey read your code base or something.So, people get way too interested in. That part. That's not that interesting.What's much more interesting is how youwork with the agent, where you work on aplan with it, and then you give itfeedback as it generates things. Youtell it, "Hey, you missed this thing. Gochange that, too." Those chat historiesare so valuable. And Cursor kind of getsto double dip here because they can usethe models at admittedly way higher aprice than other people get them for toresell to use the user building with AIin their tool, but they also get to keepthe data if the right switches are hit,so they can use that to train a smartermodel, not as smart as the ones thatthey are distilling from, but closeenough that they can collect a lot ofthe margins that they are losing tothere. Because if you choose to useCursor for open AI models instead ofusing Codex for open AI models, you openAI makes a lot more money as a result of that. And this is a necessary thing forKeep going up, the subsidizationcontinues, the incentive to use cursoris just a massive financial mess, whichis why when Composer first dropped withComposer One, and it was priced at 125per mil in and 10 per mil out, and it wassuper token hungry, I was confused. Thatis too expensive for a model that wasn'tvery good. And then, Composer 1.5, which wasmeaningfully better but still not great,was more expensive than Sonnet. AndSonnet's already overpriced, $3.50 permil in and 17.50 per mil out. This isthe point where I kind of wrote offComposer for a bit. I just didn't seethe light. That was my mistake, though. Iwas betting against Jacob, and you shouldnever bet against Jacob. If you're notfamiliar, Jacob's the creator ofSupermaven, as well as Tabnine. Supermaven was acquired by Cursor, and hemade Tabnine what it is. Hewanted bigger challenges, which is whyhe started doing the model trainingside. And then Composer 2 hit. It wasWay cheaper. It was seven times cheaperthan one five was and way smarter too. And now we have 2.5, which is still thatsame cheap price of 50 cents per mil inand 2.50 per mil out, but the numbersare where it shines. On cursor bench,and this is cost per task, by the way, soleft is more expensive, right ischeaper. Composer 2.5 is scoring between5.5 high and X high, but costs way lessthan either. That's a really big deal. Composer 2was already pretty competitive in thissense, where it was crushing things like3.5 Flash and Kimmy K26, which is themodel it’s based on, which is reallyinteresting. It was getting up therewith like Opus on Medium for waycheaper. But 2.5 has gotten slightlymore token efficient, significantlysmarter, and stayed absurdly fast. Hereare the numbers of all of the modelscursor currently publishes runs for, withwhat is admittedly their benchmark,Cursor Bench. This benchmark doesn'tHave too many details public. It's real problems that they've encountered thatthey collected is my understanding butthey have not published it. There's alot of good reasons for that, like SWbench Pro is contaminated. It just is. If you make the benchmark open source,the labs will get the data on how to doit better, and then that benchmark stopsbeing useful. It's sad that that's thecase, but that is the case. Apparently,both composer 2 and 2.5 are still basedon Kimmy K25. They didn't switch to K26as the base, which makes their numberseven crazier. They have effectivelydoubled the score of their base modelwith Kimmy K25, with composer. Now at a63%on their bench, GPT 5.5 got a 64. Opus4.7 got a little closer with 65 at 20xthe cost, though. 5.5 was only roughly 10x the cost for a point percentagepoint, but composer 2.5 being that goodfor that cheap is absurd. Showing thatyou can distill an open weight model toBe close to that level of intelligencein coding task specifically is bothsuper cool to see but also kind of showsthat all of the work for making modelsgood at code is post training in RLlargely. Like they took a model that wasokay at code and just ran a [ \_\_ ] load ofcompute and a [ \_\_ ] load of post trainingon it to try and force it through RL tobe better at coding and they succeeded. So, let's read what they have to sayabout it. 2.5 is now available inCursor. It's a substantial improvementin intelligence and behavior overComposer 2. It’s better at sustainedwork on long-running tasks followingcomplex instructions more reliably, andit’s more pleasant to collaborate with. They improved Composer by scalingtraining, generating more complex RLenvironments, and introducing newlearning methods. In addition totraining Composer for 2.5 on moredifficult tasks, they've improvedbehavioral aspects of the model likeCommunication style and effortcalibration. These dimensions are notwell captured by existing benchmarks, butwe find that they matter for real-worldusefulness. I agree. A lot of thesethings are just hard to measure. Composer 2.5 is built on the sameopen-source checkpoint as Composer 2,which was Moonshots Kimmy K25. I likethis chart. It's a little crazy, butshows how this got so good. This is aestimated measurement of how muchcompute was used for each stage oftraining. If you take all the computethat went into Composer 2.5 and break itup based on Kimmy K2, which was theoriginal base that was trained byMoonshot, then compare that with KimmyK25, where they did another roughly thesame to refine it into a better, smartermodel. Composer did another 8x-ishcomparison to that. It was 10x more totalcompute in the end. And they were ableto do this because of their collaboration withSpaceX AI, which is still a veryAnnoying, weird name collab thing. Ifyou're not familiar with that, Cursor currently has a pending deal with SpaceX,where either they will be paid $10billion for their collab, which ismostly getting their data, and alsoCursor gets access to their compute touse for things, or at the end of theyear SpaceX can purchase Cursor for 60billion. The current rumor is that that'sgoing to happen when they IPO, which iscurrently in process. The S-1 was justfiled. So, it's not unlikely this isgoing to happen and Cursor will becomepart of SpaceX, but that compute seemsto have been huge for them. For them tohave so soon after that collab startedalready put out a model that's in athird place on their benches for code isnuts. They gave some fun details on howthey were doing the training here, too. First piece they call it is the targetedRL with textual feedback. Creditassignment during RL is becoming anincreasingly difficult challenge asRollouts can span hundreds of thousandsof tokens. When a reward is computedover an entire rollout, it may be hardfor the model to tell which specificdecision helped or hurt the outcome. That's especially limiting when we wantto discourage a localized behavior likebad tool calls, confusing explanations,or style violations. That is reallyhard. If you're measuring with RLthat the output works, but it did threebad things during it, how do you tell itit failed without having it fail the actualoutput? The final reward can tell ussomething went wrong, but it's a noisysignal for where it went wrong. So, theytrained 25 with targeted textualfeedback. The idea is to providefeedback directly at the point in thetrajectory where the model could havebehaved better. For a target modelmessage, we construct a short hintdescribing the desired improvement,insert that hint into the local context,and use the resulting model distribution. As a teacher, we use the policy with theoriginal context as the student and addan on-policy distillation KL loss thatmoves the student token probabilitiestowards the teachers. Interesting. So,effectively, there is another modelthat has been steered the right way bybeing given this additional context andinformation, and then it tries to directthe thing being trained to behavesimilarly without having that samecontext. So, you tell one model, "Hey,this is the thing we want you to do."And it tries to steer the other modelinto going that way without actuallyhaving that additional instruction. Because, in the end, we shouldn't have totell our models every step along the wayhow we want them to behave. We shouldtell them what we want, and then theybehave. So, this makes sense to me whythis would work well. It's really coolto hear. It's awesome they're being sotransparent about it, too.A good example here is when you haveTool errors. For example, if the modelhas a bad tool call or tries calling atool that's unavailable, the model wouldreceive a tool not found error, but themodel would then continue to makeadditional valid tool calls. Thisscreams Gemini. I hope my friends overat DeepMind and Google are reading thisbecause you need to copy everythingthey're talking about here. No model Iuse regularly has as many malformed toolcalls as Gemini models do. That'sbecause they're RL for the end result,almost certainly. So, if it has five badcalls and eight good ones, it doesn'tcare because the result came out correct. They're trying to fix those behaviors.The fact that it hit one error in theprocess of hundreds of tool calls wouldhave a minimal impact on the finalreward. Yes, but how? So, how do you fixthat? With text feedback, we can targetthe specific mistake by inserting a hintin the context of the problematic turn,such as reminder, available tools are. With a list of available tools. Andremember, this is all justprobabilities, where based on the thingsin the context and how that's affectedthe model, it is more likely to answerin certain ways. So, if they have ateacher that has different context, itsprobabilities are better aligned withwhat we want, it will steer the studentmodel that is being RL then trained inthis process to behave more like theteacher who that has this information. So, by giving it that additionalcontext, it lowers the likelihood thatyou have the wrong tool callprobabilistically, and it increases thelikelihood that you have a valid toolcall instead. And for this specificturn, we update the student weightstowards the new probabilities. Duringthe Composer 2.5 run, we applied thismethod to a variety of model behaviors,from coding styles to modelcommunication. Here's an example. Ittried calling a tool that doesn’t exist,Read LINTs. Reminder, available toolsare read, write, shell, and stringreplace. And now the token probabilitiesthat violate the hint go down, and the model weights are updated to avoid thattype of error. The next call they haveis synthetic data. Apparently, they'renot just using the data they collectfrom us anymore. Composer's codingabilities improved substantially to thepoint where it began to get mosttraining problems correct. To continueincreasing intelligence, we both selectfor and create harder tasks dynamicallythroughout the run. Composer 2.5 istrained with 25x more synthetic tasksthan Composer 2 was. We use a range ofapproaches from creating synthetic tasksthat are grounded in real code bases. For example, one synthetic approach isfeature deletion. For these tasks, theagent is given a code base with a largeset of tasks and asked to delete codeand files in such a way that the codebase remains functional while specificTestable features are removed. Thesynthetic task is to re-implement thefeature, and the tests are used asverifiable rewards. That's a clever wayto generate this data. A lot of the workfor making models smarter is these typesof clever things. I talked about them alot in other videos, but this is a smartone. Tell the model to delete a featureand then tell it to re-add the featureusing the old tests. One downstreamconsequence of large-scale synthetictask creation is that it can causeunexpected reward hacking. As the modelgot more adept, Composer 25 was able tofind increasingly sophisticatedworkarounds to solve the task at hand. In one example, the model found aleftover Python type checking cache andreverse-engineered the format to findthe deleted function signature. Inanother, it was able to find anddecompile Java bytecode to reconstruct athird-party API. We were able to findand diagnose these problems using. agentic monitoring tools, but they demonstrate the increasing carenecessary for large-scale RL.Yeah.I even seen things like this in my ownstupid tests. Like earlier, I was tryingto demonstrate a new framework I'mworking on, and I made a bunch of demoprojects for this built. And in here,you will see that the model found ato-do app with this already implementedon another project that had a similarname nearby because I did a restart. I did aMario demo. I did multiple demos, and itkept finding other projects nearby thatit could use to cheat. I'm checkingnearby Lakebed Capsules for updatedsyntax rather than guessing the databaseAPI. Yeah, it found other folders nearbythat had examples in it. The models aretrained to succeed, not to avoidcheating, and trying to RL withoutletting the models cheat is a verydifficult thing, and it's cool they'resharing examples of the model hacking. That. Enough of this though, I want toplay. But here's also where I have tocrash out a little bit because Glassfrankly still kind of sucks. This istheir attempt at doing a Conductor Codexapp T3 code type thing, and it is slow,clunky, and obnoxious. We will use ithere because it is the best option theyprovide. I do want to call out in T3code that we support Cursor throughtheir ECP bindings for the CLI so youcan use Cursor models here, but to makesure we're using everything properly, Iam going to do this pass using CursorGlass. It's annoying. I just openedcursor dot for this directory and itbrought me to a new thread view for adifferent project. This is like thecommand I use the most, like cursorhere. Okay, the second time it opened itright. Again, like theI don't know how to put into words howfar the quality control has fallen.I was really hoping glass would be theReset we needed. It hasn't been. Thishas become one of my favorite tests todo where I take a game that I builtalready that is kind of functioning butneeds a lot of work and I tell the modelto re-implement it from scratch in thisdirectory. I'd like to re-implement mygame fish slop from scratch. I wantbetter assets and more reliable coreengine and a path to making it a realquality web game. The new directory forit is here, use that as a reference andbuild the game in its entirety in thisdirectory. Do not stop until you'redone. Let's see what it does. One of the coolest things about it isthat it's really fast. It's alreadyspinning up parallel agents to dodifferent things at the same time, whichis really cool. One of the crazy thingsthey did, I haven't seen a company trysomething this crazy before, is thatthey did an internal test last week withthe model where all cursor chats foremployees were redirected to composer. 2.5 for a few days. This employee didn'teven notice, which is a testament to howgood the model's gotten. That is supercool to hear. And you can see it flyingthrough this stuff right now. It is, itis churning. It did just do like fiveedits in a row here to like the samething. -1 +1 -1 +1 -1 +1 -1 +7 +1 -7.Interesting.It's actually checking the resultingcode, which when I tried the same thingwith 3.5 flash yesterday, it did not do. On that note, 3.5 flash, which also cameout at the same time as composer 2.5, got away lower score than 2.5, lower than2.0, and cost $1.94. I just, I don't seewhy anyone would use 3.5 flash foralmost anything. It's just not a goodoption. It's done? Oof. Page failed to load. Here's the result. So, it is one of the two models thatfailed to make the actual page render,which is sad but not super surprising.It is a Chinese open weight model thatwas RL'd to all hell. Yeah, let's seeHow this works now,I don't have enough shells to unlockthose. Well, unlike the one that Gemini made,it actually functions and has the coremechanics behaving.That is genuinely insane. I don't lovethe asset it generated here.I'll give it an annoying task, actually,to fix that, pull over all the assetsfrom the original game. Let's see ifit's able to swap out the assets for theones from the original game and howquickly it's able to do that. PNG assetsexist in Fish Game Public, but the globtool failed to detect them. Interesting. This is a big move, changing the entirerender pipeline for your game live. Youcan see the game flashing in thebackground as it does all the changes. Apparently, it's done. That took under30 seconds. And yeah, I switched over tomy original assets.It made a lot of things too small. Let'stell it to fix that. Scale is really off. and UI is hard to read, low contrast. Onthe topic of UI, I do want to talk aboutthe UI it generates cuz it's actuallykind of impressive. Friend of thechannel Dara has made an awesome toolthat lets you compare what models designand how they handle design in differentways. The site itself was designed byComposer 2, funny enough. He runsdifferent models with the same prompt togenerate different web pages to see howgood they are at design. And here arethe designs that we got out of Composer2.5. This one's actually kind of nice and chill. I like the way it used thefont. I like the little thing it madehere. Tasteful grain. Yeah, this isfine. This is clearly using thefront-end design skill. Yeah, cuz theseare all the design you get out ofthe front-end design skill. It screwedup the spacing on these buttons a bit,like that should be uh aligned to thebottom, I think.And this doesn't look great either, butLike it, it honored the goal of thisdesign, okay. This one I just, I don't like. I don'tlike this style of, like, preppy.[\_\_]I hate the retro ones.And this is fine. So, here's how it goeswithout the front-end design skill fromInfropik. This looks fine. This looks fine, but way too similar.This is different.It's different in a way that's kind ofcute. I don't love it, but it iscute. This is awful.And this islike an old Tailwind slash, likeBootstrap site. I've seen a lot ofsites that look like this. This feelslike an old era design that you wouldget in the, the purple-pink gradientclassic. Not bad. Like getting thistype of design out of a model this cheapis really cool to see. Interesting bughere. We'll resume when background workfinishes. What is the background work? That it is finishing? Cool, they calledme.Let's see how this looks now.Oh.Yeah, it screwed up the scaling a bunchfor this, sadly.It got like the core mechanics right.These buttons on the bottom don't workfor clicking things, though. Like I haveto press the hot keys. It'sfar from perfect, but I could seehow somebody who doesn't have a lot ofmoney, does have a cursor sub or like astudent discount or something, couldwith enough massaging and back and forthmake something actually decent out ofthis. Like I see the path there. It'snot a super happy path, but there isone. Okay, it actually does have a lotof things wrong with the feedingmechanic in the game. It’s sadly not theworst solution I’ve seen for this, butit is not great. This is an admittedlyvery complex task, but yeah, I wasHoping for a little more there. For mylike day-to-day random TypeScriptend-to-end [ \_\_ ] I found it to berelatively pleasant, but there is a bigproblem. The problem isn't that I don't likeglass, even though I don't. The problemisn't that it's too expensive or tooslow or too dumb or any of those things.The problem's actually over here onCursor Bench. And the problem isn'tCursor Bench itself. It's not that it'sa dishonest test or anything like that. It's that we don't have anything [ \_\_ ]else to measure against. Because unlikeevery other model that we talk about onthis channel, there is no API we can hitto test this model ourselves. The onlyway to try out Composer, regardless ofhow much money you're willing to spend,is to use it inside of Cursor. Unlessyou spend $60 billion to buy them, thenyou can probably use it for whatever. Point I'm trying to make is that wedon't have a real measurement of thisOutside of what Cursor is putting out,the Rob did come out and say that theyare reporting terminal bench and SWEbench multilingual, which sadly is apolluted test. Enough info on it hasleaked that models have it in thetraining data and it's kind ofcompromised now. It is what it is. Allegedly, they are working withartificial analysis and others for moreexternal evals, which is very good tohear. But I understand on one hand whythey're not going hard on this becausethey made this model just for coding andnothing else. It'll probably performreally poorly on stuff like SkateBench,for example, because it isn't built forthat use case. It's built just forcoding. And a lot of these benches, likeif you were to throw it on theartificial analysis intelligence index,the core bench they have here, it wouldperform very poorly, possibly as bad asKim iK25 or worse, because a lot of thebenches are things that aren't code. So IGet why they're not prioritizing this,but it does suck to have a model thatseems really useful for a bunch of different [ \_\_ ] that I can only usethrough Cursor's app or through the ACPimplementation on their CLI that we hadto like push really hard for them tofix. For a while you couldn't even useComposer models through the ACP bindingsbecause they had a limited hard-codedset of models that didn't include thenew Composer models. Obnoxious. They'vesince fixed that after a lot of back andforth with the team. They also just putout the Cursor SDK, which allows you tobuild agents using Cursor's harness andinfrastructure and everything, whichhypothetically means you can write codelike this, where you create an agentthat uses Composer 2 and has access to aspecific directory, and then you canprogrammatically run things against it. This is cool. Especially cuz you canpoint it at cloud repos as well. There'sa lot of promise here. And as I'veMentioned many times, I do genuinelyreally, really like the Cursor Cloudstuff. Cursor.com/agents hasconsistently been my favorite way to useCursor as of late because it has a fullLinux box in the cloud that it can useto spin up whatever your project is, aslong as it's not like an iOS app, andactually test against it with computeruse. Super cool when combined with fastmodels like Composer 2 and 2.5 and theability to trigger these from my phoneor from Slack. I like where they'regoing with all of this. I just hate whenI see a new model with numbers that arethis good, and I can't it. This is cool,especially because you can point it at cloudrepos as well. There's a lot of promisehere. They're the only lab, in quotes, worthlooking at that doesn't expose theirstuff over API at all. This is verydifferent for companies like Anthropic,who subsidize usage through their tool,but charge you full price to hit it. Directly. The only thing going on right now that's close is Gemini 3.5 Flash,where you can pay API prices, but if you want the full speed version, that is the one that runs at 1,000 TPS instead of 200.You have to use antigravity IDE or CLI to get the full speeds.There are very few instances nowadays, thankfully, of labs restricting access to their models on a fundamental level.Usually, they just subsidize. But now we have both Google restricting us from using the fast version of their model over the API.We can only use the slow one.Previously, we had Mistral do the same thing, where they had a version of the model that could do 1,000 TPS that was working through their partnership with Cerebras at the time.But if you hit it over the official Mistral API, you'd be around 50 TPS.And now we have Composer, which is literally only accessible via Cursor surfaces.You cannot do it over API at all.Apparently, people have it working in tools like Pi by using theAgent SDK that Cursor put out. That'scool to see, and I don't think they'regoing to be as egregious with limitinghow people use the SDK as companies likeyou know, the one big A has been aboutthat. But it does suck that I feel like I can't meaningfully benchmark this orintegrate it in other tools. Like if Ijust wanted to swap out the API endpointin something like Snitch Bench or insideof something like Pi to use what theyhave here, I can't. I have to rebuildthe application to use their SDK. Andfor what it is worth, I have not heardthe best things about the SDK. I haveadmittedly not had a chance to try itmyself, but I have heard that it ismissing things and is not the mostreliable stuff yet. That said, if enoughof you guys build cool things with theCursor SDK, I will have the leverage Ineed to force the team to push itfurther. And that would be really cool. So I do still recommend you go buildcool things with this. Just make sureYou do it in a way where if you do have to swap out, it's not that hard to do. So what are my final thoughts on all of this? Well, if you're the type of developer that is still staring at the agent as it runs and are doing very collaborative work with it, where you ask it to research a thing, it researches it, it gives you a result, you ask it to then go make a change, and then it does that, you look at the result, where you're having a back and forth that you're engaged with the model, having a model this fast and smart is really, really powerful. That's why I was excited for 3.5 Flash and ultimately was disappointed because it wasn't good enough at coding to have a good experience doing that with.Composer 2.5 is a much better balance here. It might not be as capable of unblocking itself in solving complex tasks as things like the latest GPT models or the latest Opus models, but it is really fast and close enough that youCan have this back and forth with it ina way that reminds me of the old daysdoing a gentle coding with thedifference being it's way smarter andway more capable. So, the people who areworking at big companies on giant reposthat just want to like sit in the codebase with a real IDE and talk back andforth with the model about the changesthey want to make and not have to wait20 minutes for it to go explore [ \_\_ ]. This is an incredible model. But, ifyou're the type of person that'sspinning up 20 agents in parallel on thesame thing, all doing different [ \_\_ ].And by the time you spin up agent 17, agents1 through 6 are done, this will not beas useful to you. That said, I findmyself steering away from the superparallelization stuff lately because it'sjust too much context in your head tokeep, and if you're doing anything else,it's just impossible to maintain. Ican't tell you how many cool projectsare probably sitting on random workTrees on this computer that I've never had a chance to even look at yet. So, ifyou're doing this type of engaged backand forth with the model, a fast modelis really powerful. I'm saying thisconfidently because I've experienced itmyself recently because I've been usingGPT-4-5 on the fast mode for like 2weeks now because I got way too muchfree usage from OpenAI because I went to the4-5 event and everybody who applied togo got, I think, a 10x bump. So, despitebuilding a literal cloud from scratchover the last few days using thesemodels, yeah, I'm building a cloud. We'll talk about that another time,don't worry. I'm not struggling to eventake down 1% of my 5-hour limit, muchless my weekly usage limits. I'm tryingso hard to burn through this and Iliterally can't because they gave me toohigh of a limit as a result of thatevent. Insane. It is what it is, but assuch, I've gotten addicted to fastbecause it's 2x more expensive and I'mStill barely able to burn the [\_\_] tokens. I think Composer 2.5 fast is amuch more realistic option for companieswhere they can't have devs doing thesubsidized token usage that also don'thave that free 10x bump temporarily. Forcompanies with big code bases that have developers actually in their editorsreading the files and looking at thechanges, like hand-in-hand with themodel. It's a good workflow, and havinga model this fast, this cheap, thisreliable for that type of real-worldwork fits the archetype of Cursor'scustomer very well. Because Cursor iskilling it in enterprise. When I talk tomy friends at big companies that aren’tAmazon, where they built their ownshitty IDE, almost everyone else I talkto is using Cursor. So much of theFortune 500 is on Cursor, it's insane. This is a really good win for them thatwill keep them from taking a potentialdeal with OpenAI or with Anthropic thatI'm sure they're in negotiations with. All of those companies. This is amassive play. And unlike those otherlabs, if you go to Anthropic, forexample, with a deal, you can only usethe latest Opus and Sonnet models. IfOpenAI does something cool, you're[ \_\_ ] If Cursor does something cool,you're [ \_\_ ]. Same deal if you go withOpenAI. But if you go with Cursor, youat the very least have their awesomemodels with Composer. But if one of theother labs leapfrogs at any point, youstill get access to that, too. So,Cursor has continued to be a reallycompelling bet for enterprises, and Ithink these developments help that aton. By having a really good defaultthat they can subsidize themselves andmake really compelling valuepropositions for their users with, theyare able to stay a player in this race. And as silly as it sounds, I have morefaith in Cursor to sustain the AGIbubble or whatever happens than I do acompany like Microsoft. Because at theVery least, Cursor's moved far beyondthe VS Code fork we made fun of them for. It's into a real lab focused on realdeveloper workflows at a level thatMicrosoft cannot even comprehend.If Iwas to guess where we'll be in 2 years,I legitimately think it's more likelythat we're using xAI Cursor Composer 7than anything by Gemini for ourday-to-day dev work. Even if it thinksthings are going on in the backgroundwhen they're not, bugs will happen.Ihave one last thing I want to end onhere.Cursor posted in the thread wherethey announced the new model, "Togetherwith SpaceX AI, we are training asignificantly larger model from scratch.Using 10x more total compute, that meansit's going to be 100x more than Kimmywas trained on,"With Colossus 2'smillion H100 equivalents and ourcombined data and training techniques, we expect this to be a major leap inmodel capability. It is legitimatelypossible that Cursor doesn't just catchUp to the state of the art for code, theymight leapfrog. There's a real chancethat in just a few months, Cursor willhave the best model for code. And whenyou look at the trajectory, when youlook at how fast they have scaled up,where they are, and combine that withthe insane amount of compute they haveand data and all of the new trainingmethods they've been figuring out, Ithink there's a real chance they do it. At least I hope so so my investmentbecomes something, but we'll see wherethat all goes. I got nothing else onthis one. If you're still on Cursor, youshould definitely try it. And if you'renot on Cursor, I'm curious what you'reusing instead. Let me know in thecomments and until next time,peace, nerds., Cursor just crushed Claude Code, YouTube, 01ksfq45grknx383tzqjmypd90.