GLM-5 is unbelievable (Opus for 20% the cost)

GLM-5 is unbelievable (Opus for 20% the cost??) 🔗 **Source:** [GLM-5 is unbelievable (Opus for 20% the cost??)](https://www.youtube.com/watch?v=vtWMgVCMsx8) > [!note]+ Document Notes > tags: [ai-models, open-source, code-generation, model-evaluation, glm-5] > author: Theo - t3․gg > source: YouTube > reference: https://www.youtube.com/watch?v=vtWMgVCMsx8 > date created: 2024-12-19 14:30 > > # 🏷️ Title: GLM-5 is unbelievable (Opus for 20% the cost??) >  ## 🧰 Action Inbox - [H1] Test GLM-5 for long-running code migrations — significantly cheaper than closed models #testing - [H2] Evaluate cost savings for agency work — 3-6x cheaper than Opus/Claude #business - [H3] Set up GLM-5 access through multiple providers — Kilo, Modal, OpenRouter #setup ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ### 🔗 References - [GLM-5 Official Blog](https://blog.zhipuai.com) - [Artificial Analysis Report](https://artificialanalysis.ai) ## 📝 Summary - GLM-5 from Zhipu AI is the best open-weight model released to date, particularly for coding tasks - Successfully completed a 1-hour code migration that closed models typically can't handle - Costs 3-6x less than premium models (Opus, Claude) while delivering comparable performance - Key limitations: no multimodal support, requires significant compute resources (1.5TB model) - Bottom line: First open-weight model that's genuinely competitive with frontier closed models for engineering work ## 📖 Main Content GLM-5 represents a significant breakthrough in open-weight AI models, particularly for software engineering tasks. The model uses a mixture-of-experts architecture with 744 billion total parameters but only 40 billion active parameters, making it more efficient while maintaining high capability. The model's standout feature is its ability to handle long-horizon tasks that typically require expensive closed models. In testing, it successfully completed complex code migrations running over an hour, demonstrating sustained reasoning and execution capabilities previously unseen in open-weight models. Pricing is substantially lower than comparable closed models: approximately $3 per million output tokens and $1 per million input tokens, making it 3-6x cheaper than Claude Opus or GPT-4 variants while delivering similar performance on coding benchmarks. ## 🗒️ Notes #### ⏱️ [00:00](https://youtu.be/vtWMgVCMsx8?t=0) — [H1] GLM-5 Introduction and Performance Claims - Key idea: GLM-5 is the best open-weight model released, especially for code - Evidence: "By far the best openweight model I have seen, especially for code stuff" #### ⏱️ [02:15](https://youtu.be/vtWMgVCMsx8?t=135) — [H2] Sponsor Segment - DNS Simple - Key idea: DNS hosting service built by engineers for engineers - Evidence: "DNS Simple is a product built by engineers for engineers at its very core" #### ⏱️ [04:30](https://youtu.be/vtWMgVCMsx8?t=270) — [H1] Real-World Testing with Hour-Long Migration - Key idea: GLM-5 successfully ran a complex codebase migration for over an hour - Evidence: "This migration for an old codebase that actually ran for an hour for free" #### ⏱️ [06:45](https://youtu.be/vtWMgVCMsx8?t=405) — [H2] Pricing and Provider Access - Key idea: Available through multiple providers at competitive pricing - Evidence: "About $3 per mill out and a dollar per mill in. That puts it at a third to a fourth the price of Sonnet" #### ⏱️ [09:20](https://youtu.be/vtWMgVCMsx8?t=560) — [H1] Ping.gg Migration Project Details - Key idea: Testing GLM-5 on outdated codebase with complex dependencies - Evidence: "This codebase has gotten out of date... A lot of it is because I made early bets on things that have changed" #### ⏱️ [12:10](https://youtu.be/vtWMgVCMsx8?t=730) — [H3] Vision Capability Limitations - Key idea: GLM-5 lacks multimodal support, which limits debugging workflows - Evidence: "It doesn't have vision... 20 to 50% of my prompts have an image in them" #### ⏱️ [15:30](https://youtu.be/vtWMgVCMsx8?t=930) — [H1] Technical Specifications Deep Dive - Key idea: 744B parameters with 40B active using mixture of experts - Evidence: "744 billion parameters, 40 bill active... 1.5 terabytes" #### ⏱️ [18:45](https://youtu.be/vtWMgVCMsx8?t=1125) — [H2] Benchmark Performance Analysis - Key idea: Competitive with Opus 4.5 and Codex 5.2 on key coding benchmarks - Evidence: "Getting neck and neck with that on S webench is huge" #### ⏱️ [22:00](https://youtu.be/vtWMgVCMsx8?t=1320) — [H1] Successful Migration Completion - Key idea: The hour-long migration completed successfully and the app works - Evidence: "All working as expected. That is very impressive" #### ⏱️ [25:30](https://youtu.be/vtWMgVCMsx8?t=1530) — [H2] UI Design Generation Testing - Key idea: GLM-5 can generate reasonable UI designs for front-end work - Evidence: "The fact that you can get design this good out of an open weight model is kind of freaking nuts" #### ⏱️ [28:15](https://youtu.be/vtWMgVCMsx8?t=1695) — [H1] Hallucination Rate Breakthrough - Key idea: GLM-5 has the lowest hallucination rate seen in any model - Evidence: "Its hallucination rate is in the 30% range versus Gemini 3 Pro at 88%" #### ⏱️ [30:45](https://youtu.be/vtWMgVCMsx8?t=1845) — [H1] Cost-Intelligence Value Proposition - Key idea: GLM-5 offers near-frontier intelligence at a fraction of the cost - Evidence: "GLM5 cost $500 to run this benchmark, whereas Opus 4.6 cost $1,500. 3x cheaper" #### ⏱️ [33:20](https://youtu.be/vtWMgVCMsx8?t=2000) — [H1] Market Competition Analysis - Key idea: Open-weight models are rapidly catching up to closed frontier models - Evidence: "The speed at which these labs are catching up with Frontier is insane" ## 📚 References - [GLM-5 Technical Blog Post](https://blog.zhipuai.com) - [Artificial Analysis Intelligence Benchmarks](https://artificialanalysis.ai) - [SWEBench Coding Evaluation](https://swebench.com) - [Zhipu AI Model Releases](https://zhipuai.com) ## 🗣️ Paraphrase - GLM-5 is the first open-weight model that can genuinely compete with premium closed models like Claude Opus for software engineering tasks - The model successfully completed complex, long-running code migrations that typically require expensive frontier models, demonstrating sustained reasoning capabilities - At 3-6x lower cost than comparable closed models, GLM-5 makes high-quality AI-assisted development accessible to more developers and organizations ## 📰 Newsletter 🚀 **GLM-5: The Open-Weight Model That Changes Everything** The AI landscape just shifted dramatically with GLM-5's release from Zhipu AI. This open-weight model is the first to genuinely compete with premium closed models for software engineering tasks. **Why This Matters for WordPress Developers:** - **Cost Savings**: 3-6x cheaper than Claude Opus while delivering similar coding performance - **Long-Running Tasks**: Successfully handles complex migrations and refactoring that typically require expensive models - **Open Access**: MIT licensed with no usage restrictions, making it accessible through multiple hosting providers **Key Limitations to Consider:** - No multimodal support (can't analyze screenshots) - Requires significant compute resources (1.5TB model size) - Currently limited provider availability with some latency issues **Getting Started:** Try GLM-5 through providers like Modal (free tier) or OpenRouter ($3/million output tokens). Perfect for complex WordPress migrations, plugin development, or large-scale refactoring projects. The gap between open and closed models is closing faster than expected. This could fundamentally change how we budget for AI-assisted development. Resources: [Artificial Analysis](https://artificialanalysis.ai/), [OpenRouter API](https://openrouter.ai/), [Modal Hosting](https://modal.com/) ## ✅ Action Items #### DevRel - Developer advocacy: Create GLM-5 integration tutorials for WordPress development workflows - Education: Workshop on cost-effective AI model selection for agencies - Developer experience: Test GLM-5 with WP-CLI and core contribution workflows - Developer marketing: Case study on $10 migration vs $1000+ traditional costs - Community management: Survey community interest in open-weight model hosting #### WordPress - Core: Evaluate GLM-5 for automated code modernization in legacy components - Core Performance: Test model's ability to identify and fix performance bottlenecks - Documentation: Assess GLM-5's capability for generating developer documentation - Plugins: Pilot program for automated plugin security reviews - Meta: Investigate hosting GLM-5 for community development assistance - CLI: Test scaffolding and code generation capabilities ## 🧪 Hypotheses - Open-weight models will achieve parity with closed frontier models for coding tasks within 12 months - GLM-5's low hallucination rate makes it more reliable for production code generation than current closed models - The 3-6x cost advantage will drive significant enterprise adoption for development workflows - Mixture-of-experts architecture will become the standard for all large language models ### 📊 Methodology - Benchmark GLM-5 against Claude Opus 4.6 on WordPress-specific coding tasks - Measure cost per successful code generation across different model types - Survey developer satisfaction with open-weight vs closed model outputs - Track accuracy rates for WordPress plugin development assistance ### ❓ Questions - Will GLM-5's lack of vision capabilities limit its usefulness for debugging UI issues? - Can open-weight models maintain competitive performance as closed models continue improving? - What hosting costs will make GLM-5 economically viable for small development teams? ### 🔮 Future Research - Comparative analysis of GLM-5 vs upcoming GPT-5 and Claude 4.0 models - Investigation of fine-tuning GLM-5 for WordPress-specific development patterns - Study of mixture-of-experts optimization for domain-specific coding tasks ### 💡 Implications - Practice: Developers can access near-frontier AI capabilities at significantly lower costs - Policy: Open-weight models may reduce dependency on major tech platforms for AI services - Security: MIT licensing allows for internal hosting and security auditing - Education: More accessible AI tools for learning programming and software development - Funding: Development budgets can allocate more resources to human developers vs AI costs ### 🧠 Learning Styles Assessment Reading level estimate: Flesch-Kincaid Grade Level 12 #### 👀 Visual learners - Present: Benchmark charts, performance comparisons, UI design examples - Improve: Architecture diagrams showing mixture-of-experts structure #### 🔊 Aural learners - Present: Video format with detailed narration and code explanations - Improve: Audio-only version highlighting key performance metrics #### 🗣️ Verbal learners - Present: Technical terminology defined in context, analogies for model scaling - Improve: Glossary of AI model terms and cost comparison frameworks #### 🧩 Kinesthetic learners - Present: Live coding demonstrations, real migration examples - Improve: Step-by-step setup guides for testing GLM-5 #### 🧠 Logical learners - Present: Systematic benchmark comparisons, cost-benefit analyses - Improve: Decision trees for model selection based on project requirements #### 🧑‍🤝‍🧑 Social learners - Present: Community discussions about model adoption and experiences - Improve: Developer forums for sharing GLM-5 use cases and results #### 🧘‍♀️ Solitary learners - Present: Detailed technical specifications and independent testing methodology - Improve: Self-assessment tools for determining optimal model choice #### 🌿 Naturalistic learners - Present: Evolution analogies for model development and improvement cycles - Improve: Ecosystem thinking about open vs closed model sustainability ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Clear heading hierarchy, descriptive video narration - Improve: Image alt text for benchmark charts and code examples #### Keyboard navigation - Present: Standard YouTube controls, timestamp navigation - Improve: Keyboard shortcuts for code section jumping #### Contrast and readability - Present: High contrast code examples, clear visual distinctions - Improve: Enhanced color coding for different model performance levels #### Text resizing and reflow - Present: Responsive layout on video platform - Improve: Transcription with adjustable text sizing options #### Media and captions - Present: Auto-generated captions available - Improve: Professional captions with technical term accuracy #### Forms and interactions - Present: Standard video player interactions - Improve: Interactive benchmark comparison tools ### 🌍 DEIB Assessment #### Cultural breadth - Present: Global perspective on AI model development from Chinese lab - Improve: Discussion of model availability across different regions #### Inclusive language - Present: Technical focus without exclusionary terminology - Improve: Explanation of barriers to accessing large model hosting #### Neurodiversity support - Present: Clear section breaks, consistent presentation structure - Improve: Executive summary for main conclusions #### Representation - Present: Recognition of diverse AI development ecosystem - Improve: Highlighting accessibility benefits of cost-effective AI tools #### Psychological safety - Present: Balanced assessment of model strengths and limitations - Improve: Discussion of potential risks with open-weight model adoption ### ✅ Logical Validity #### Strong reasoning - Deductive strength: Benchmark comparisons logically support performance claims - Inductive strength: Single migration example generalizes to broader capability claims - Causal reasoning: Cost advantages clearly linked to technical architecture choices - Analogical strength: Comparison to previous model releases shows development trajectory #### Fallacies - Hasty generalization: Single successful migration doesn't prove consistent reliability across all codebases - Appeal to novelty: Newer model isn't automatically better for all use cases #### Tone of voice - Overall tone: Enthusiastic but technically grounded, with clear acknowledgment of limitations ### 🛡️ DARVO Analysis - Deny: No denial patterns detected - Attack: No attacks on competing models or companies - Reverse victim/offender: No victim/offender reversal present - Emotional cues and rhetoric: Genuine excitement about technical capabilities balanced with honest limitation discussion ## 🎯 Conclusion GLM-5 represents a watershed moment for open-weight AI models, delivering frontier-level performance for software engineering tasks at dramatically lower costs. While limitations like lack of multimodal support exist, the successful completion of hour-long code migrations demonstrates genuine capability for complex development work. The 3-6x cost advantage over premium closed models could democratize access to high-quality AI-assisted development, particularly benefiting smaller --- ## Full Transcript [00:00] Get your counter ready, folks, cuz [00:01] there's another new model. GLM5 is [00:03] blowing me away. Just to jump in front [00:05] of everybody's usual question, is it [00:07] better than Opus 4.6? Is it better than [00:09] codeex 5.3? No, it's an openweight model [00:11] from a much smaller lab in China. That [00:13] said, it is by far the best openweight [00:15] model I have seen, especially for code [00:17] stuff. KK 2.5 was incredible. It's [00:19] mind-blowing how good it is for an [00:21] openweight model you can go download the [00:22] files for and run yourself. But GLM5 is [00:25] a meaningful improvement. It's neck [00:26] andneck with models like 5.2 2 codecs in [00:29] Opus 4.5. Obviously, the most recent [00:31] things like Opus 4.6 are still slightly [00:33] edging it out, but at the very least, [00:35] I've had a much better experience with [00:36] it than I have with Gemini 3 Pro. That [00:38] said, I really want to push it to its [00:40] limits. They also put out a desktop app [00:41] similar to the one that OpenAI put out [00:43] for Codeex. That is very, very exciting. [00:45] But obviously, this only works with [00:47] their hosting cuz it's their product. [00:49] So, you probably need to sign up for [00:50] that, right? We're going to give this [00:50] model an honest shot, especially this [00:52] early when it's not available in all of [00:54] the platforms just yet. We got to do the [00:56] max plan. 80 bucks a month is a little [00:59] high for a sub that's not going to be my [01:00] main, but I want to make sure we're [01:02] testing this to its fullest. If you [01:03] needed proof that I'm not being paid by [01:05] Zi, here you go. I just paid them. The [01:07] only people paying me for today's video [01:09] is today's sponsor. There's a lot of [01:10] things that are hard about software dev, [01:11] but there's one in particular that's so [01:13] hard that we meme on it constantly. DNS. [01:16] I can't tell you how many times I've had [01:17] crazy problems that were results of [01:19] doing one tiny minuscule thing wrong [01:21] with DNS. If only it was simple. Oh, DNS [01:25] simple. reliable DNS hosting and domain [01:27] names. Okay, real talk. I need you guys [01:29] to pay attention to this one because [01:30] they're not your normal sponsor on this [01:32] channel. Most of the companies you see [01:34] me talking about here are VCbacked [01:35] businesses trying to scale the billions [01:37] of dollars of revenue with teams that [01:38] are growing like mad. Meanwhile, I got [01:40] this shirt from Dian Simple in a box [01:42] with a handwritten address to and from [01:45] because the founders shipped it from [01:46] their home themselves. Dian Simple is a [01:48] product built by engineers for engineers [01:50] at its very very core. They're not a [01:53] domain reseller trying to rack up crazy [01:55] margins or some weird sketchy company [01:57] trying to buy domains right out from [01:58] under you. They just want to make DNS [02:00] easier. When you have problems and hit [02:02] them up, the person you talk to is a [02:03] dev. If you ever wondered how good DNS [02:05] could be as a developer, look no [02:07] further. Want to list all the domains in [02:09] your account? Here's the Ruby API for [02:11] it. Want to register a new domain with a [02:12] new contact? Here's the TypeScript API [02:14] for it. Need to set up a Let's Encrypt [02:16] SSL certificate? Here's the line of code [02:18] you have to run. If you want to do [02:19] anything even vaguely complex with DNS, [02:22] like I don't know, making subdomains for [02:24] your users when they vibe code an app or [02:25] letting users purchase domains [02:27] themselves in your product, integrating [02:28] all of that is hellish. I know companies [02:30] with 10 plus people teams that should [02:32] just be using dnsimple instead. Stop [02:34] wasting your time and do domains right [02:36] at sidyv.link/dnsimple. [02:38] We are in. We're definitely going to [02:39] dive deep on the Zcode app, but I've [02:41] been doing work behind the scenes while [02:43] prepping for stream today, running very [02:46] real work, including this migration for [02:48] an old codebase that actually ran for an [02:51] hour for free. This UI might look [02:53] familiar, like you know, open code, but [02:55] if you look at the top here, it's Kilo [02:57] code. Kilo is a sponsor, so counter some [02:59] bias there. They recently forked open [03:01] code to make their own CLI, and it's [03:03] been pretty cool cuz you can use it with [03:04] the Kilo subub. They also were the first [03:06] provider to have consistent support for [03:08] this new model. And not only are they [03:10] supporting GLM5, they're supporting it [03:11] for free. And it seems like a handful of [03:13] providers are doing the same thing, [03:14] including companies like Modal where [03:16] they have an API endpoint where they are [03:17] hosting this new model that you can use [03:19] for free right now. In the future, do I [03:21] expect it to stay free? Probably not. [03:23] The current pricing I'm seeing on Open [03:24] Router is about $3 per mill out and a [03:27] dollar per mill in. That puts it at a [03:28] third to a fourth the price of Sonnet [03:30] and a fifth to a sixth the price of [03:32] Opus. It's a really really good value [03:34] for what it is, but also uptime's not [03:36] great because these just dropped. A lot [03:38] of the platforms have an insane amount [03:39] of latency, like seven plus seconds. But [03:41] this will all get resolved in very short [03:43] time. It tends to when these new models [03:45] drop. What we really want to care about [03:46] now is the performance. Not the speed, [03:48] but the capabilities. And the fact this [03:50] is the first openweight model that I've [03:52] successfully run a job that took over an [03:54] hour for without issues just absolutely [03:57] blew me away. And I watched it as it was [03:59] running. It did a lot of smart things. [04:01] The project I threw this at immediately [04:03] was Ping. If you're not familiar with [04:04] Ping, I built this a while ago right [04:05] after leaving Twitch to make it easier [04:07] to do live collaborations. It's used by [04:08] everybody from like Linus Tech Tips to [04:10] Elgato to the Xbox team. It was made to [04:12] make live streaming collaborations [04:14] easier. Still proud of the product, but [04:15] it has not been our main focus because [04:17] it turns out creator tools don't make a [04:18] whole lot of money. So, we've been [04:19] focusing on everything else we're [04:20] building. As such, this codebase has [04:22] gotten uh out of date is a little little [04:26] generous. It's not a fun codebase to [04:28] work in anymore if I'm being real. A lot [04:30] of it is because I made early bets on [04:31] things that have changed a lot since, [04:33] like really old versions of TRPC. [04:35] Migrating this codebase to modern tools [04:37] is one of my favorite challenges to [04:38] throw models at because it's not [04:40] something they really can do, but the [04:42] closer they get, the more interesting [04:44] they are to me. As I mentioned in my [04:45] codeex 5.3 video, it was the first model [04:47] that could make meaningful progress in [04:48] this and it did some novel things around [04:50] patch packaging in order to make it work [04:52] and unblock itself throughout. Really [04:54] impressed with 5.3's performance, but I [04:56] just ran it with GLM and it seems like [04:58] it is doing very well. It hasn't done [05:00] the last phase of this migration just [05:02] yet. I want to see how it runs for now. [05:04] We don't have great Firefox support [05:06] because it is a media product and [05:07] Firefox's AV layer is garbage. Okay, [05:10] this is a common error that I get. Every [05:12] single one of the tools I've used for [05:14] this has had this particular bug. [05:15] Actually, this is a great first issue I [05:17] have with GLM5. Paste. Fix it. Send. [05:22] Okay, it's a little confused or the [05:24] servers are lagging. It doesn't have [05:25] vision. I know that's silly. It's a code [05:27] model. Why would you need it to have [05:29] vision? I paste images all of the time. [05:32] It is one of the main things I do when [05:34] I'm writing code. I would honestly guess [05:36] that 20 to 50% of my prompts have an [05:38] image in them. So, not having that [05:40] sucks. It's one of the things that I [05:41] loved about commun 2.5 is they finally [05:43] fixed it. I am getting an error. Please [05:46] fix it. We will let it do this while I [05:48] look at other tasks that I ran. Jesus [05:50] Christ. This model is willing to just [05:52] work, which is really, really cool. It's [05:55] nice having an openweight model that can [05:57] go for this long. Oh, here was the first [05:59] time I tried to paste an image. You're [06:00] saying the job is working, but I can't [06:02] see the image. Yeah, annoying, but it [06:04] works. While I let this job run in the [06:06] background, let's read a little bit [06:08] about what they have to say about the [06:10] model, as well as all of our friends who [06:11] review these things. Let's start with [06:12] their blog post. I feel like the vibe [06:14] coding to Agentic Engineering title is a [06:16] bit clickbaity, but I understand why [06:18] they're saying it. This from my [06:19] experience so far feels like a [06:21] meaningful jump in what the openweight [06:23] models are capable of from like [06:24] oneshotting cool looking demos over to [06:27] something a little more useful that can [06:29] go a lot longer and do harder things. [06:31] Like I've never seen an openweight model [06:33] that can do migration work like I've had [06:35] GLM5 doing and I can't wait to see if [06:37] that work ends up working. We are [06:38] launching GLM5 targeting complex system [06:41] engineering and long horizon agentic [06:43] tasks. Yeah, I can tell. Scaling is [06:45] still one of the most important ways to [06:46] improve intelligence efficiency of AGI. [06:49] Compared to GLM 4.5, 5 scales from 355 [06:52] billion parameters, 32 bill active, to [06:54] 744 billion parameters, 40 bill active. [06:56] If you're not familiar with what this [06:58] means, it's the mappings inside of the [07:00] model for what points to what and how [07:02] strongly it does. The parameters are [07:04] basically like tokens and vectors that [07:06] map together what should happen based on [07:09] what text you already have. For [07:10] reference, Kimmy K 2.5 is a one trillion [07:13] parameter model. 744 bill is close, [07:16] which means this model is going to be [07:17] huge. 1.5 terabytes. Yeah. Versus 4.7, [07:23] which was 700 gigs. But they do have FP8 [07:26] versions and flash versions that are [07:28] lighter of the old model that this 4.7 [07:30] FP8 version is only 360 gigs. And 4.7 [07:33] flash is a measly tiny 62 gigs. But [07:36] you're not going to be running this [07:37] model on local hardware. I see a lot of [07:39] people who assume that open weight means [07:41] they can run it on their computers. [07:43] You're not running a terabyte model on [07:45] your own hardware. This this is not [07:47] something that is accessible to people [07:48] at their houses. This is more useful to [07:50] the hosting platforms like modal and [07:52] others that can host these openweight [07:54] models to provide them for us to use and [07:55] potentially places like Cerebrus or [07:57] Grock with a queue being able to do [07:59] their own versions of the models that [08:00] are significantly faster. One nice [08:02] detail now that I'm here is that it's [08:03] licensed as a proper MIT license. It [08:05] doesn't have any of the usual carveouts [08:07] as far as I can see. things like you [08:09] have to call it out that you're using us [08:11] or pay us if you have a certain number [08:12] of users. None of those weird things. So [08:15] more parameters means the model is [08:16] bigger. For example, the GPT OSS120B [08:18] model is around 80 to 90 gigs. I can run [08:21] that on my laptop locally, but it's 120 [08:23] bill per. So there's less intelligence [08:24] when there's less parameters. Generally [08:26] speaking, the more parameterization you [08:28] have, the more value you have, the more [08:30] data you have in the model, the more [08:32] intelligence is capable of displaying. [08:34] Generally speaking, obviously over time [08:35] the amount of intelligence you can [08:36] squeeze into a specific size changes, [08:39] but for now bigger model definitely [08:41] means more potential for capability. But [08:43] this 40B active piece, this is mixture [08:45] of experts. What this means is that on a [08:47] given request, instead of going through [08:49] all 740 billion parameters, the 40 bill [08:53] section that is the most likely to be [08:55] relevant to what you're doing, that's [08:56] what's traversed instead. Mixture of [08:58] experts models have been pretty [08:59] prominent for a while now. how it seems [09:01] to be what everyone is doing because you [09:02] can have a super intelligent model that [09:05] has different knowledge of different [09:06] languages, different sciences, all these [09:08] different things that don't necessarily [09:09] need to be traversed. Like you don't [09:11] need to go through all of the Chinese [09:13] data in a given model when you're [09:15] responding in English. You don't need to [09:16] go through all of theoretical physics [09:18] when you're writing a hello world app. [09:19] That's effectively how mixture of [09:21] experts works. 40 bill active is pretty [09:23] cool, but again the size of this model [09:25] is going to make it inaccessible for a [09:26] lot of people. Also, their pre-training [09:28] is insane at 28.5 trillion tokens. GLM5 [09:31] also integrates Deepseek sparse [09:32] attention, significantly reducing [09:34] deployment cost while preserving long [09:35] context capability. Deepseek's research [09:37] has been pushing a lot of things, and [09:39] it's cool to see them call it out here. [09:40] Definitely is going to help with [09:41] performance. They've also made huge [09:43] improvements to their reinforcement [09:44] learning, which is how you tune the [09:45] model after you've made it, so to speak. [09:47] Reinforcement learning aims to bridge [09:49] the gap between competence and [09:50] excellence in pre-trained models. [09:51] However, deploying it at scale for LMS [09:53] is challenging due to RL training and [09:55] efficiency. To this end, we've developed [09:56] Slime, a novel asynchronous RL infra [09:59] that substantially improves training [10:00] throughput and efficiency, enabling more [10:02] fine grain post-training iteration. [10:04] Yeah, that's really cool. I'm assuming, [10:05] I haven't looked into this yet, and it's [10:07] not what we're here for. They found a [10:08] way to run a lot of RL at the same time [10:10] instead of being like somewhat linear. [10:12] That could be really, really cool. [10:13] Excited to see how it works. What I'm [10:14] more excited to see though are these [10:16] benchmarks. Humanity's last exam, it [10:18] beat out pretty much everything other [10:21] than Google's models without tools, but [10:23] with tools, it won. What's really [10:25] interesting here is you can see the gap [10:27] of without to with tools for this [10:29] benchmark because with tools it can like [10:30] do research, write code, stuff like that [10:32] instead of just spitting out text. [10:34] You'll notice here that the ZI models do [10:36] like 40 to 50% better with tools. [10:38] Enthropic models do 20 to 30% better, [10:41] but for some reason it looks like the [10:43] Google and the OpenAI models don't [10:45] anywhere near as much. I have my [10:47] suspicions about that, especially the [10:48] Google models. They don't use tools [10:50] great. Then we have SB bench verified [10:52] which it got neck inneck with Opus and [10:55] Codeex. I believe this is 4.5 and 5.2. [10:59] Yeah, it is. They made these before 4.6 [11:01] dropped. This happens a lot. And 5.3 [11:03] isn't on the API yet, so it's really [11:04] hard to bench against. So, this is the [11:06] best we can do now. But honestly, both [11:08] Opus 45 and 5.2 are more than good [11:11] enough for meaningful day-to-day work. [11:12] 5.3 is a cool improvement. I do use it [11:14] as my daily driver right now. 5.2 is [11:16] still a phenomenal model. Opus 4.5 is [11:18] arguably better than Opus 4 6. Getting [11:20] neck andneck with that on S swebench is [11:23] huge. S SW bench verified. It is not [11:25] only neck andneck with Anthropic. It [11:28] beat out Codex 5.2 and it crushed Gemini [11:32] 3 Pro. And in terminal bench, it beat [11:34] out both Gemini and Codex and is again [11:36] neck andneck with Anthropic. Vending [11:38] bench is the benchmark. How much money [11:40] can it make running a vending machine? [11:41] It did really well there. Tow bench, it [11:43] did very well too. This is a tool [11:44] calling bench. MCP Atlas, it got right [11:47] up there with 5.2. And then browse comp [11:49] which is a browser use benchmark. It got [11:51] the best score as far as I know of all [11:53] time which is really really crazy. You [11:55] see BenchV2 4.7 versus five versus opus [11:58] 4.5. This is a benchmark that they made [12:00] in order to test long running tasks. [12:02] That seems to be the real focus for them [12:03] here which is super cool to see from an [12:05] openweight model. Build success rate has [12:07] bumped a ton from previous models. End [12:09] toend correctness has bumped [12:10] meaningfully as well. For back end, it [12:11] is also improved. But you'll see here [12:13] pretty significant gap if you look at [12:15] the right where front-end tasks can be [12:16] completed 98% of the time. End to end [12:19] correctness 75% of the time across top [12:22] tier models, but backend goes all the [12:24] way down to 26 to 27%. There's a lot of [12:27] opportunity to improve backend work with [12:30] these models still. I can say this [12:31] confidently having been doing a lot of [12:32] it lately, but they slaughter front end [12:34] now. And again, long horizon, apparently [12:37] it's now performing better than [12:39] anthropic models for long horizon tasks. [12:41] Multi-step chain tasks, it's slightly [12:42] worse, but it's still doing way better. [12:44] I will admit, it's kind of funny looking [12:46] at one of these charts where almost all [12:48] of the bold values are to the right of [12:50] the thing we're here to look at. Like, [12:51] they aren't winning anything against the [12:54] best closedweight models. Like, claude [12:55] opus 4.5 is still going to be better [12:58] than this. When you consider that is a [12:59] fifth the price or less and it's open [13:01] source, this is a very very compelling [13:04] model. Let's check in on that job we [13:06] were running. Cool. It fixed it pretty [13:07] quick. Under five minutes. Let's see if [13:09] I can load the page now. It's one of [13:11] those errors that's so nice to [13:12] screenshot and so miserable for copy [13:14] pasting. Apparently their other apps can [13:17] do images. So let's try out Zcode quick [13:20] up into a real project. Is it going to [13:24] use cloud code with GLM? Does it not [13:28] even have GLM5? Yeah, I was under the [13:31] impression that this would have that [13:33] being it's that their product GM5 [13:36] testing. Looks like that worked. Let's [13:40] do a new task. Let's see if it can take [13:42] a screenshot here. Fix. It is a little [13:45] silly that it says cloud code when it's [13:46] using GLM5 through cloud code. At least [13:49] as far as I know, that's how it's [13:50] working. You can feel the slowness. I [13:53] again want to be clear, the speed will [13:55] almost certainly improve. With closed [13:57] weight models from the major labs, if [13:58] they come out slow, they're going to [14:00] stay slow. So if GPT5 is slow when it [14:02] launches, that's probably not going to [14:04] change. If Opus 4.6 is slow when it [14:06] watches, spoiler, it was. It's probably [14:08] not going to change. When the open [14:09] weight models are slow initially, [14:10] there's a very good chance it'll improve [14:12] as hosts find ways to optimize them and [14:14] capacity gets better. All of these [14:16] things, these labs also have less [14:17] capacity overall. Very interesting. It [14:20] did analyze the image, but it used a [14:22] tool to do it because it can't see the [14:24] images itself. It's not multimodal. It [14:26] doesn't have the ability to view things. [14:28] It just has the ability to do things. It [14:31] can't run bun commands. Is it sandboxed? [14:34] Why couldn't it do that? Is it not [14:36] starting? Oh, I know the things they did [14:38] wrong here. I know too much cuz I may or [14:41] may not have been building a similar app [14:43] myself recently. I don't think this is [14:45] going to work. So, we're going to go [14:46] back to using Kilo for now. It's cool [14:48] they're baking an app like this, but [14:49] it's clearly not their primary focus. As [14:52] you can see, for the fact that claude [14:54] code appears on my screen more than ZI's [14:56] branding does, even though I'm using [14:57] ZI's model, the very least, this appears [15:00] to be working. At least [15:03] what did it just break? Oh no. Let's rm [15:07] RF node modules fun install rundev. [15:11] Okay, we appear to be good. I don't know [15:13] what it's compiling. It's taking so [15:14] long. But uh I can sign in still. Give [15:16] me a sec. Cool. This appears to be [15:19] working so far. See how it goes when I [15:22] actually activate my camera. Hi nerds. [15:24] You're seeing behind the scenes a little [15:25] bit. Mike, join. Okay, we got an error [15:28] immediately upon joining. To be fair, [15:30] the Aly client is one of the most cursed [15:32] pieces of software I've ever worked [15:33] with. So, I'm not surprised it's broken. [15:35] There are a few companies I can so [15:36] confidently say are evil, and I've had [15:39] such a bad experience with them. I'm not [15:40] surprised that's what broke. The build [15:42] I'm working on for this that I actually [15:43] hope to ship in the future is one where [15:45] we move to convex and I can deprecate a [15:48] lot of the broken things like that. [15:50] Testing. Cool. And if I refresh, will [15:52] the message still be there? It is. And [15:55] if I do an external quick embed link, [15:59] will I appear? I do. Nice. All working [16:02] as expected. That is very impressive. [16:05] The fact that this model ran for an [16:07] hour, updated the vast majority of my [16:10] core depths, everything other than TRPC, [16:12] and succeeded, and now is working is [16:15] nuts. No closed weight model could do [16:18] this even at the start of this year. The [16:19] fact that an openweight model is doing [16:21] it this early into the year is actually [16:24] really, really impressive. Obviously, [16:26] Simon made his own post as well. It's [16:28] interesting to see Zai take a position [16:29] on what we should call professional [16:31] software engineers building with LMS. [16:33] I've seen agentic engineering show up a [16:35] few other places recently, most notably [16:37] from Cararpathy and Addiosmani. That's [16:39] fair. It seems like the industry is [16:40] trying to move away from the vibe coding [16:42] term because it has so much negative [16:43] association and this is a lot deeper. Oh [16:46] [ __ ] apparently DAX is hanging out in [16:48] chat. Why am I using Kilo CLI? Cuz you [16:49] guys didn't have GLM5 yet. [16:52] Don't worry, Dax. I called out the fact [16:55] that it's a fork of open code at the [16:56] start. People know and that I'm using it [16:58] for this. Is GM good? Oh, I'd miss this. [17:00] Dax, uh, it's insane. It is actually [17:03] capable of super long running things. I [17:05] just ran a job in it that went for [17:08] almost an hour. It was a huge migration [17:10] to I was migrating the old ping.gg [17:12] codebase. The only other model that's [17:14] come close to getting this working [17:15] properly is 5.3 codeex and it did it. [17:19] Yeah. 59 minutes 30 seconds. Almost an [17:21] hour on the dot it ran for. So y'all [17:23] definitely want to get the supported [17:24] ASAP. Can we commit the current changes [17:28] and implement phase 4? Now, it has good [17:31] git etiquette, too. That was not a thing [17:33] I necessarily expected, but it's cool to [17:34] see. I told it once during this to [17:36] commit for me and make this new branch, [17:38] and it seems to be doing all of that [17:40] quite well. This is the most I've felt [17:42] that like competence thing that you get [17:45] from just Opus and Codeex and like top [17:48] tier models. This is the most I felt [17:49] that outside of the major labs. I'm [17:52] feeling something with this one. And [17:54] yes, I did that 1 hour refactor on the [17:56] free tier in Kilo. It's kind of nuts. [17:59] And I would expect a lot of other places [18:01] to offer this for free or really cheap, [18:03] too, because it's not the hardest thing [18:04] in the world to host if you have enough [18:06] GPUs that are competent and able to do [18:07] that. For those of y'all who are looking [18:09] for like near best-in-class [18:10] intelligence, but don't have the budget [18:12] for these $200 plans, this is really [18:15] compelling. This is a big task. Let me [18:17] start with the package updates planning. [18:19] Honestly, a new index y'all should start [18:21] tracking rather than just best new model [18:22] index is how much of the video am I [18:24] actually using the model in coding and [18:26] showing code versus reading other stuff. [18:28] The more I'm time I'm spending in the [18:30] details of a codebase with it, it's [18:32] probably a better model. Man, the [18:33] dashboards for all these sites are [18:35] rough. The amount of times I've been [18:36] signed out randomly on the ZI site is a [18:39] little frustrating. And now I have open [18:41] code proper setup. And look at that. I [18:43] can use GLM5 through my coding plan [18:46] here. Let's try this on a different [18:48] project just to see if it feels any [18:49] faster. I'm planning on releasing this [18:52] app soon. Anything stand out that I [18:56] should address before we launch? [18:59] Throwing it at shoe. See how it does. We [19:01] should do a quick UI test on this. [19:04] Initialize it with vit [19:08] tailwind react and typescript. Make five [19:11] renditions of the homepage. Use your [19:15] front end design skill to make them look [19:19] awesome. [19:21] This did a legit audit. It only ran for [19:23] two minutes, but it found a lot of good [19:24] stuff. Like, it actually properly [19:27] scanned the app, found all the things [19:29] that mattered, and gave good [19:30] suggestions. This is a good model. Like, [19:34] I don't know how else to put it. I'm [19:36] excited to see how it does at UI, [19:37] though. That's where a lot of these [19:38] openweight models struggle. Kimmy 2.5 [19:40] did a good job there, but remains to be [19:42] seen how good. I'll write them all in [19:44] parallel for efficiency. [ __ ] yeah. [19:46] While we wait for that, let's see what [19:48] artificial analysis had to say. The LM5 [19:50] is ZI's first new architecture since [19:52] 4.5. All the other models were part of [19:54] the original architecture. 5 scaled all [19:56] the way up to 744 bill. As we mentioned [19:58] before, it's also BF16 floatingoint [20:01] precision versus FP8 in four for Deep [20:03] Seek V3 and Kimmy K2, which is a big [20:05] part of why the model is so massive. GM5 [20:08] scores 50 on the intelligence index and [20:09] is the new open weights leader up from [20:11] 4.7 score of 42. An eight-point jump [20:13] driven by improvements across agentic [20:15] performance and knowledge and [20:16] hallucination. The hallucination part in [20:18] particular is really really cool. It [20:19] appears to be the model that [20:20] hallucinates the least of any model to [20:22] date. The fact that Google's already [20:24] been crushed by an open weight model is [20:26] kind of crazy. Like Gemini 3 is bad and [20:29] is now it was already pretty irrelevant. [20:31] Now it's done. JM5 abstains more [20:34] frequently in the hallucination bench [20:36] which keeps it from getting negative [20:37] points for lying and being wrong. And [20:39] that is why it got the lowest score [20:41] they've ever seen on the hallucination [20:43] benchmark. Its hallucination rate is in [20:45] the 30% range versus I don't know Gemini [20:48] 3 Pro at 88%. Yeah, it's much more [20:51] willing to say sorry, I don't know that [20:52] than other models that will just lie and [20:54] be wrong. 200K token context [20:56] multimodality is text input and output [20:58] only. K2.5 remains the leading [21:00] openweight model that supports image [21:01] inputs. Yep. And K2.5 is also really [21:04] really good with that. I can honestly [21:05] see value in something like a generic [21:07] image recognition tool that uses Kimmy [21:10] K2.5 just for the image part and then [21:12] you can use this model for everything [21:13] else. I don't think any of the [21:15] openweight models are good enough by [21:17] themselves to be the only model you use, [21:18] but some level of orchestrating between [21:20] them could work really well. I know a [21:22] lot of people were previously using GLM [21:24] as the like workhorse model and had [21:26] something like Miniax commanding and [21:28] controlling it. Hum 2.5 shook things up [21:30] a bit and I'm sure GLM 5 is going to [21:32] even more so. Its placement on the cost [21:34] versus intelligence is really really [21:37] solid. It is a little more expensive [21:38] than Kimmy because it's freaking huge. [21:40] Cost might go down, but hard to know for [21:43] sure, but its performance is a [21:44] meaningful improvement. It doesn't quite [21:46] hit this green range, but it's not far [21:48] off either. And it just kind of makes [21:50] other models feel way more expensive. [21:52] Remember, this bottom bar is log scale. [21:55] So something that is 50% further on [21:57] isn't twice as bad. It's 20x sometimes. [22:00] It's it's brutal. It also uses fewer [22:02] tokens to run the benchmark than before, [22:05] even though it is performing much [22:06] better. That is cool to see because GLM [22:08] 4.7 was definitely token hungry for [22:10] these reasoning benchmarks. And that [22:12] seems to have been mostly addressed with [22:14] this release. If we go look at the [22:15] actual cost to run, you'll see it's way [22:18] lower than the leading models. GLM5 cost [22:22] $500 to run this benchmark, whereas Opus [22:24] 4.6 cost $1,500. 3x cheaper. 5.2 high [22:29] was $2,300. And Opus 4.6 in the adaptive [22:32] reasoning where it can reason as much as [22:34] it decides it wants to. 2500 and codecs [22:37] on X high $3,200. So between 3 to 6x [22:42] cheaper for intelligence that is not as [22:45] good, but meaningfully close enough. [22:47] Let's see how the designs came out. The [22:49] fact that you can get design this good [22:51] out of an open weight model is kind of [22:53] freaking nuts. What? Obviously K2.5 can [22:56] do well, but like that is a really good [22:58] starting point. Let's check the other [22:59] ones. Ah, they all do this font [23:01] clipping. This I've seen so many of the [23:04] models screw this up. It's pretty, but [23:07] uh has problems. Three. I like where [23:10] they're going here, but this side is [23:12] just not done quite properly. Again, I [23:15] like where they're going with it, but I [23:17] have some issues with like the backdrop [23:18] and things. But this would be a [23:20] reasonable starting point. Like, if you [23:22] don't want to spend the money on the [23:23] more expensive models, this is fine. [23:26] Obviously, one of the things that sucks [23:28] about this is if you have a problem, you [23:29] can't just screenshot the thing you [23:30] don't like and say, "Hey, fix this." [23:32] Because it doesn't have the ability to [23:33] look at images, but like these are [23:36] reasonable starting points. None of this [23:38] is like as good as what I expect from [23:40] Opus 46 with the design scale, but it's [23:42] honestly better than what I get from GPT [23:44] models. And I'm still so blown away with [23:46] the hallucination rate thing. That's so [23:48] cool to see. And if they can get the [23:50] cost down a tiny bit with optimizations [23:52] in the hosting, they will quickly end up [23:54] in this range and be by far the smartest [23:56] model at a reasonable price. Like pretty [23:59] much everything to the right of this [24:00] model is not as smart. Other than Opus [24:04] 46 in adaptive mode and Codeex in extra [24:06] high, pretty much everything else is [24:08] dumber, which is kind of crazy that it's [24:11] only halfway on the cost index and it is [24:13] near the top on the performance. If [24:15] you've ever doubted that intelligence [24:17] would be commoditized, hopefully the [24:19] recent releases of K2.5 and GLM5 are [24:21] swaying you the other way. The speed at [24:23] which these labs are catching up with [24:25] Frontier is insane. Especially when you [24:27] consider that at the same time labs like [24:29] Google and XAI are not keeping up. Like [24:32] where even is Gro 4 now? Oh, it's all [24:35] the way down here 41 and 39 [24:38] respectively. I'll never understand the [24:40] Grock fanboys, but especially now all [24:41] their researchers left, things are a [24:43] little scary. They're getting crushed. [24:45] Like everyone's being crushed by the [24:46] main two labs here and the handful of [24:48] really talented ones in China. I really [24:50] don't think that at the end of the year [24:52] anyone other than Anthropic, OpenAI, [24:54] ZAI, Deep Seek, Miniax, or Moonshot. [24:57] Like those are the only labs that matter [24:59] right now. It feels like things could [25:01] change in the future, but no one's [25:02] moving as fast as they are. I suspect [25:04] the future is going to look very similar [25:07] to what we see here. The numbers might [25:08] be bigger, but the order I would suspect [25:10] to stay relatively similar. But it is [25:13] really cool that one of the top four [25:15] models right now is an openweight one. [25:17] I'm excited. And by the way, that [25:20] migration I was doing for the TRPC thing [25:22] is still running. It can kind of just [25:24] go. And all of this migration I've been [25:27] doing here, if I was charged proper API [25:29] rates, would have cost about $10. I [25:32] almost paid an engineer a few,000 to do [25:35] this migration. And it will just grind [25:37] it out in the background. This would [25:38] have cost 100 plus with the [25:39] best-in-class close weight models. That [25:41] is insane. I think I'm going to use this [25:43] model for a lot of stuff. Congrats to [25:45] Zai for the incredible ship. Love the [25:47] focus on engineering and long running [25:50] task work here. It is far ahead of [25:52] everything else. I don't think I would [25:54] like talking to it a whole lot. And the [25:55] lack of multimodal support for images is [25:57] a little bit rough, but other than that, [25:59] feels like we're kind of there. It's [26:00] wild seeing how fast the openweight [26:02] world is moving, and I couldn't be [26:03] happier with this release. I am so so [26:05] pumped. While it isn't going to replace [26:07] my daily driver of codeex 5.3 just yet, [26:10] it is so surprisingly close that I can [26:12] easily recommend this to anybody trying [26:13] to save money or run models that they [26:15] can host themselves. Curious how y'all [26:16] feel though. Are you as excited as I am [26:18] about this release, or are you just [26:19] tired of all the new releases? Let me [26:20] know how you feel. And until next time, [26:22] peace nerds.