Elon Promised This One Would Be Good...  🔗 **Source:** [Elon Promised This One Would Be Good...](https://www.youtube.com/watch?v=jLgpzgpsWPc) > [!note]+ Document Notes > tags: [grok-4-7, token-efficiency, ai-model-benchmarks, inference-cost, vendor-claims, coding-agents] > author: Theo Browne > source: Theo - t3.gg (YouTube) > reference: https://www.youtube.com/watch?v=jLgpzgpsWPc > date created: 2026-09-23 10:10 > > > [!info] Generated analysis — not the author's words. Sections below are machine-written > > from the captured source. Only text inside `> [!quote]` blocks is verbatim from the author. > > Do not quote or attribute any other line to the named author. > > # 🏷️ Title: Elon promised this one would be good... > <!-- deep-read:sections start --> ## 🧭 Table of contents - [📝 Summary](#-summary) - [📖 Main content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning styles assessment](#-learning-styles-assessment) - [📊 Accessibility assessment](#-accessibility-assessment) - [🌍 DEIB assessment](#-deib-assessment) - [✅ Logical validity](#-logical-validity) - [🛡️ DARVO analysis](#️-darvo-analysis) - [🎯 Conclusion](#-conclusion) - [🔄 Refactor appendix](#-refactor-appendix--elon-promised-this-one-would-be-good) ## 📝 Summary - **What happened.** Theo Browne (Theo - t3.gg) published a 27:53 review of Grok 4.7 on 2026-09-22. The auto-transcript renders the model name as "Gro," "Grock," and "Rock" and the developer as "SpaceX" / "SpaceXi" throughout; every quote below preserves that spelling. The video has no chapters. Blacksmith sponsors it (00:00:42 to 00:02:53). - **The claim.** Elon Musk said Grok 4.7 would be a new 2.1-trillion-parameter base that's slower to serve but more token-efficient than Grok 4.6 (00:03:36). Theo says the shipped model uses about 2x the tokens of Grok 4.6 on Artificial Analysis (38k vs 81k per task) and in his own runs, which pushes its cost per task above GPT-6 Astra ($3.26 vs $3.74, AA figures at 00:15:01). The description states the efficiency loss as "30 to 80%"; that range never appears in the narration. - **Why it matters.** Theo's central argument is methodological: a per-request token comparison (Cursor CEO Michael Truell's "5% more at median, 20 to 30% more at P99," 00:04:19) can be true while per-prompt cost doubles, because agentic prompts fan out into many requests (00:06:28). Anyone reading vendor token-efficiency claims for coding agents needs that distinction. - **Key stakeholders.** xAI (the transcript says SpaceX, and states Cursor and SpaceX are now one company at 00:04:19), Michael Truell, Elon Musk, Artificial Analysis, Cognition (Frontier Code bench), Cursor and Grok Build subscribers, and Ben Davis, Theo's podcast co-host and self-described Grok user (00:15:43). - **Risks and tradeoffs.** Theo mixes three evidence tiers: third-party benchmarks (Artificial Analysis, Cognition), the vendor's own announcement post, and two of his own ad-hoc benches (a T3 Code codebase-improvement task judged by Fable 5.1 and Astra, and a 3D "Fish Slop" game build). He says outright that his bench is hard to formalize and that judging needs an LLM (00:21:33). His anger at Truell is stated in personal terms ("Either I'm right and this is misleading or I'm wrong and you're lying," 00:07:11). - **Bottom line.** Theo likes the model's behavior (higher floor, stays on task, digs deeper, 00:21:33) and calls the release "a bit of a flop" (00:25:52) on economics: roughly 2x the tokens, subscription limits that burn 8% of a weekly allowance for $40 of usage (00:16:28), and "last gen performance in various tasks at a current generation price" (00:27:18). ## 📖 Main content **The thesis.** Theo opens by saying benchmarks "no longer represent reality" (00:00:00) and that this release in particular has produced a flood of numbers he considers useless (00:00:42). He states early that Grok 4.7 is one of his favorite models of the year, which frames everything that follows as a complaint about the vendor's messaging rather than the model's behavior. **Elon's promise versus the announcement.** Per Theo's reading of Musk's earlier posts (00:03:36): Grok 4.6 would be a 1.5-trillion-parameter model in early August; Grok 4.7 would be a new 2.1-trillion base, better in every way, slightly slower to serve, with better token efficiency. The official post (read aloud at 00:02:53) says "same price and speed as Grok 4.6." Theo first assumes the new base failed and xAI kept RL-training the 1.5T base (00:05:43), then corrects himself on camera: the post claims the larger base, so he concludes the efficiency and speed parts of the promise are the false ones. The transcript's "PAM model" and "per model" are auto-caption garbles of "parameter model." **The requests-versus-prompts argument.** This is the video's load-bearing idea. Truell's rebuttal to Theo's cost post said production usage at SpaceX and Cursor shows 4.7 using 5% more tokens than 4.6 for median requests and 20 to 30% more at P99, plus a shift in what users ask (00:04:19). Theo's response (00:06:28): a request is one API call; a prompt that triggers tool calls generates many requests. If tokens per request rise 20 to 30% and requests per prompt also rise, per-prompt cost compounds. Artificial Analysis is weighting agentic workloads more heavily (00:07:11), so its 2x cost result is consistent with Truell's per-request numbers. Theo's framing is that both can be true and Truell's is the misleading one. **Cost numbers and their provenance.** Every figure below is tagged by source. | Figure | Source | Where | |---|---|---| | Grok 4.6 XHigh ~38k tokens/task; Grok 4.7 ~81k; Soul ~29k; Astra similar to Soul | Artificial Analysis, read by Theo | 00:03:36 | | Grok 4.6 $1.86, Astra $3.26, Grok 4.7 $3.74 per task (transcript renders "$186" and "326") | Artificial Analysis | 00:04:19, 00:15:01 | | 5% more tokens at median, 20 to 30% at P99 | Michael Truell, production data at SpaceX and Cursor | 00:04:19 | | Worst Grok 4.7 run $20 at API prices; worst Astra run $11.44; every Opus 5 run under a quarter of that | Theo's own ad-hoc bench | 00:07:54 | | Grok Build 56 vs Codex and Claude Code 62 on the coding agent index | Artificial Analysis | 00:13:36 | | $40 of usage burned 8% of a weekly limit on the $300 plan; other labs' $200 plans yield "8 to 12,000 a month" | Ben Davis's account, relayed by Theo | 00:16:28 | | Price doubles above 200,000 tokens; fast mode costs more | Theo and Ben's reading of pricing | 00:15:43 | | AA Briefcase: third behind Opus 5 Max and Fable 5.1, at about half the price; presentation ELO below Grok 4.6 | Artificial Analysis public note | 00:17:10 | | Terminal Bench: Fable 5.1 about 20 points higher, "almost a 50% improvement" | xAI announcement post as read by Theo | 00:10:46 | | Grok 4.7 best case costs about the same as Astra and Fable; worst case 2x | Theo's T3 Code bench | 00:23:42 | | Fish Slop build took almost 1.5 hours | Theo's own run | 00:24:26 | | Blacksmith: 50%+ faster, 60% less spend; GitHub CI would cost "tens of thousands, if not a hundred thousand plus dollars a month" | Sponsor claims | 00:01:26 | **The announcement post's benchmark set.** Theo walks the official post (00:08:36 to 00:11:29): Cursor Bench (past contamination he says is now fixed; no OpenAI models because OpenAI banned xAI from using them), a software-engineering bench the transcript renders as "Deep Sweet V1.1" (likely SWE-bench-family; verify), electrical engineering, a briefcase bench where xAI claims second in the world, Terminal Bench, legal work, and HealthBench. He calls it "the weirdest set of benches I've ever seen in any model release" and says the vendor "cherrypicked the hell out of this" (00:10:46), while granting that benchmarks don't capture what he likes about the model. He notes GDPval in the same post showed Grok 4.6 beating GPT-6 Astra, which he treats as evidence the bench is noise (00:11:29). **Third-party indexes.** Cognition's Frontier Code (00:11:29): Theo calls it noisy across reasoning levels, but flags that every Grok 4.7 score sits below Grok 4.6's best; the vendor's own explanation is that 4.7 "tends to overcope on others" (likely "over-scope"), trailing in aggregate because the bench scores mergeability, not just correctness (00:12:53). Artificial Analysis general index: Grok 4.7 barely above GLM 5.3 Max and below Muse Spark 1.3 (00:12:53); Theo notes AA recently reweighted toward agentic benches after Astra debuted too low (00:13:36). On AA's coding agent index, Grok Build scores 56 against 62 for Codex and Claude Code. **Why six points matter.** Theo's argument at 00:14:18: a small index gap maps to how often a model does something stupid inside a time window; lowering the error rate extends the unattended window more than it raises task difficulty. A weaker model handles most work if you spec and hand-hold it, as GPT-4.1 could 18 months earlier; the cost is upfront effort. **Behaviors benchmarks miss.** Front-end output: Theo reacts on camera to Grok 4.7 designs he calls among the worst he's seen from any model, worse than his memory of GPT-5 (00:18:35). Anthropic remains "the king of design" in his view. A runaway loop from a Grok Build user ("I'm the machine. I trust it into infinity," 00:19:20 to 00:20:05) and a leaked markdown dump when he clicked "show translation" on an X post (00:20:50) show the model emitting reasoning-trace-style output through standard API surfaces. Musk's own post (00:18:35) says the team "may have penalized response length too much" in RL and spent a week making the model more willing to run long and verify; Theo ties that fix directly to the vanished token efficiency (00:19:20). **Theo's own benches.** The T3 Code bench (00:21:33 to 00:23:42): several models read the T3 Code codebase and propose improvements; Fable 5.1 and GPT-6 Astra judge viability. Results: Astra first, Grok 4.7 second, Fable 5.1 third with 5 suggestions versus 8 from the others but all valid. Theo notes Fable, as half the judging panel, scored itself below Grok. He also admits he used Gemini 3.7 Flash instead of 3.8 by mistake. Fish Slop (00:24:26 to 00:25:10): the same prompt and environment given to Astra earlier; Grok 4.7's first run animated fish and submarine backward, mouse tracking and cursor position are wrong, and the build took about 90 minutes. **Verdict.** "A stepping stone" (00:26:34). More pleasant to use and better in some real-world work, but 2x tokens, aggressive subscription limits, slower, higher error rates, and worse code overall than the Fable and Astra tier. He recommends it mainly for people locked into Cursor as their inference source (00:26:34). **Contested or unverifiable in this source.** The description's "30 to 80%" efficiency loss has no matching statement in the narration; the numbers Theo cites are 2x (AA) and 2x worst case (his bench). Theo's T3 Code bench has no published methodology, uses LLM judges including one of the contestants, and he says so. The per-task dollar figures are shown on screen; the transcript renders "$186" and "326," which this analysis reads as $1.86 and $3.26 based on the later "$3.26 ... 374" phrasing at 00:15:01. Ben Davis's 8%-per-$40 figure is a single user's anecdote. ## 🗒️ Notes #### ⏱️ [00:00:00](https://www.youtube.com/watch?v=jLgpzgpsWPc&t=0) Hype setup and the benchmark thesis - Musk hyped Grok 4.7 for over a month, saying at the 4.5 launch that 4.7 would exceed all current models and that SpaceX's training corpus would make it best at real-world coding. Theo notes it scores below Grok 4.6 on several benches. - Stated purpose of the video: benchmarks no longer describe reality, and the volume of useless numbers around this release is frustrating. He also says it's one of his favorite models this year. > [!quote] [00:00:00] > But the thing I want to talk about here is how the benchmarks no longer represent reality. #### ⏱️ [00:00:42](https://www.youtube.com/watch?v=jLgpzgpsWPc&t=42) Sponsor: Blacksmith - Pitch: GitHub Actions runners are slow, expensive, and on old hardware; Blacksmith replaces the runners with a one-line workflow change. Claimed 50%+ faster builds and 60% less spend. Theo says his team runs hundreds of PRs a day with 8 to 30 checks each (00:02:09). - Codesmith, Blacksmith's built-in agent, produced three PRs Theo merged from example prompts (00:02:53). All figures here are sponsor claims. #### ⏱️ [00:02:53](https://www.youtube.com/watch?v=jLgpzgpsWPc&t=173) The announcement post versus Elon's promise - Official post language: most capable model for coding and knowledge work, works longer, checks its own work, best calibrated safeguards, same price and speed as Grok 4.6. - Musk's earlier posts: 4.6 as a 1.5T model in early August, 4.7 as a new 2.1T base, slightly slower to serve but more token-efficient (00:03:36). Theo: "This was a lie." - Artificial Analysis token counts per task (third-party, read by Theo): Grok 4.6 XHigh ~38k, Grok 4.7 ~81k, above Fable 5.1; Soul ~29k, Astra about the same. > [!quote] [00:03:36] > Gro 46 XHigh, according to artificial analysis, did around 38k tokens per task. Gro 47 it did 81k tokens putting it above even Fable 5.1. #### ⏱️ [00:04:19](https://www.youtube.com/watch?v=jLgpzgpsWPc&t=259) Michael Truell's rebuttal and the requests-versus-prompts argument - Theo's post showed AA cost per task: Grok 4.6 $1.86, Astra $3.26, Grok 4.7 $3.74 (transcript: "$186," "326," "$3.74"). Truell, Cursor's CEO now working with xAI after the Cursor and SpaceX combination, replied that production shows 5% more tokens at median and 20 to 30% at P99, plus a distribution shift in requests. - Theo's steelman (00:05:01 to 00:05:43): maybe the new base underperformed and xAI kept RL-training the 1.5T base, explaining same cost, speed, and behavior. He then reads the post claiming the larger base and drops that theory. - The core rebuttal (00:06:28): a request is one API call; a single prompt with tool calls fans out into many requests. Per-request token growth of 20 to 30% understates per-prompt cost if request count also rises, and AA is leaning into agentic workloads (00:07:11). > [!quote] [00:06:28] > request is not an accurate way to measure token usage because one prompt can result in a very wide varying number of requests being made. > [!quote] [00:07:11] > Either I'm right and this is misleading or I'm wrong and you're lying. And I would like for you to pick which it is. #### ⏱️ [00:07:54](https://www.youtube.com/watch?v=jLgpzgpsWPc&t=474) Theo's own cost data and the Opus rumor - From his ad-hoc bench (own data): worst Grok 4.7 run would cost $20 at API prices; no other model breaks $10 except one Astra run at $11.44; every Opus 5 run under a quarter of the Grok worst case. - He expects a new Opus (5.1, 5.2, or 5.5) "later this week according to the rumors," priced similarly to Grok 4.7 but benching far better, which would make 4.7 harder to justify. Rumor, not sourced. #### ⏱️ [00:08:36](https://www.youtube.com/watch?v=jLgpzgpsWPc&t=516) Walking the official benchmark set - Cursor Bench: past contamination from training data, since repaired. No OpenAI models because OpenAI banned xAI from using them; Theo offers to run the bench as a third party. Grok 4.7 beats GPT-5.6 Soul for the price at most tiers, but Grok XHigh scores about the same as Fable low while Fable low is cheaper (00:09:21). In token view, Grok is only efficient at its lowest reasoning level. - Trained to natively understand the Grokbot harness (transcript: "Grockbot"); Theo says everyone he knows using it is obsessed and offers a deep dive. - Other benches (00:10:03 to 00:11:29): "Deep Sweet V1.1" software engineering (likely a SWE-bench variant; garbled), electrical engineering, a briefcase bench (second in the world per xAI), Terminal Bench (Fable 5.1 about 20 points higher), legal work, HealthBench. GDPval shows Grok 4.6 beating GPT-6 Astra, which Theo calls nonsense. > [!quote] [00:10:46] > You can tell they cherrypicked the hell out of this. #### ⏱️ [00:11:29](https://www.youtube.com/watch?v=jLgpzgpsWPc&t=689) Cognition Frontier Code and the Artificial Analysis indexes - Cognition Frontier Code: Theo calls it noisy (scores jump between reasoning levels; Fable 5.1 benches below Fable 5 at high reasoning; Astra dips at XHigh and spikes at max). Still, every 4.7 score sits below 4.6's best. xAI's explanation: 4.7 "tends to overcope" (likely over-scope) on some tasks and the bench measures mergeability (00:12:53). - AA general index: Grok 4.7 barely above GLM 5.3 Max and below Muse Spark 1.3. Theo notes AA reweighted toward agentic benches a few weeks earlier after Astra debuted too low; Astra and Fable 5.1 now tie, which he thinks is fair (00:13:36). - AA coding agent index: Grok Build 56 versus 62 for Codex and Claude Code; beats Gemini 3.8 Flash and barely beats Muse Code and Muse Spark 1.3. > [!quote] [00:12:53] > Not only did the Gro 47 just barely score above GLM53 Max, it actually scored below Muse Spark 1.3. #### ⏱️ [00:14:18](https://www.youtube.com/watch?v=jLgpzgpsWPc&t=858) Why six points matter, and the real cost picture - Six index points indicate how often the model does something stupid in a time window; lowering error rates extends the unattended window more than it raises task difficulty. GPT-4.1 handled most work 18 months ago with hand-holding; Grok 4.7 needs more upfront effort. - Cost (00:15:01): Theo rejects the "caching bug" speculation and says the model uses 2x the tokens of prior Grok models, making it cost more than Astra ($3.26 vs $3.74) though still under Opus and Fable. - Ben Davis (podcast co-host, Grok Code Fast fan, Grokbot user) reports aggressive pricing: doubling above 200k tokens, more on fast mode; $40 of usage burned 8% of his weekly limit on the $300 plan, versus "8 to 12,000 a month" on other labs' $200 plans (00:16:28). Single-user anecdote. > [!quote] [00:14:18] > those six points indicate a gap in how likely the model is to do something stupid in a given time window. > [!quote] [00:16:28] > He did $40 of usage and it burned 8% of his weekly limit. #### ⏱️ [00:17:10](https://www.youtube.com/watch?v=jLgpzgpsWPc&t=1030) Where the model does win, and the front-end disaster - Grok 4.7 is not a premium-tier model like Fable or Astra; it sits with Soul and Opus but now with heavy usage limits and lots of competition (00:17:10). - AA Briefcase (public AA note): Grok 4.7 jumps to third behind Opus 5 Max and Fable 5.1, ahead of every OpenAI model, at about half the price. AA also records a presentation-ELO regression versus Grok 4.6. - Front-end designs shown on screen (00:17:53 to 00:18:35): Theo's live reaction is that they're among the worst he's seen from any model, worse than his memory of GPT-5. Anthropic remains "the king of design"; Astra is now steerable and decent. > [!quote] [00:18:35] > I remember GPT5 being better than this at front end. This is this is insulting. #### ⏱️ [00:18:35](https://www.youtube.com/watch?v=jLgpzgpsWPc&t=1115) Elon's RL explanation, the infinite loop, and the X translation leak - Musk's post on the week's delay: RL may have penalized response length too much, so the model gave up early on hard tasks and didn't check its work; the team spent a week fixing it. Theo ties this to the lost token efficiency (00:19:20). - A Grok Build user's transcript shows a loop of self-affirmation ("I'm the unlimited giga intellect ... I'm Bellard. I'm ready. I'm in.") repeating forever (00:19:20 to 00:20:05). - Clicking "show translation" on a Spanish K-pop post tagging Theo produced a markdown dump that looked like translation instructions, on X.com itself (00:20:50). Theo's read: the model's oddities leak past reasoning traces into standard API surfaces. > [!quote] [00:19:20] > This is likely where that token efficiency disappeared because they were bragging before about it and now they can't. #### ⏱️ [00:21:33](https://www.youtube.com/watch?v=jLgpzgpsWPc&t=1293) What Theo likes, and the T3 Code improvement bench - Likes: does what it's told without intervention, higher floor than its tier, stays on task, and is "inquisitive," finding things even Fable and Astra missed. Still below Fable and Astra at the extremes. - The bench (own, ad hoc, admittedly hard to formalize): models read the T3 Code codebase and propose improvements; Fable 5.1 and Astra judge. Astra first (most novel, best vetted), Grok 4.7 close second, Fable 5.1 third with 5 suggestions versus 8 but all valid. Fable, as a judge, scored itself below Grok. He mistakenly used Gemini 3.7 Flash instead of 3.8 (00:23:00). > [!quote] [00:21:33] > It stays on task longer and can complete real work. > [!quote] [00:23:00] > So, even though Fable was half the judging work, it actually scored itself below Grock. #### ⏱️ [00:23:42](https://www.youtube.com/watch?v=jLgpzgpsWPc&t=1422) Cost of his runs and Fish Slop - Best case, Grok 4.7 costs about the same as Astra and Fable; worst case 2x, because it verifies things that don't matter. The capability gain doesn't match the cost jump; this continues a trend from 4.6 of spending tokens to squeeze capability. Grok 4.5 was the efficient, cheap, fast one. - Fish Slop (same prompt and environment as an earlier Astra run): about 90 minutes to build; first run animated fish and submarine backward; mouse tracking speed and cursor location wrong; Theo warns motion-sensitive viewers. Astra's version "actually looks decent" (00:25:10). > [!quote] [00:24:26] > For example, in Fish Slop, it took almost an hour and a half to build it. #### ⏱️ [00:25:52](https://www.youtube.com/watch?v=jLgpzgpsWPc&t=1552) Verdict and sign-off - Theo wanted a model that doesn't burn Cursor usage so he could use Cursor's cloud features, and hoped the Cursor acquisition would show up in the model. Verdict: "a bit of a flop." Cost increase doesn't justify switching; subscription subsidy is too weak; more expensive, slower, higher error rates, worse code than the frontier tier, and nowhere near the Codex and Claude subsidies. - "A stepping stone" (00:26:34): Fable and Astra were a generational leap, GPT-5.6 Soul close, Grok 4.7 "still last gen" in his words. Worth playing with, especially if stuck on Cursor for inference; hard to recommend otherwise. - Close (00:27:18): last-gen performance at current-gen price. Asks viewers whether he's too harsh. Final gag: no chance of a Musk repost this time. > [!quote] [00:25:52] > I think this model is a bit of a flop. > [!quote] [00:27:18] > we're getting last gen performance in various tasks at a current generation price, which just is not the right compromise at all. ## 📚 References The transcript itself contains no links; the following come from the video description or are primary sources for named entities. - [Elon promised this one would be good...: Theo - t3.gg](https://www.youtube.com/watch?v=jLgpzgpsWPc): the source video, 27:53, uploaded 2026-09-22. - [Blacksmith (sponsor link)](https://soydev.link/blacksmith): the transcript renders it "soyv.link/blacksmith" at 00:02:53; the description gives the correct URL. - [t3.gg](https://t3.gg): Theo's site, linked in the description for Twitch, Twitter, and Discord. - [Artificial Analysis](https://artificialanalysis.ai): source of the tokens-per-task, cost-per-task, general index, coding agent index, and AA Briefcase figures Theo reads on screen. - [Cognition](https://cognition.ai): publisher of the Frontier Code bench discussed at 00:11:29. Theo names the company; the bench page isn't linked in the description. - [Cursor](https://cursor.com): Cursor Bench and Michael Truell, whom the transcript renders as "Michael Truel." - xAI / Grok: the transcript says "SpaceX" and "SpaceXi" for the developer; the official announcement post is read aloud at 00:02:53 but not linked in the description. Find it on x.ai before citing any benchmark number from the post. - Ben Davis and the podcast Theo co-hosts: the transcript renders the show name as "Nurse" and "Nerdsip" (00:15:43); confirm the real name before linking. ## 🗣️ Paraphrase Theo's argument is that xAI shipped a model he enjoys using and then described it in ways the numbers contradict. Musk promised a bigger base with better token efficiency; Artificial Analysis measures Grok 4.7 at about double Grok 4.6's tokens per task, and Theo's own runs agree. When Cursor's CEO pushed back with production data showing only 5% more tokens per median request, Theo's answer is that "request" is the wrong unit: an agentic prompt fans out into many requests, so a modest per-request increase compounds into a doubled per-prompt bill. He asks Truell to say which of them is misleading. The second half sorts what benchmarks catch from what they miss. They catch the cost, the small index gap that predicts how long a model runs unattended before doing something dumb, and the odd choice of benches in the announcement. They miss the front-end output, which Theo calls the worst he's seen from a current model, the runaway self-affirmation loops, and the model's upside: a higher floor and an inquisitive habit of finding things Fable and Astra miss. His verdict is that this is a stepping stone priced like a destination, worth using only if Cursor is already your inference source. ## 📰 Newsletter **Count prompts, not requests.** Theo Browne's 28-minute Grok 4.7 review is worth watching for one argument that outlives the model. xAI's side says the new model uses 5% more tokens per median request. Artificial Analysis says it uses about 2x the tokens per task. Theo's point: both can be true, because an agentic coding prompt turns into dozens of API requests, and the number that hits your invoice is the per-prompt total. If you're evaluating any coding agent for WordPress work, that's the unit to ask vendors about. The rest is a useful case study in reading a launch post. The benchmark set (legal work, clinical reasoning, electrical engineering) doesn't match the "best for coding" headline; the strongest third-party result (third on AA Briefcase at half the price) sits next to a front-end showcase Theo reacts to in real time with visible pain. Theo is open about the limits of his own evidence: his codebase-improvement bench is judged by two of the contestants, and he says it's hard to formalize. Practical takeaway for anyone on a $200 or $300 coding-agent plan: subscription limits, not list price, decide what a model costs you. One user's $40 of usage ate 8% of a weekly allowance. Links: [the video](https://www.youtube.com/watch?v=jLgpzgpsWPc), [Artificial Analysis](https://artificialanalysis.ai), [Blacksmith](https://soydev.link/blacksmith) (the sponsor). ## ✅ Action items #### DevRel - Developer advocacy: when demoing an agent, publish per-prompt token totals and request counts side by side, so the requests-versus-prompts confusion at 00:06:28 can't happen to your numbers. - Education: one office-hours segment on "what unit is this benchmark in" using the 5% versus 2x contradiction as the worked example. - Developer experience: add a cost-per-completed-task line to any agent quickstart that reports token usage. - Developer marketing: never quote a model-efficiency percentage without naming the benchmark, the unit (request, prompt, task), and who measured it. This video is the cautionary tale. - Community management: when a launch post circulates in Slack or Discord, attach the third-party index alongside it before discussion starts. - Internal/external/cross-project: ask any vendor pitching "token efficiency" for their P50 and P99 per-prompt distributions, not per-request. #### OSPO - Policy & Governance: write down which model-cost unit the org standardizes on (cost per completed task) so procurement comparisons are like-for-like. - License Compliance: no direct action from this source; nearest relevance is the OpenAI-bans-xAI detail at 00:08:36, a reminder that model ToS can block benchmarking. - Security Management: the X translation leak at 00:20:50 and the runaway loop at 00:19:20 are output-integrity incidents. Add "model emits internal-instruction text through a production endpoint" to the incident taxonomy. - Software Inventory: track which model version and reasoning tier each internal tool runs, since Theo shows scores swing by tier (00:09:21, 00:12:10). - Developer Support: publish a one-page guide on reading Artificial Analysis and similar indexes, including what a 6-point gap means in practice (00:14:18). - Community Engagement: no direct action from this source; nearest relevance is Theo's offer to run Cursor Bench as a neutral third party, a model for community-run evals. - Strategic Alignment: set a KPI for inference spend per merged PR so a model swap is measurable. #### Community - Maintainers: before switching a project's default agent model, run the same 10 real issues through old and new and compare cost per merged fix, as Theo does at 00:23:42. - Contributors: document which reasoning tier the project recommends, since the cheapest tier is the only one where Grok 4.7 is token-efficient (00:09:21). - Sponsors: if a vendor funds tooling, require per-prompt cost disclosure in the report, not per-request. - Ecosystem: no direct action from this source; nearest relevance is a shared, reproducible "improve this codebase" bench that multiple projects could run. #### WordPress - Core: no direct action from this source; nearest relevance is Theo's note that mergeability, not just correctness, is what Cognition's bench scores (00:12:53). Useful framing for AI-assisted patch review. - Core Performance: no direct action from this source; nearest relevance is the 90-minute Fish Slop build as a reminder to time agent runs, not just judge output. - Core AI: adopt cost-per-completed-task as the evaluation unit in any model pilot, and log request counts per prompt. - **Core Program Team:** when summarizing model releases for contributors, separate vendor claims, third-party indexes, and anecdotes explicitly, as the Main content table does here. - Mobile: no direct action from this source. - Playground: a browser demo that shows one prompt fanning out into N requests would teach the 00:06:28 argument in one page. - Community: no direct action from this source. - Training: lesson idea: "three tiers of evidence in a model review," built from this video's mix of AA data, vendor post, and ad-hoc bench. - Marketing: no direct action from this source; nearest relevance is the cherry-picked-benchmark critique at 00:10:46 as a what-not-to-do example. - Support: no direct action from this source. - Design: Theo's live reaction to Grok 4.7 front-end output (00:18:35) supports keeping human design review on any AI-generated UI. - Documentation: if docs mention model costs, state the unit and the measuring party every time. - Photos: no direct action from this source. - TV: caption accuracy matters; this video's auto-captions garble every model and company name (see Accessibility). - Polyglots: the X translation leak at 00:20:50 is a concrete example of an AI translation surface emitting non-translation text; worth noting for any machine-translation pilot. - Accessibility: no direct action from this source; nearest relevance is Theo's motion-sickness warning at 00:25:10, a good habit for any demo video. - Meta: no direct action from this source. - CLI: no direct action from this source. - Hosting: no direct action from this source; nearest relevance is the Blacksmith sponsor segment on CI runner cost, which applies to any host running PR checks. - Tide: no direct action from this source. - Openverse: no direct action from this source. - Themes: no direct action from this source. - Plugins: no direct action from this source. #### The WPOCC - Programs: no direct action from this source; if The WP Open Community Collective funds AI tooling stipends, per-task inference cost is the line item to require. - Projects: no direct action from this source. - Funding experiments: require before/after cost per merged fix in any AI-tooling grant report. ### FAIR - Technical independence: no direct action from this source; nearest relevance is the reminder that a vendor's ToS (OpenAI banning xAI, 00:08:36) can prevent independent comparison. Prefer models that allow third-party benchmarking. - Policies and covenants: any covenant that references model cost should name the unit. - Productized governance artifacts: add "who measured this" as a required field on any scorecard row that quotes a model metric. ## 🧪 Hypotheses 1. For agentic coding prompts with 5+ tool calls, Grok 4.7's total tokens per completed prompt exceed Grok 4.6's by more than 80%, even though per-request tokens differ by less than 30%. (Tests the 00:06:28 argument directly.) 2. Grok 4.7's request count per prompt is at least 1.5x Grok 4.6's on the same agentic task set, which is the missing variable in Truell's per-request framing. 3. On a fixed set of 20 real issues, Grok 4.7's cost per merged fix is within 20% of GPT-6 Astra's at best case and at least 1.8x at P95, matching Theo's "same at best, 2x at worst" (00:23:42). 4. A 6-point gap on the AA coding agent index corresponds to a measurable difference in mean time to first unrecoverable error on unattended runs, in the direction Theo claims at 00:14:18. 5. Grok 4.7's token usage at its lowest reasoning tier is within 10% of GPT-5.6 Soul at high, and diverges by more than 50% at every higher tier (00:09:21). 6. LLM-judged benches where a contestant is also a judge produce lower self-scores than independent judging, not higher, as the Fable-scores-itself-below-Grok result at 00:23:00 suggests. ## 📊 Methodology - **H1 and H2.** Take 50 agentic coding prompts from a real repo's issue tracker. Run each through Grok 4.6 and Grok 4.7 in the same harness (Cursor or Grok Build). Log every API request with input and output tokens. Report per-request P50 and P99 tokens (Truell's unit) and per-prompt totals and request counts (Theo's unit) in the same table. The ratio between the two views is the finding. - **H3.** Use 20 issues with known merged fixes. Run Grok 4.7, GPT-6 Astra, and Fable 5.1 to a PR. Score mergeability with two human reviewers blind to model. Divide invoiced spend by merged count. Report best case, median, and P95. - **H4.** Take three models 6 or more points apart on the AA coding agent index. Give each 30 unattended one-hour tasks. Record time to first error that a human must fix. Test whether the index gap predicts the time gap. - **H5.** Fix one prompt set. Run Grok 4.7 at every reasoning tier and Soul at high. Plot tokens per task by tier against Cursor Bench-style scores. The crossover tier is the finding. - **H6.** Rerun Theo's T3 Code bench design with three judge configurations: Fable only, Astra only, and a model that isn't a contestant. Compare each contestant's score across configurations. Self-judging bias shows as a systematic shift. - **Controls throughout.** Same harness, same system prompt, same repo state, three seeds. Publish request logs, not summaries. ## ❓ Questions - What exactly does Truell's "request" mean in Cursor's telemetry: a single model call, or a full user turn? The whole disagreement turns on that definition. - Does Artificial Analysis publish request counts per task alongside token counts? If so, the 38k-to-81k jump can be decomposed into per-request growth and request-count growth. - Which real benchmark is the transcript's "Deep Sweet V1.1"? SWE-bench Verified, a Deep SWE variant, or something else? - Is the "30 to 80%" efficiency loss in the description Theo's own estimate, and from which data? The narration only supports "about 2x." - Does the weekly usage limit on the $300 plan scale with reasoning tier, or is the 8%-per-$40 figure tier-independent? - What changed in the last week of RL (00:18:35) in measurable terms: mean response length, verification steps per task, or both? - Is the X "show translation" output at 00:20:50 a Grok 4.7 endpoint, and does xAI acknowledge the leak? - Would Grok 4.7 at its lowest reasoning tier, where token use matches Soul high (00:09:21), still beat Soul on Cursor Bench? ## 🔮 Future research - **Per-prompt cost accounting as a standard.** No public index reports request fan-out per task. A benchmark that publishes tokens per request, requests per prompt, and cost per completed task together would end arguments like the one at 00:06:28. - **Reasoning-tier economics.** Theo's observation that Grok 4.7 is only efficient at its lowest tier (00:09:21) suggests tier choice, not model choice, may dominate cost. Nobody has mapped cost versus quality across tiers for the current frontier set. - **Self-judging bias in LLM-judged benches.** The Fable-scores-itself-lower result at 00:23:00 is one data point against the assumption that models favor themselves. A controlled study across judge configurations is cheap and unpublished. - **Subscription subsidy as a market signal.** Theo argues that the subsidy on Codex and Claude plans, not API list price, drives adoption (00:26:34). Tracking effective cost per token on subscription plans across labs over time would test that. - **RL length penalties and efficiency.** Musk's post (00:18:35) attributes the model's early behavior to over-penalizing length and the fix to relaxing it. The tradeoff curve between length penalty, task completion, and token cost is a research question with a public natural experiment attached. ## 💡 Implications - **Practice.** Evaluate coding agents on cost per completed task, logged from your own harness, before trusting either vendor percentages or index dollar figures. Theo's own numbers agreed with AA (00:07:11) but only because he ran them. - **Policy.** Any internal standard that quotes model efficiency should require the unit and measuring party in the same sentence. The 5%-versus-2x dispute exists because neither side stated the unit up front. - **Security.** A model that emits its own operating instructions through a production translation button (00:20:50) or loops indefinitely in an agent harness (00:19:20) is a reliability and data-exposure risk. Treat model output surfaces as untrusted until observed under load. - **Education.** The video is a compact teaching case in three evidence tiers: vendor post, third-party index, and personal ad-hoc bench. Learners should be able to sort every number in it by tier, as the Main content table does. - **Funding models.** Theo's point that subscription subsidy sets the real price (00:16:28, 00:26:34) means organizations budgeting AI tooling should model plan limits, not per-token rates. A $300 plan that burns 8% per $40 has an effective ceiling around $500 of API-equivalent usage per week. - **Vendor literacy.** A launch post that benchmarks legal work and clinical reasoning for a model headlined as "most capable for coding" (00:10:46) is a pattern to recognize: benchmark breadth substituting for benchmark relevance. ## 🧠 Learning styles assessment - **Reading level estimate.** Flesch-Kincaid Grade Level on the transcript body: approximately grade 7 to 8. Flesch Reading Ease near 70. The sentences are short and conversational, but the vocabulary assumes familiarity with reasoning tiers, P99, RL, and named benchmarks, which pushes the effective audience to working developers who already follow model releases. #### 👀 Visual learners - Present: on-screen posts from Musk and Truell, Artificial Analysis charts, the xAI announcement post scrolled live, Cognition's Frontier Code chart, the T3 Code bench results table, and the Fish Slop game rendered live. - Improve: no diagram of the requests-versus-prompts argument, which is the video's central idea and the one that most needs a picture. A single sketch of one prompt fanning into N requests would carry it. Chart axes are never described aloud. #### 🔊 Aural learners - Present: continuous narration; Theo reads the key posts aloud, so the argument is followable without the screen. - Improve: the dollar figures are spoken ambiguously ("$186," "326"), and the model names blur together in audio (Astra, Aster, Astro). A spoken recap of the five key numbers at the end would help. #### 🗣️ Verbal learners - Present: the request-versus-prompt distinction is explained with a concrete example ("respond with hello" versus a tool-calling task, 00:06:28). "Steelman" is used and enacted. - Improve: P99, reasoning tier, XHigh, and RL are never defined. One clause each would cost 15 seconds. #### 🧩 Kinesthetic learners - Present: two hands-on benches (T3 Code improvements, Fish Slop) with results shown. - Improve: no repo link, no prompt text, no harness config. The Fish Slop prompt is described as identical to the Astra run but never shown. Publishing it would let viewers reproduce. #### 🧠 Logical learners - Present: the strongest section of the video. The 00:06:28 argument is a clean decomposition (per-request tokens × requests per prompt = per-prompt tokens), and the 00:14:18 argument links index points to unattended time windows. - Improve: the 2x claim would be stronger with request counts shown, not just token totals. Show the arithmetic once. #### 🧑‍🤝‍🧑 Social learners - Present: direct engagement with Truell's public reply, a co-host's counter-view (Ben Davis as "Grok defender"), a podcast pointer, and an explicit closing question to viewers. - Improve: no link to the specific posts being argued with, so viewers can't join the thread. #### 🧘‍♀️ Solitary learners - Present: self-contained; the argument doesn't require prior videos. - Improve: no reflection prompt. "Which unit does your own agent bill report in?" would turn the video into an exercise. #### 🌿 Naturalistic learners - Present: nothing explicit. Fish Slop's underwater theme is incidental. - Improve: 2x tokens per task is 2x compute per task. Naming the energy dimension of token inefficiency would land with this group and is a real, checkable claim. ## 📊 Accessibility assessment #### Screen reader semantics - Present: YouTube supplies landmarks and player controls; the auto-transcript is machine-readable. - Improve: charts from Artificial Analysis and Cognition, the pricing table, and the front-end screenshots carry information the narration never states. WCAG 2.2 AA 1.1.1 (Non-text Content) and 1.2.5 (Audio Description, Prerecorded) are unmet for those segments. #### Keyboard navigation - Present: the player is keyboard operable with visible focus. 2.1.1 and 2.4.7 pass via the platform. - Improve: no chapters, so keyboard users can't jump by topic; they scrub. Adding chapters at the 13 segment points above would fix 2.4.5-adjacent wayfinding. #### Contrast and readability - Present: unverifiable from the transcript. The screen-recorded posts and charts are likely readable at desktop size. - Improve: dense chart text at default YouTube resolution risks 1.4.3 (Contrast Minimum) on mobile. Zoom into the specific numbers being read. #### Text resizing and reflow - Present: the player scales; captions resize independently. - Improve: on-screen post text and chart labels are burned in and don't reflow. 1.4.4 (Resize Text) and 1.4.10 (Reflow) fail for that content. #### Media and captions - Present: an auto-generated transcript exists and is how this analysis was produced. - Improve: the auto-captions garble the model name ("Gro," "Grock," "Rock"), the developer ("SpaceXi"), the Cursor CEO ("Truel"), the sponsor URL ("soyv.link"), the podcast name, at least one benchmark ("Deep Sweet"), and the dollar figures ("$186"). Captions exist but are wrong on nearly every proper noun and several numbers, which is a 1.2.2 (Captions, Prerecorded) failure in substance. Human-corrected captions for 28 minutes are the highest-value fix. - Motion: Theo warns motion-sensitive viewers before the Fish Slop footage (00:25:10). Good practice; a description-level warning would extend it to viewers who skip ahead. #### Forms and interactions - Present: none in the video. The sponsor URL is spoken and shown. - Improve: the spoken URL is wrong in the captions; the description has the correct one. Not a WCAG in-video criterion, but 3.3.2 (Labels or Instructions) applies to the landing page. #### Standards reference - Primary gaps against WCAG 2.2 AA: **1.2.2** (inaccurate captions on names and numbers), **1.2.5** (no audio description for charts and screenshots), **1.1.1** (chart data visual only), **1.4.4 / 1.4.10** (burned-in text doesn't reflow). The captions failure is highest impact and cheapest to fix. ## 🌍 DEIB assessment #### Cultural breadth - Present: the K-pop tangent (00:20:05) is the only non-US cultural reference, and it's incidental. Examples are US companies and a US-centric model-release news cycle. - Improve: pricing in USD only; a note on how the $300 plan compares in other markets would widen relevance. #### Inclusive language - Present: no exclusionary terms. Second person and contractions throughout. - Improve: "stupid" (00:14:18), "insulting," "meme tier bad" (00:25:10), and "the hell out of" (00:10:46) are aimed at software, not people, but the register is combative. "Lying" directed at a named person (00:07:11) is the one place the language targets an individual. #### Neurodiversity support - Present: the argument has a spine (promise, contradiction, rebuttal, evidence, verdict) that's easy to follow. - Improve: no chapters in a 28-minute video; tangents (K-pop, Opus rumors, podcast plug) interrupt the thread; and the Fish Slop footage is explicitly motion-heavy. Chapters and a "skip to verdict" marker would help. #### Representation - Present: Theo, Musk, Truell, and Ben Davis are the only voices. All male. - Improve: no independent evaluator is quoted directly. Artificial Analysis is cited as data, not as a person. A short clip or quote from an independent bench maintainer would diversify the perspective. #### Psychological safety - Present: Theo steelmans the vendor twice on camera (00:05:01, 00:05:43) and corrects himself when the post contradicts his theory. He says repeatedly that he likes the model. - Improve: the "you're lying" ultimatum to Truell (00:07:11) and "SpaceXi is spreading misinformation" (00:05:01) are accusatory. The same argument lands without them. The closing question ("Am I being too harsh?") does invite pushback. ## ✅ Logical validity #### Strong reasoning - **Decompositional, 00:06:28.** Per-prompt tokens equal per-request tokens times requests per prompt. If the second factor rises, a small change in the first understates the total. This is valid arithmetic and explains how Truell's 5% and AA's 2x can coexist. Clarify by showing request counts. - **Self-correction, 00:05:43.** Theo proposes the "same base, more RL" theory, reads the post, finds it claims the new base, and abandons the theory on camera. That's reasoning updated by evidence. - **Convergent evidence, 00:07:11 and 00:15:43.** AA's 2x cost, Theo's own runs ($20 worst case), and Ben Davis's usage-limit burn point the same way from three independent sources. Convergence strengthens the cost claim more than any single source. - **Mechanism for a small gap, 00:14:18.** Six index points map to error frequency per time window, and error frequency bounds unattended run length. This gives a reason the number matters instead of asserting it. Strengthen with a measured time-to-first-error. - **Concession against interest, 00:10:46 and 00:21:33.** Theo grants that benchmarks don't capture what he likes about the model, and names specific strengths (floor, on-task persistence, inquisitiveness). He also discloses that his bench uses a contestant as judge and that he used the wrong Gemini version. These raise credibility. #### Fallacies - **False dilemma, 00:07:11.** "Either I'm right and this is misleading or I'm wrong and you're lying." A third option exists: Truell is right about per-request tokens and wrong about what the benchmark measures, without lying. *Fix:* "Your per-request number and AA's per-task number measure different things; the per-task one is what users pay." - **Overstatement of a prediction as a lie, 00:03:36.** "This was a lie" describes Musk's pre-release forecast about efficiency. A missed forecast isn't necessarily a lie; Musk's own later post (00:18:35) attributes the change to a late RL fix. *Fix:* "This promise didn't hold." - **Anecdote as trend, 00:16:28.** One user's $40-for-8% figure is presented as the plan's economics. *Fix:* label it a single data point and ask viewers to report theirs. - **Unsourced rumor, 00:07:54.** "I believe it's coming out later this week according to the rumors" about a new Opus, immediately hedged by Theo himself. *Fix:* drop it or name the source. - **Cherry-picking critique that itself samples, 00:10:46 to 00:11:29.** Theo accuses xAI of cherry-picking benches, then walks only the benches that support his read and dismisses briefcase and GDPval as unimportant or nonsense without saying why. *Fix:* state the criterion for a relevant bench before applying it. - **Ambiguity from uncorrected captions, throughout.** Not Theo's fallacy, but "$186" and "326" as dollar figures, "Deep Sweet V1.1" as a bench name, and "overcope" as a behavior aren't usable as cited facts. *Fix:* corrected captions. #### Tone of voice - Conversational, fast, and combative in the developer-YouTube register, with frequent self-interruption ("Okay, I lied. I have one more thing," 00:20:05). Theo announces his own emotional state ("everything I'm pissed off about," 00:05:01) and apologizes in advance for rudeness twice, then delivers it. The register shifts to genuine enthusiasm at 00:21:33 when describing the model's floor and inquisitiveness, and to disgust at 00:18:35 during the front-end reveal. The pattern worth knowing: the argument is most careful where it's most technical (00:06:28, 00:14:18) and most heated where it's personal (00:07:11). Read the technical sections as evidence and the heated ones as opinion. ## 🛡️ DARVO analysis Partially applicable. Theo positions himself as the person corrected by Michael Truell (00:04:19) and responds; this analysis checks whether his response follows the Deny, Attack, Reverse Victim and Offender pattern. - **Deny.** Theo doesn't deny Truell's per-request numbers. He accepts them as probably true (00:06:28) and argues they measure the wrong thing. No denial or minimization of his own error; he corrects his base-model theory on camera at 00:05:43. - **Attack.** Present in tone, not substance. "Either I'm right and this is misleading or I'm wrong and you're lying" (00:07:11) and "a lot of misinformation being spread by SpaceXi" (00:05:01) attack credibility rather than engage. The substantive engagement happens anyway, which is what separates this from DARVO proper. - **Reverse victim/offender.** Mild. "I just don't enjoy being corrected in a way that is arguably more misleading than what you think you're correcting" (00:07:11) recasts the correction as the offense. Theo's argument stands without this framing. - **Emotional cues.** Pre-emptive apologies ("I really don't want to be as rude as I feel like I have to be here," 00:05:01) followed by the rudeness are a rhetorical pattern that grants permission for the attack. The video as a whole is not DARVO: the core response to criticism is an argument, and the argument is sound. ## 🎯 Conclusion The durable takeaway isn't about Grok 4.7. It's the unit problem at 00:06:28: per-request token growth and per-prompt cost growth are different numbers, and agentic workloads drive a wedge between them. Truell's 5% at median and Artificial Analysis's 2x per task can both be true. The one that hits your bill is per prompt, and Theo's own runs, AA's data, and Ben Davis's plan-limit burn all land there. On the model itself, Theo's evidence sorts into three tiers. Third-party data (AA, Cognition) says roughly 2x tokens, cost above Astra, an index score below the Fable-Astra tier and a bright spot on AA Briefcase. The vendor post says "same price and speed" and benchmarks legal and clinical work for a coding model. Theo's ad-hoc benches say the model is thorough and finds real issues, then produces the worst front-end output he's seen in a year and takes 90 minutes on a game the competition built acceptably. He discloses the limits of his own bench, which is more than the vendor does for theirs. **Next steps for readers:** pull request counts and token totals per prompt from your own agent harness for one week and compute cost per completed task; that's the only number in this debate you can verify yourself. If you're on a subscription plan, log what fraction of your weekly limit a known workload consumes, since list price is not the price. And before citing any figure from this video, get the corrected names and numbers from the xAI post and Artificial Analysis directly, because the auto-captions garble nearly all of them. ## 🔄 Refactor Appendix — Elon promised this one would be good... > Density: 2–5 Literature atoms (H3 pad), 0–3 Permanent candidates (H4 pad). > Emitted: 5 Literature, 2 Permanent. Resonance filter applied — the Opus rumor, > the K-pop tangent, and the sponsor figures were dropped as unevidenced or off-topic. > > Konik claim-header rules: present tense, declarative, ≤10 words, searchable, unique. > > Atomicity: one declarative sentence per atom. "and"/"also" → split. > > Tag pool: #bmf · #addendum · #xref · #articleseed · #storystem · #share · #aphorisms ### 📚 Literature Split Pad #### [H1] Grok 4.7 doubles tokens per task versus 4.6 > [!quote] [00:03:36] > Gro 46 XHigh, according to artificial analysis, did around 38k tokens per task. Gro 47 it did 81k tokens putting it above even Fable 5.1. **Context & use**: - 00:03:36, immediately after Theo reads Musk's promise of better token efficiency. Source is Artificial Analysis, third party, read on screen by Theo. - This is the measured contradiction the whole video hangs on. Cite it as AA's number, not Theo's. **Tags**: #xref <!-- atomicity: 1 idea ✓ --> #### [H2] Per-request token counts hide per-prompt cost > [!quote] [00:06:28] > request is not an accurate way to measure token usage because one prompt can result in a very wide varying number of requests being made. **Context & use**: - 00:06:28, Theo's rebuttal to Truell's "5% more at median" production figure. - The reusable idea: an agentic prompt fans out into N API requests, so per-request percentages understate what the user pays. Ask vendors for per-prompt distributions. **Tags**: #xref #articleseed <!-- atomicity: 1 idea ✓ --> #### [H3] Small index gaps predict unattended error frequency > [!quote] [00:14:18] > those six points indicate a gap in how likely the model is to do something stupid in a given time window. **Context & use**: - 00:14:18, explaining why a 56-versus-62 gap on AA's coding agent index matters. - Gives a mechanism for caring about small benchmark differences: error rate bounds how long an agent runs before a human must intervene. **Tags**: #xref #aphorisms <!-- atomicity: 1 idea ✓ --> #### [H4] Subscription limits set the real model price > [!quote] [00:16:28] > He did $40 of usage and it burned 8% of his weekly limit. **Context & use**: - 00:16:28, Ben Davis's experience on the $300 plan, relayed by Theo. Single-user anecdote. - Pair with 00:26:34: Theo argues the Codex and Claude subscription subsidies, not list price, are why people switch. Plan limits are the cost metric users feel. **Tags**: #xref #share <!-- atomicity: 1 idea ✓ --> #### [H5] Last-gen performance at current-gen price fails > [!quote] [00:27:18] > we're getting last gen performance in various tasks at a current generation price, which just is not the right compromise at all. **Context & use**: - 00:27:18, the closing verdict, after "a stepping stone" at 00:26:34. - A compact rule for evaluating any mid-tier release: capability tier and price tier have to match or the model has no niche. **Tags**: #xref #aphorisms <!-- atomicity: 1 idea ✓ --> ### 🧠 Atomic Split Pad #### [P1] Measure agent cost per completed prompt **Claim** (one sentence, Courtney's voice): The unit that predicts an agent's bill is tokens per completed prompt, because tool-calling turns one prompt into many requests and per-request averages can't see the fan-out. **Evidence pointer**: [[#H1]] · [[#H2]] **Scope & conditions**: Holds for agentic workloads with tool use. Collapses to per-request when a prompt makes exactly one call (chat, single completion). Breaks as a comparison across models if the harness differs, since harness design also drives request count. **Links**: - [[AI cost architecture]] — because extends - [[Classify before you generate]] — because contrasts on where cost is saved: fewer requests versus cheaper requests - [[Coding agent evaluation]] — because applies-in model selection for developer tooling <!-- atomicity: 1 idea ✓ --> #### [P2] A model's floor sets its unattended run length **Claim** (one sentence, Courtney's voice): For agents, the rate of unrecoverable mistakes matters more than peak capability, because it bounds how long the agent can work before a human has to step in. **Evidence pointer**: [[#H3]] **Scope & conditions**: Applies to long-running, low-supervision agent tasks. Doesn't apply to single-shot generation where peak quality is the deliverable. Theo's own counterexample: Grok 4.7 has a higher floor than its tier (00:21:33) yet still costs 2x, so floor is necessary, not sufficient. **Links**: - [[Reading vendor benchmarks]] — because extends - [[A precise number without a method is a marketing claim]] — because contrasts on what a benchmark number is for - [[Delegated work verification]] — because applies-in deciding how often to check on an agent <!-- atomicity: 1 idea ✓ --> ### ✏️ Fleeting Queue - [F1] Find the xAI Grok 4.7 announcement post and confirm the real name of the bench the captions render as "Deep Sweet V1.1" and the exact "same price and speed" wording — https://www.youtube.com/watch?v=jLgpzgpsWPc 00:02:53 #fleeting #inbox - [F2] Check whether Artificial Analysis publishes requests per task alongside tokens per task; if so, decompose the 38k-to-81k jump — 00:03:36 #fleeting #inbox - [F3] The description says "less by 30 to 80%" but the narration says 2x; find where the 30 to 80% figure comes from before quoting either #fleeting #inbox - [F4] Article seed: "Count prompts, not requests," the unit problem in agent cost claims, using the Truell versus AA disagreement as the case — 00:06:28 #fleeting #inbox #articleseed - [F5] Open question: does self-judging in LLM-judged benches bias low, as Fable scoring itself below Grok suggests at 00:23:00? #fleeting #inbox - [F6] Confirm the real name of Theo and Ben Davis's podcast (captions: "Nurse" / "Nerdsip") before linking it anywhere — 00:15:43 #fleeting #inbox ### → Things 3 (max 2 actions) - [ ] Log request count and token total per prompt from my own agent harness for one week and compute cost per completed task — AI cost review, 30 min setup #things - [ ] Verify the Grok 4.7 benchmark names and per-task dollar figures against the xAI post and Artificial Analysis before citing this video — research, 15 min #things <!-- deep-read:sections end --> 📄 **Full text:** [[Full Document Contents/Articles/Elon Promised This One Would Be Good....md|Elon Promised This One Would Be Good... by Theo - t3.gg]] ## Highlights - [00:02:53] improvements that it could make to my CI. This means it cut our bills and our build, well, our build time. Sorry for the pun. You get the idea, though. Stop making your agents wait. Get them better feedback faster at soyv.link/blacksmith. As I was hinting at in the start, the benchmarks don't tell the whole story here at all. Let's go through what they have to say about Grock 47 and then I can share a bit more about my own experience because I've actually seen it outperforming Fable in a handful of my own real world work. Grock 47 is our most capable model for coding and knowledge work. It works longer on difficult tasks. It checks its own work more carefully and it comes with their best calibrated safeguards to date. Served at the same price and speed as Gro 46, it's highly competitive in its class. The same price and speed part is particularly interesting because of things Elon has mentioned in the past. [00:03:36] Specifically, he said that Gro 46 would come out early August and it would be their 1.5 trillion or PAM model and 47 would be a new base, a 2.1 trillion per model that would be released a few weeks later. It'll be better than Gro 46 in every way except slightly slower to serve but with even better token efficiency. This was a lie. Gro 46 XHigh, according to artificial analysis, did around 38k tokens per task. Gro 47 it did 81k tokens putting it above even Fable 5.1. By the way, Soul is all the way on the edge here at 29K tokens. Aster is around the same. Normally I wouldn't call out these contradictions, but I have Michael Truel, the CEO and founder of Cursor, who now is of course ([View Highlight](https://read.readwise.io/read/01m37a5nf63rpmh78gd9s5ce28)) - Note: [H1] Grok 4.7 doubles tokens per task versus 4.6 - Tags: atom/literature/h1 - [00:05:43] they've done instead is continue doing RL on the previous base of the 1.5 trillion per model. That's why the cost is the same, the speeds are the same, and the behaviors are also largely similar. Since all of the attributes that we have at the bottom half of this post aren't true, I think it's fair to assume that the 2.1 trillion part also isn't true. Never mind. They claim in the official post that it is using the new larger base model compared to Grock 46. Okay, so I guess that part's true, which makes all of the rest a lie, which is annoying. But again, back to what Michael said because I can steal man this one a little bit harder. In production, 4.7 is allegedly using 5% more tokens for median requests and 20 to 30% more for P99 requests. Notice that he said request here, though. A [00:06:28] request is not an accurate way to measure token usage because one prompt can result in a very wide varying number of requests being made. If I send a single prompt like respond with hello, the model only has to make one request. It will respond with hello. But if I ask for something that requires a lot of different tool calls to be made, every single tool call results in another request being made. So my assumption here is that what Michael said is true in the sense that on a given request made to API the number of tokens is only increasing 20 to 30% in the worst case. But if the number of requests per prompt is going up which it is then that doesn't matter at all. And when you combine that with a benchmark like artificial analysis, which is leaning ([View Highlight](https://read.readwise.io/read/01m37a5qmknjc1xgdam4w1mb9x)) - Note: [H2] Per-request token counts hide per-prompt cost - Tags: atom/literature/h2 - [00:14:18] skeptical of models that aren't Astra and Fable 5.1 recently. There's a notable tier gap between these two. And I know it seems silly that I care so much about six points on this arbitrary benchmark, but those six points indicate a gap in how likely the model is to do something stupid in a given time window. And as you decrease the levels of stupid, you increase the time window more than you increase the difficulty of task. So if you're looking at this, we're like, yeah, it's slightly dumber, but most of my work isn't that hard. I'm sure it can handle it fine. It can absolutely. I would argue GPT 4.1 a year and a half ago could also handle a lot of your work if you took the time to handhold it and spec the work. exactly as needed. This is all a matter of how much effort do you want to put in up front and Grock 47 will require more [00:15:01] effort up front. It just will. The much scarier thing though is how much money is it going to require and these numbers are not good. I mentioned this earlier and I saw a lot of people speculating that this is due to a caching bug or something where you're not hitting cash properly. No, that's just a lie. It is outright using two times more tokens than prior models in the Grock family, which results in the cost being higher than Astra. $3.26 to run this with Astra. 374 to run it with Gro 47. Still cheaper than Opus and Fable, but that's a given at this point. This type of price increase is just outright unacceptable. If Artificial Analysis were the only ones reporting this, that'd be one thing, but Ben Davis, who's my podcast co-host, shout out to Nurse. If you haven't started watching ([View Highlight](https://read.readwise.io/read/01m37a5sspj2494ffgmc3nd4e9)) - Note: [H3] Small index gaps predict unattended error frequency - Tags: atom/literature/h3 - [00:15:43] yet, I surprised at how good the podcast got as quick as it did. It's much more conversational rather than me just ranting. So, if you want to watch something where someone pushes back on me, check out Nerdsip on your favorite podcast platforms. Regardless, Ben has been my go-to Grock defender. Huge fan of Grock Codefast, dedicated Grockbot user. Ben's a Grock guy, so he was very excited for Grock 47 and was happy with how it behaves, but he called out how aggressive the pricing is. Again, even though Michael disagrees, we have our numbers from our usage and it's clearly not very efficient. The pricing isn't the same as Gro 46 even though the default token price that they were showing is because they double the price over 200,000 tokens and they also [00:16:28] increase it more with fast mode which can result in normal usage being brutal. The result is that the usage limits now actually feel bad. Previously the $300 a month plan was basically unlimited Grock and now he is burning it real fast and the numbers aren't great either. He did $40 of usage and it burned 8% of his weekly limit. That means that you're not getting much per week on the $300 plan, which is kind of crazy when you realize that you get 8 to 12,000 a month on the $200 plans from the other labs. So, not looking great here. As Ben put it, one of the best parts of previous Grock models is that you basically got unlimited usage and cursor and Grock build on the big plans, but that now seems to be done. and also frames this ([View Highlight](https://read.readwise.io/read/01m37a5w2zdaa5pfhx6cps6g7e)) - Note: [H4] Subscription limits set the real model price - Tags: atom/literature/h3 - [00:27:18] recommend. I hope that the next model will be the one where Elon makes good on his promises and we can finally get frontier level performance at a meaningfully better price. But right now, it just kind of feels bad cuz we're getting last gen performance in various tasks at a current generation price, which just is not the right compromise at all. How do you guys feel about this release? Am I being too harsh or is this not something they should have put out in this state? Let me know how y'all feel. And until next time, peace nerds. I have one last gag I want to sneak into the end here. There's no way we're getting that good review repost this time, huh? A man can hope. Sorry, Elon. Next time. ([View Highlight](https://read.readwise.io/read/01m37a5ye2s41wq76m9nsra20v)) - Note: [H5] Last-gen performance at current-gen price fails - Tags: atom/literature/h3