OpenAI Should Be Scared of This One - CourtneyR.dev's 2nd Brain

OpenAI Should Be Scared of This One ![rw-book-cover](https://d34adp677peecb.cloudfront.net/static/images/article0.00998d930354.png) 🔗 **Source:** [OpenAI Should Be Scared of This One](https://www.youtube.com/watch?v=8WbW_n95wc4) > [!note]+ Document Notes > tags: [claude-sonnet-5-5, llm-pricing, prompt-caching, reasoning-effort, agent-orchestration, ai-benchmarks] > author: Theo - t3․gg > source: Theo - t3․gg (YouTube) > reference: https://www.youtube.com/watch?v=8WbW_n95wc4 > date created: 2026-09-29 10:30 > > > [!info] Generated analysis — not the author's words. Sections below are machine-written > > from the captured source. Only text inside `> [!quote]` blocks is verbatim from the author. > > Do not quote or attribute any other line to the named author. > > # 🏷️ Title: OpenAI should be scared of this one >  ## 🧭 Table of contents - [📝 Summary](#-summary) - [📖 Main content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning styles assessment](#-learning-styles-assessment) - [📊 Accessibility assessment](#-accessibility-assessment) - [🌍 DEIB assessment](#-deib-assessment) - [✅ Logical validity](#-logical-validity) - [🛡️ DARVO analysis](#️-darvo-analysis) - [🎯 Conclusion](#-conclusion) - [🔄 Refactor appendix](#-refactor-appendix--openai-should-be-scared-of-this-one) ## 📝 Summary - **What happened.** Theo (channel "Theo - t3․gg") published a 31:46 review on 2026-09-29, the day Anthropic released Claude Sonnet 5.5. The video has no chapter markers. The auto-transcript spells several names phonetically: "Sonet" for Sonnet, "cash" for cache, "Soul" and "Astra"/"Astron" for what are likely model or tier names in OpenAI's GPT-6 line, "Xi high" for xhigh, "Grock" for Grok, "Witch AI" and "Dra" for a design showcase and its author, and "Fish slop" for the demo game. Names inside `> [!quote]` blocks keep the transcript spelling. - **The claim.** Sonnet 5.5 is not a model you should pick directly. It is cheaper per token than Opus 5.5 (half the list price on input, output and cache writes) but it burns more tokens, its cache-read price didn't drop, and its cost on real agent work often matches or beats Opus. Its value is as a sub-agent that Opus or Fable calls for codebase research and planning. - **The evidence offered.** Terminal Bench 4 score of 70.6% versus 10.3% for Sonnet 5 (00:05:03). Artificial Analysis run cost at max effort of $7.60 versus $3.26 for a comparison model (00:13:00). 271,920 tokens per task on Cursor Bench versus 218K for Opus (00:17:17). A 43-minute versus 36-minute run on the fish game demo (00:18:01). A codebase-breakdown benchmark where Sonnet scored slightly above Opus at about half the price in about 5 minutes (00:23:02). - **Why it matters.** Listed per-token prices mislead once cache reads dominate agent cost. Theo's estimate is under 5% of cost on Fable, near 20% on Opus and over 50% on Sonnet 5.5 (00:10:09). That changes which model to route each subtask to. - **Key stakeholders.** Anthropic (Sonnet 5.5, Opus 5.5, Fable 5.1, coming Haiku 5.5), OpenAI (named as the loser on intelligence, efficiency, price and orchestration), CodeRabbit (paid sponsor, 00:01:25 to 00:02:51), Artificial Analysis and Cursor Bench (benchmark sources), and developers paying for agent usage. - **Risks and tradeoffs.** The video is one developer's benchmarks and subscription usage. The sponsor read is part of the runtime. Some price figures are internally inconsistent in the speech (see Main content). The headline "OpenAI should be scared" rests on a routing argument, and the video shows no head-to-head with a current OpenAI orchestrator setup. - **Bottom line.** The reasoning about cache-read share and reasoning-effort floors is checkable and useful. The conclusion that OpenAI has lost "all of the places they are strongest" is opinion backed by the speaker's own usage, and the source contains no OpenAI-side test of the sub-agent pattern. ## 📖 Main content **The thesis.** Theo opens by saying they expected to skip Sonnet 5.5, then reverses: the model is "incredible" but not in ways the benchmark tables show. On the Artificial Analysis intelligence index Theo says it is both more expensive and lower scoring than Opus 5.5 at every effort level (00:00:41). Theo's resolution comes at 00:20:54: Theo doesn't think users should select Sonnet in Claude Code. Its strength is as a tool that stronger models call. **What Anthropic announced.** Per Theo's reading of the announcement at 00:02:51: Sonnet 5.5 is the second model in the 5.5 family, 30% faster and up to 30% cheaper than Sonnet 5 for most work, "strongest at well scoped everyday tasks," and also good at bug fixing and documents, slides and spreadsheets. Haiku 5.5 is promised "in the coming weeks." Pricing is $2 per million input tokens, $10 per million output, and 20 cents per million cache reads (00:07:12). Theo calls Sonnet 5 a "garbage tier model" and says comparing against it flatters the new model. **The cache-read argument.** This is the technical core (00:07:58 to 00:10:09). Agent coding sessions re-read a long context from cache on every step, so cache reads are the largest cost line. Theo's figures, all spoken and unverified against an on-screen chart the transcript can't show: - Sonnet 5.5: $2 in, $10 out, $2.50 cache write (as Theo states it), 20 cents cache read. - Opus 5.5: $4 in, $20 out, $5 cache write, and a cache-read price Theo says is unchanged from Sonnet's. - Fable 5.1: $10 in and $50 out, cache read 25 cents. If Opus really holds at 20 cents, Sonnet and Opus share a cache-read price while every other rate doubles from Sonnet to Opus. That is why the cheaper model saves less than the price sheet implies. Theo's shares of cost from cache reads: under 5% on Fable, close to 20% on Opus, above 50% on Sonnet. Those shares are Theo's estimates from Theo's own runs, not published figures. **Inconsistencies to check against the pricing page.** At 00:08:44 Theo says Opus dropped cache-read cost about 60% and that "usually cache reads are 90% off," yet a 20-cent Opus rate on $4 input is 95% off. At 00:09:26 Theo says Fable is "double the cost of Opus 5.5" at $10 and $50 against $4 and $20, which is 2.5x. Both look like slips of speech rather than argument-breaking errors, but the source doesn't settle them. Theo also says every website reporting the pricing is "garbage" and reads the numbers from a Google AI summary, which is a weak source for a pricing claim. **Reasoning effort.** Theo argues effort levels are budgets, not targets, and that max works differently: it sets a floor on reasoning tokens (00:11:36 to 00:12:18). In Theo's benchmarks, low to xhigh raised token use 5 to 8%, and xhigh to max raised it 1,500%, or 15x. Theo says max can lower accuracy because forced overthinking produces second-guessing. Theo gives cost per Artificial Analysis run: Sonnet 5.5 at max $7.60 against $3.26 for the comparison model, with cheaper figures at xhigh, high and medium that the transcript garbles ("$18 and 59"). Theo's advice: skip max and low, medium is "fineish" and works better on Opus than on Sonnet. This is an experience-based claim from Theo's own benches and is not documented by Anthropic in the video. **Benchmarks and their limits.** Terminal Bench 4 jumps from 10.3% to 70.6% (about 6.9x), which Theo says makes them skeptical of the benchmark. GDPval lands slightly under Opus 5.5, and Theo doesn't care about it. Sonnet 5.5 is the first Sonnet to beat Pokémon Red working from screenshots only. Computer use ties Opus 5.5. On Frontier Code, xhigh beats max, and both medium and low drop hard. On Cursor Bench, Sonnet uses 271,920 tokens per task against Opus at 218K and Gemini Flash at 162K, and more than 5x the tokens of the OpenAI models Theo names. Theo notes Anthropic charts score against cost, never against tokens. **Speed.** OpenRouter shows about 94 tokens per second for Sonnet 5.5 and about 70 for Opus 5.5, against about 30 for a GPT-6 tier Theo calls Astra. Theo's own subscription numbers: about 150 for Sonnet, about 100 for Opus. Because Sonnet emits more tokens, the fish game took 43 minutes against 36 for Opus, and 39 minutes of generation against 27. **Design quality.** In the design showcase Theo finds Sonnet 5.5 "definitely a little worse" than Opus and "significantly worse" than Fable, with weak card designs, awful animations and scroll issues, though still ahead of OpenAI and xAI output. **Where Sonnet wins.** Theo's own benchmark, first built for a Grok review: models audit a giant orchestrator pull request in the T3 Code repo (hundreds of thousands of lines) and propose a plan to land it in pieces. Astra planned best. Sonnet scored slightly above Opus at about half the cost and finished in about 5 minutes. Opus took nearly 2x as long and Astra nearly 3x. GPT-6 Soul was faster at about half the score. Judging is an automated panel Theo built, so the score is not independent. **The fish game demo.** One prompt, 40 minutes, about $16 at API prices, by Theo's estimate. Input tokens ran over 3.5x Opus's, and total cost matched Opus's, but the Opus run used a Fable sub-agent by accident, so the comparison is contaminated. Theo rates the output among the best they've seen and better than anything from OpenAI models. **Subscription math.** Across six Claude accounts (three closed in recent days) Theo measures about $2,300 of usage per week on the $200 plan, which annualizes to roughly $9,900 per 30 days by simple proration (2,300 x 30 / 7). Theo says the $200 Codex plan gives them far less real work. **Contested or unverifiable in this source.** The chart values behind every cost claim are on screen and not in the transcript. The rank order of the OpenAI models is stated with no test shown. The claim that Anthropic "saw everybody else distilling" its models and copied the approach (00:06:28) is speculation. "OpenAI dev day" timing is hedged ("may have already happened"). ## 🗒️ Notes #### ⏱️ [00:00:00](https://www.youtube.com/watch?v=8WbW_n95wc4&t=0) Cold open: Anthropic's run and the reversal - Lists Fable 5.1, Opus 5.5 and now Sonnet 5.5 as the year's Anthropic releases they like, and says they haven't chosen Fable for any task in over a week. - Says Sonnet 5 was among Theo's least favorite releases. Sonnet 5.5 is a model Theo didn't expect to care about. > [!quote] [00:00:41] > On the artificial analysis intelligence index, it comes out as more expensive and dumber than Opus 5.5 at every single level. #### ⏱️ [00:01:25](https://www.youtube.com/watch?v=8WbW_n95wc4&t=85) Sponsor segment: CodeRabbit change stack - Paid read for CodeRabbit (link in description: soydev.link/coderabbit). Product claims: a change stack that groups a pull request by what changed and why, a security blast radius diagram, and an activity view. - Theo's stated motive: AI-assisted contributions make it harder to track what merged, including Theo's own agent's changes they approved by trusting AI reviewers. #### ⏱️ [00:02:51](https://www.youtube.com/watch?v=8WbW_n95wc4&t=171) The official announcement - 30% faster and up to 30% cheaper than Sonnet 5, described as a clear upgrade. Theo says the comparison target is weak because Sonnet 5 was poor. - Product ladder as Theo states it: Sonnet is the smallest shipping model, then Opus, then Fable. Haiku 5.5 is promised in a few weeks. - Anthropic's stated fit: everyday scoped tasks, bug fixing, polished documents, slides and spreadsheets, and a "sharp eye for design," which Theo says the demos don't support. #### ⏱️ [00:05:03](https://www.youtube.com/watch?v=8WbW_n95wc4&t=303) Benchmark scores and what Theo trusts - Terminal Bench 4: 70.6% against 10.3% for Sonnet 5, the highest score on that board. - Slightly below Opus 5.5 on GDPval. First Sonnet to beat Pokémon Red from screenshots only. - Theo's read: Anthropic's smaller models now feel like faster, slightly weaker Fable, which Theo credits to internal reinforcement-learning changes and to distillation. #### ⏱️ [00:07:12](https://www.youtube.com/watch?v=8WbW_n95wc4&t=432) Writing quality and the "30% less per task" line - Sonnet 5.5 writes more clearly, with fewer of the "Claudisms" Theo dislikes. - Price: $2 in, $10 out, 20 cents cache read, unchanged from Sonnet 5. Anthropic says fewer tokens per task make it up to 30% cheaper. - Theo won't call the claim a lie, but says it dances around max effort and cache reads. #### ⏱️ [00:08:44](https://www.youtube.com/watch?v=8WbW_n95wc4&t=524) Cache-read pricing across the tiers - Compares Sonnet, Opus and Fable rates. Fable's other rates are double Opus's, but its cache read costs only 25% more. - Cache reads grow as a share of cost as you move down the tier list. > [!quote] [00:10:09] > Which means that the costs for using this model don't end up being as much cheaper as you might think when you look at these numbers #### ⏱️ [00:10:53](https://www.youtube.com/watch?v=8WbW_n95wc4&t=653) Artificial Analysis cost and the max-mode rant - On real-task cost, Sonnet 5.5 sits neck and neck with Fable 5.1 on the most expensive run Artificial Analysis has done. - Effort levels are budgets, not levels. Low to xhigh raised token use 5 to 8% on Theo's benches. #### ⏱️ [00:12:18](https://www.youtube.com/watch?v=8WbW_n95wc4&t=738) Max sets a floor, and the cost gap that follows - Max raises the minimum number of reasoning tokens, so easy tasks can't finish early. > [!quote] [00:12:18] > And from X high to max you get, and I'm not exaggerating, a 1,500% increase in token usage. - Sonnet 5.5 at max cost $7.60 against $3.26 for the comparison model, and Theo says xhigh is closer to that model's price. - Theo would still take Sonnet over that comparison model for daily code work because of fewer "dumb spikes." #### ⏱️ [00:14:26](https://www.youtube.com/watch?v=8WbW_n95wc4&t=866) Charts, effort guidance and instant mode - Frontier Code: xhigh scores above the OpenAI GPT-6 Soul entry, max scores below it. Theo calls Soul "DOA." - Anthropic no longer ships a no-reasoning mode, because the models perform poorly without reasoning. Theo's rule: avoid low, avoid max, medium is fine and stronger on Opus. - The first chart in the announcement shows Sonnet 5.5 costing more than Opus at max while scoring higher, which Theo reads as a sign Terminal Bench is a weak measure. #### ⏱️ [00:17:17](https://www.youtube.com/watch?v=8WbW_n95wc4&t=1037) Token use and speed - Cursor Bench tokens per task: 271,920 for Sonnet 5.5, 218K for Opus, 162K for Gemini Flash. > [!quote] [00:18:01] > So, it is faster, but it's also not very token efficient. - OpenRouter TPS: about 94 (Sonnet 5.5) versus about 70 (Opus 5.5). Theo's own: about 150 versus about 100. - Fish game: 43 minutes on Sonnet, 36 on Opus, with 39 and 27 minutes spent generating tokens. #### ⏱️ [00:18:46](https://www.youtube.com/watch?v=8WbW_n95wc4&t=1126) Design showcase results - Fable, Opus and Sonnet runs compared in the showcase. Sonnet's outputs show scroll bugs, low-readability background lines, ugly cards and poor animation. - Theo still ranks Fable 5.1 best for marketing-site front ends and Opus as steerable for design. #### ⏱️ [00:20:54](https://www.youtube.com/watch?v=8WbW_n95wc4&t=1254) The turn: Sonnet as a tool, not a pick - Restates every weakness, then says few people should choose Sonnet in Claude Code. Theo pre-empts the "not everyone can afford Opus" reply by noting Sonnet isn't cheaper on like-for-like work. > [!quote] [00:21:38] > We shouldn't be calling Sonnet directly. It should be used as one of many things that Opus or Fable will orchestrate #### ⏱️ [00:22:20](https://www.youtube.com/watch?v=8WbW_n95wc4&t=1340) The orchestrator breakdown benchmark - Task: audit a giant orchestrator v2 pull request and propose a plan to land it in smaller pieces. Theo revised the judging, which lifted Fable's score. - Ranking as Theo states it: Astra best, then Opus (Fable below Opus), with Sonnet slightly above Opus at about half the price. Soul was faster at about half the score. - Timing: Sonnet about 5 minutes, Opus almost 2x, Astra almost 3x. #### ⏱️ [00:24:28](https://www.youtube.com/watch?v=8WbW_n95wc4&t=1468) Behind the scenes: TypeScript-in-Rust and an intent test - Context: Theo is rewriting the TypeScript compiler in Rust, and Opus has left "millions of lines" of legacy code from earlier OpenAI-model runs. Theo asked a Sonnet thread for a cleanup update. - Sonnet kept following the thread's earlier topic (numbers) instead of Theo's wrong-thread question. Theo first read that as a regression, then judged it correct. The agent reports about 1.6 million lines deleted since filming began. #### ⏱️ [00:26:37](https://www.youtube.com/watch?v=8WbW_n95wc4&t=1597) The fish game demo - Input tokens: over 3.5x Opus's, total cost similar. The Opus run spawned a Fable sub-agent by accident, so the cost comparison is unclean. - Output: recognizable 3D models, 120 fps on their laptop, good animation and the best alien they've seen in these tests. Audio broke on their end mid-demo and they fixed it on camera. #### ⏱️ [00:28:46](https://www.youtube.com/watch?v=8WbW_n95wc4&t=1726) API price versus subscription usage - About $16 of API cost for a game they say is worth more than some they bought as a kid. - Six Claude accounts, three closed: about $2,300 of usage per week on the $200 plan. #### ⏱️ [00:30:12](https://www.youtube.com/watch?v=8WbW_n95wc4&t=1812) Verdict and the OpenAI claim - If Opus is configured to call Sonnet for sub-agent research, Theo expects Opus to feel faster and cost slightly less. - Theo says OpenAI lacks the smartest, most efficient and cheapest models and the best model for calling other models, and that Theo's $200 Codex plan yields far less than a $200 Claude plan. Sign-off at 00:31:37 predicts faster releases ahead. ## 📚 References The transcript itself contains no links; the following come from the video description or are primary sources for named entities. - [OpenAI should be scared of this one — Theo - t3․gg](https://www.youtube.com/watch?v=8WbW_n95wc4) — the source video, 31:46, uploaded 2026-09-29. - [CodeRabbit sponsor link](https://soydev.link/coderabbit) — from the video description; the sponsor segment runs 00:01:25 to 00:02:51. - [Sponsor inquiry link](https://soydev.link/sponsor-me) and [t3.gg](https://t3.gg) — from the description. - Anthropic's Claude Sonnet 5.5 announcement post — Theo reads from it on screen; the transcript gives no URL. Find it on Anthropic's site before citing pricing. - Artificial Analysis intelligence index — the cost and score source Theo cites for effort-level comparisons. No URL in the transcript. - Cursor Bench and OpenRouter — the sources for token-per-task and tokens-per-second figures. - T3 Code — the codebase used in Theo's orchestrator benchmark. - The description credits @Ph4seon3 for the edit. ## 🗣️ Paraphrase Theo starts by saying the numbers make Sonnet 5.5 look like a mediocre release. It scores lower than Opus 5.5, it costs about the same on real work, it emits more tokens, its design output is weak, and its speed advantage disappears when you count the extra tokens. Theo spends the middle of the video explaining why: the price sheet halves everything except the cache-read rate, and agent work is mostly cache reads, so the discount mostly doesn't reach the bill. Theo adds a warning about max reasoning effort, which Theo says forces long thinking and can cost 15x more than xhigh with no accuracy gain. Then Theo flips. Theo built a benchmark where a model reads a huge codebase and proposes how to split a giant pull request. Sonnet 5.5 beat Opus on Theo's judging panel at about half the price and finished in about 5 minutes. Theo's conclusion is a routing rule: don't pick Sonnet yourself, let Opus or Fable hand it research and scoping subtasks. Theo ends with a shot at OpenAI, whom Theo says trails on capability, efficiency, price and orchestration, and with a plug for Theo's Claude subscription math. ## 📰 Newsletter **Cheaper per token is not cheaper per task.** Theo's 32-minute look at Anthropic's new Sonnet 5.5 is worth your time for one habit: read the cache-read line, not the headline price. Theo estimates cache reads at over 50% of Sonnet 5.5's agent cost against about 20% on Opus, which is why a model priced at half of Opus can cost the same on a real job. Two takeaways for anyone wiring AI into WordPress tooling. First, treat reasoning effort as a budget. Theo's benchmarks show a 5 to 8% token change from low to xhigh and a 15x jump at max. Second, route by task shape. Theo's codebase-audit benchmark had Sonnet matching Opus at half the price in 5 minutes, which suggests a cheap research sub-agent under a stronger planner. The sponsor read and the single-developer benchmarks are worth discounting, and the pricing math is worth checking against Anthropic's own page. Links: [the video](https://www.youtube.com/watch?v=8WbW_n95wc4), [CodeRabbit](https://soydev.link/coderabbit). ## ✅ Action items #### DevRel - Developer advocacy: publish a worked example that logs cache-read, cache-write, input and output token counts per agent step, so cost claims come from a receipt and not a price sheet. - Education: run an office-hours session on reasoning effort as a budget, using the 5 to 8% versus 1,500% token-change claim as a hypothesis attendees test on their own prompts. - Developer experience: in any agent quickstart, show a two-tier setup where a stronger model delegates research to a cheaper one, and report cost per completed task. - Developer marketing: don't repeat "30% cheaper" or the OpenAI ranking in Pantheon or community material without citing the pricing page and a reproducible test. - Community management: add a "sponsored video, single-developer benchmark" note where the link gets shared. - Internal/external: ask model vendors for token-per-task and cache-read-share figures next to their score-per-dollar charts. #### OSPO - Policy and governance: write down which model tiers contributors may use for triage and who pays, before AI-assisted review scales. - License compliance: check each model provider's terms for training on submitted code before repository content goes into a sub-agent. - Security management: a research sub-agent that reads a whole repo is a prompt-injection surface via file contents. Add it to the threat model and log what the sub-agent read. - Software inventory: record model name, tier and pricing date per pipeline, since a price change or model retirement breaks cost assumptions. - Developer support: publish guidance on effort settings: default to high or xhigh for hard tasks, and require a stated reason for max. - Community engagement: no direct action from this source; nearest relevance: sponsors that fund AI credits for maintainers should get cost-per-task reports. - Strategic alignment: set a KPI for inference spend per merged change, measured from invoices. #### Community - Maintainers: pilot a cheap-model research pass for "what does this pull request touch" summaries on one repo and compare against human summaries for 30 days. - Contributors: document how to disclose AI-assisted changes, since Theo describes merging Theo's own agent's changes, which Theo didn't recall. - Sponsors: if a sponsor covers AI review tooling, ask for the tool's own accuracy data, not a demo. - Ecosystem: no direct action from this source; nearest relevance: a shared benchmark of large-pull-request breakdown tasks would let projects compare models on comparable work. #### WordPress - Core: no direct action from this source; nearest relevance: large pull request breakdown is a real Core problem, and the benchmark idea applies. - Core Performance: no direct action; the speed claims concern model output, not PHP requests. - Core AI: add reasoning-effort budgets and cache-read pricing to any guidance on running agents against Core, and require evaluation on cost per completed task. - **Core Program Team:** track model-tier costs as a line in cross-team AI experiments so pilots stay comparable. - Mobile: no direct action from this source. - Playground: a demo that runs the same audit task at two effort levels and shows token counts would teach the budget-versus-floor distinction. - Community: no direct action from this source; nearest relevance: meetup talk on routing cheap models under expensive ones. - Training: a lesson on reading vendor pricing tables, with cache reads as the worked example. - Marketing: no direct action from this source. - Support: no direct action from this source; nearest relevance: a cheap research model could gather context before a human answers a ticket. - Design: the video reports weak Sonnet design output; no action beyond noting design tasks need a stronger model. - Documentation: add a note to AI-assisted docs workflows that codebase-audit steps can run on a cheaper tier, and verify the result. - Photos: no direct action from this source. - TV: no direct action from this source. - Polyglots: no direct action from this source. - Accessibility: no direct action from this source; the video itself has auto-caption gaps noted below. - Meta: no direct action from this source; nearest relevance: cost tracking for any .org tooling that calls models. - CLI: a WP-CLI command that reports token and cost per AI-assisted operation would make the cache-read issue visible. - Hosting: no direct action from this source. - Tide: no direct action from this source; nearest relevance: a cheap research pass to pre-summarize scan findings. - Openverse: no direct action from this source. - Themes: review queues could test a research sub-agent, with a human gate. - Plugins: same as Themes, with a mandatory human gate on security-adjacent findings. #### The WPOCC - Programs: if The WP Open Community Collective funds maintainer AI tooling, track cost per completed task in the grant report. - Projects: no direct action from this source. - Funding experiments: require published token and cost data as a condition for any AI-tooling microgrant. ### FAIR - Technical independence: a workflow that depends on one vendor's cache pricing or one model tier inherits that vendor's price changes. Document the dependency. - Policies and covenants: require disclosure when a decision came from a model-generated audit, including which model and effort level. - Productized governance artifacts: add a "model tier and effort policy" row to operator scorecards. ## 🧪 Hypotheses 1. For agent coding tasks that re-read a fixed context, cache reads account for over 50% of Sonnet 5.5's cost and about 20% of Opus 5.5's, so per-task cost is within 15% between them despite a 2x list-price gap. 2. Raising effort from xhigh to max increases reasoning tokens by more than 10x on the same task set and doesn't raise accuracy. 3. Sonnet 5.5 uses at least 1.25x the tokens per task of Opus 5.5 on a fixed coding benchmark, enough to cancel its tokens-per-second lead in wall-clock time. 4. On codebase-analysis and planning tasks, Sonnet 5.5 matches Opus 5.5's judged quality within 5% at 60% or less of the cost. 5. An Opus orchestrator that delegates read-only research to Sonnet 5.5 finishes real work faster and no more expensively than Opus alone. 6. The judged advantage in the video's benchmark shrinks when a different judging panel or rubric is used, because the panel is the author's own. ## 📊 Methodology - **Design for H1 and H3.** Fix a 30-task suite of repo-fixing and feature tasks, run each model three times at the same effort, and record cached input, uncached input, cache writes, output tokens and wall-clock time from provider invoices. Report cost per completed task and cache-read share. - **Design for H2.** Run one task set at low, medium, high, xhigh and max. Log reasoning tokens and pass rate per level, with three seeds. Pre-register the "no accuracy gain" threshold before scoring. - **Design for H4.** Reuse the large-pull-request breakdown task on 5 public repositories with more than 100,000 changed lines. Use two independent judging panels and a human review of a 20% sample. Measure quality score, cost and time. - **Design for H5.** A/B one real backlog: Opus alone versus Opus with a Sonnet research sub-agent. Count completed tasks, not requests, and divide spend by completions. - **Design for H6.** Replay the judged outputs through a second rubric and compare rank agreement (Kendall tau). - **Controls.** Pin model versions and dates, disable accidental delegation (the video's Opus run spawned a Fable sub-agent), and use the same prompt, same repo commit and same tooling. ## ❓ Questions - What exact cache-read price does Opus 5.5 carry, and does it match the 20 cents Theo implies at 00:09:26? - Which models are "Soul" and "Astra" in the transcript, and what did those runs cost in Theo's benchmark? - What did the garbled xhigh, high and medium cost figures ("$18 and 59") actually show? - How is the automated judging panel built, and does it favor a particular answer style? - How much of Sonnet 5.5's higher token count comes from reasoning versus tool-call output? - Does the sub-agent pattern hold when the research subtask needs judgment, not just search? - How does cost per task change when a cache-read discount lands on Sonnet, as Theo wishes at 00:10:09? - What does Anthropic's Haiku 5.5 change in this routing picture? ## 🔮 Future research - **Cache-read share as a headline metric.** Vendors chart score per dollar. A published cache-read share per model tier would make claims like the 50% figure checkable. - **Effort budgets versus floors.** A public measurement of reasoning-token distributions per effort level, per vendor, would confirm or refute the max-as-floor claim. - **Orchestrator and sub-agent routing.** Systematic studies of which subtasks a cheaper model can take without hurting the parent's result. - **Judged benchmarks for repository-scale planning.** Task families like large-pull-request breakdown are scarce and mostly private. - **Prompt injection through delegated reads.** Sub-agents that read whole repositories carry injection risk that current coding benchmarks don't score. ## 💡 Implications - **Practice.** Pick the model by cost per completed task, and route read-only research to a cheaper tier only after measuring it. Treat effort levels as budgets and test max only with a stated reason. - **Policy.** Teams should record model tier, effort and date in every AI-assisted change, since Theo describes losing track of what an agent merged. - **Security.** A research sub-agent reading a full repository ingests untrusted text. Scope its permissions to read-only and log its reads. - **Education.** Reading a vendor pricing table is a teachable skill. The cache-read row is the worked example this video supplies. - **Funding.** Grants for AI tooling should ask for tokens and dollars per outcome, not seat counts or list prices. ## 🧠 Learning styles assessment - **Reading level estimate.** Flesch-Kincaid Grade Level of the transcript body is about grade 8, and Flesch Reading Ease lands near 65 (my estimate from sentence length and syllable density, not a computed score). Conversational, long run-on sentences, with domain terms (cache reads, TPS, effort levels, sub-agent) that assume a working developer audience. #### 👀 Visual learners - Present: on-screen pricing chart, Artificial Analysis and Cursor Bench charts, design showcase, and a playable game demo. - Improve: the key figures (cache-read rate per tier and cost shares) live in charts the narration only partly reads. Add a one-slide table of the three tiers with cache-read share labeled. #### 🔊 Aural learners - Present: continuous narration with clear emphasis and repeated summaries of each point. - Improve: many figures are spoken quickly with no restatement (for example the garbled "$18 and 59"). A short spoken recap of the price table would help listeners. #### 🗣️ Verbal learners - Present: analogies like "budget, not level," and plain statements of the thesis at 00:20:54. - Improve: cache reads, TPS, floor and ceiling of reasoning tokens go undefined for newcomers. One clause each would fix it. #### 🧩 Kinesthetic learners - Present: live demos of the game and the thread mix-up. - Improve: no repository, prompt or benchmark harness link, so a viewer can't reproduce the orchestrator benchmark. Publish the prompt and judging rubric. #### 🧠 Logical learners - Present: a clear chain from price sheet to cache-read share to per-task cost, and a stated counterexample to Theo's own thesis (the benchmark where Sonnet wins). - Improve: show the arithmetic behind the 50%, 20% and under-5% shares, and reconcile the 2x versus 2.5x Fable slip. #### 🧑‍🤝‍🧑 Social learners - Present: asks viewers to comment on whether they use models for documents and slides, and preempts the expected affordability argument. - Improve: no discussion prompt tied to the core claim. Invite viewers to post their own cache-read share. #### 🧘‍♀️ Solitary learners - Present: a self-contained 31-minute walkthrough with a clear verdict at the end. - Improve: no self-check. "Compute your own cache-read share this week" would turn the video into an exercise. #### 🌿 Naturalistic learners - Present: none explicit. The fish game and coral visuals are incidental. - Improve: the 15x token increase at max implies a compute and energy cost. Naming it would connect the cost story to resource use. ## 📊 Accessibility assessment #### Screen reader semantics - Present: YouTube provides player landmarks and a machine-readable auto-transcript. The description contains links as text. - Improve: the charts and pricing table carry data the narration doesn't fully speak. WCAG 2.2 AA 1.1.1 (Non-text Content) is unmet for those charts, and 1.2.5 (Audio Description, Prerecorded) is not addressed. #### Keyboard navigation - Present: the YouTube player is keyboard operable with visible focus. - Improve: no barriers added by the content. 2.1.1 and 2.4.7 pass via the platform. There are no chapter markers, which weakens 2.4.5 (Multiple Ways) for navigating a 31-minute video. #### Contrast and readability - Present: unverifiable from a transcript. Theo warns of a "flashbang" light-mode page at 00:10:53, which suggests Theo switches themes on screen. - Improve: sudden theme flips can strain low-vision viewers and risk 1.4.3 (Contrast Minimum) in dense charts. Zoom into chart regions. #### Text resizing and reflow - Present: the player scales, and captions resize independently. - Improve: chart labels and on-screen text baked into the frame don't reflow. 1.4.4 (Resize Text) and 1.4.10 (Reflow) fail for burned-in text. #### Media and captions - Present: an auto-generated transcript exists, and the audio is mostly clear. - Improve: auto-captions garble product and model names ("Sonet," "cash," "Soul," "Astra," "Xi high," "Grock," "Witch AI"). That's a 1.2.2 (Captions, Prerecorded) failure in substance, because proper nouns and prices carry the argument. Human-corrected captions and a chapter list are the fixes. #### Forms and interactions - Present: no forms. The sponsor link and description links are selectable text. - Improve: the spoken sponsor URL ("soy.link/code rabbit") is garbled, so the description link is the only reliable path. Say the link letter by letter. #### Standards reference - Primary gaps against WCAG 2.2 AA: **1.2.2** (captions inaccurate on proper nouns and figures), **1.2.5** (no audio description for charts), **1.1.1** (chart data conveyed visually only), **1.4.4 / 1.4.10** (burned-in text), and **2.4.5** (no chapters). The caption accuracy fix has the highest impact here because prices are the evidence. ## 🌍 DEIB assessment #### Cultural breadth - Present: references to Western consumer examples (Fortnite skins, PlayStation games) and US-centered pricing in dollars. - Improve: no non-US developer perspective or pricing in local currency, and the "not everyone can afford Opus" point is dismissed rather than analyzed for regions where $200 a month is out of reach. #### Inclusive language - Present: direct second-person language, defined product names. - Improve: "insane," "crazy," "garbage," "call me insane" appear repeatedly (00:00:00, 00:08:44, 00:13:42), and "shut the hell up" at 00:20:54 is aimed at hypothetical commenters. Replace with concrete descriptors. #### Neurodiversity support - Present: a stated thesis early and again at 00:20:54, and repeated recaps of key claims. - Improve: no chapters, and long digressions (the thread mix-up, audio fix) interrupt the structure. Chapter markers would help viewers find the cache-read and max-mode segments. #### Representation - Present: one narrator, credit to an editor by handle in the description. - Improve: no outside voices. A guest with a different stack or budget would broaden the view. #### Psychological safety - Present: mostly a lighthearted tone toward viewers, with invitations to comment. - Improve: dismissive lines aimed at anticipated commenters ("shut the hell up") can discourage viewers on tighter budgets from participating. ## ✅ Logical validity #### Strong reasoning - **Causal, 00:08:44 to 00:10:09.** If cache reads are a fixed price across tiers while other rates scale, then cheaper tiers see cache reads take a larger share of cost. The mechanism is clear and checkable against a price page. To strengthen: publish the per-step token mix. - **Mechanistic, 00:11:36 to 00:12:18.** Explains a floor versus a ceiling on reasoning tokens and gives a specific measured jump (5 to 8% versus 1,500%). Falsifiable with a public test. - **Self-limiting disclosure, 00:26:37.** Admits the Opus run spawned a Fable sub-agent and so isn't clean, and that Theo hasn't tested Sonnet's design limits. - **Counterexample to Theo's own thesis, 00:22:20 to 00:24:28.** After arguing Sonnet doesn't deliver value on price, Theo presents a benchmark where it does. That is honest and it sets up the routing conclusion. - **Correcting Theo's own misread, 00:24:28 to 00:25:55.** Theo starts to call a wrong-thread reply a regression, checks, and reverses. #### Fallacies - **Hasty generalization, 00:21:38 to 00:23:46.** One custom benchmark on one codebase (T3 Code) with an automated judge supports the claim that Sonnet suits research sub-agent work. Fix: run more repositories and a second judging panel. - **Confounded comparison, 00:26:37.** The fish game cost comparison is contaminated by the accidental Fable sub-agent in the Opus run. Theo notes it, but still says the costs were "quite similar." Fix: rerun Opus alone. - **Unsupported universal, 00:30:56.** "They have lost in all of the places they are strongest." Shown evidence covers a few benchmarks and Theo's own usage, not the OpenAI models' full range. Fix: name the tasks and numbers behind the ranking. - **Appeal to personal experience, 00:31:00 (approx., same paragraph).** "$200 Codex plan gets me nowhere near as much usage" compares two plans on Theo's own workload. Fix: give tokens or tasks per dollar for each. - **Weak sourcing for a numerical claim, 00:08:44.** Theo reads pricing from a Google AI summary while saying other sources are unreliable. Fix: cite the vendor pricing page. - **Internal inconsistency, 00:08:44 and 00:09:26.** Opus cache reads described as a 60% drop but a 20-cent rate implies 95% off, and Fable is "double" Opus at a 2.5x ratio. Fix: read the prices from the chart and state them once. - **Undisclosed interest in framing, 00:00:00 and 00:30:56.** The video's title and closing frame OpenAI as the loser. The sponsor is CodeRabbit (disclosed as a sponsor), and Theo says they've shifted spend to Claude accounts, which isn't a disclosure of any financial tie but is context for the ranking. Fix: state any relationships once at the top. #### Tone of voice - Energetic, opinionated and conversational, with profanity and hyperbole ("mindblowingly crazy," "garbage tier"). Confident and combative toward imagined commenters and toward OpenAI. The middle of the video (cache math, effort levels) is more methodical, and the edges (title, opening, closing) carry the strongest claims. Second person and contractions throughout. ## 🛡️ DARVO analysis Not applicable to this source. DARVO (Deny, Attack, Reverse Victim and Offender) describes a response pattern to an accusation, and this is a product review with no accuser. For completeness: **Deny.** Theo doesn't deny wrongdoing; the nearest analog is pre-empting a pricing objection at 00:20:54. **Attack.** Theo attacks a hypothetical commenter ("shut the hell up") and criticizes OpenAI's models, which is product criticism and not a response to an accusation. **Reverse victim/offender.** No such move appears. ## 🎯 Conclusion The durable content is a method: read the cache-read row, treat effort levels as budgets, and route by task shape. At 00:08:44 to 00:10:09 Theo shows why a half-price model can cost as much per task when cache reads dominate. At 00:11:36 to 00:12:18 Theo gives a specific measured token jump for max effort. At 00:22:20 to 00:24:28 Theo shows a task where the cheaper model matches the stronger one. The weaker part is the ranking claim. "OpenAI should be scared" rests on Theo's own benchmarks, an automated judge Theo built, a contaminated cost comparison and a plan-price comparison on Theo's workload. Some spoken prices don't add up, and the video's charts aren't in the transcript. Treat the mechanism as testable and the verdict as opinion. **Next steps for readers:** pull your own last month of agent invoices and compute cache-read share per model. Check Anthropic's pricing page for the cache-read rate on each tier. If cache reads exceed 40% of spend, test routing read-only research subtasks to a cheaper tier, and measure cost per completed task, not per token. Avoid max effort unless a task fails at xhigh. ## 🔄 Refactor Appendix — OpenAI should be scared of this one > Density: 2–5 Literature atoms (H3 pad), 0–3 Permanent candidates (H4 pad). > Emitted: 4 Literature, 2 Permanent. Resonance filter applied: the design showcase > and Pokémon claims were dropped as thinly evidenced. > > Konik claim-header rules: present tense, declarative, ≤10 words, searchable, unique. > > Atomicity: one declarative sentence per atom. "and"/"also" → split. > > Tag pool: #bmf · #addendum · #xref · #articleseed · #storystem · #share · #aphorisms ### 📚 Literature Split Pad #### [H1] Cache-read share erases half-price model discounts > [!quote] [00:10:09] > the costs for using this model don't end up being as much cheaper as you might think when you look at these numbers **Context & use**: - 00:10:09, after showing cache reads at 20 cents on Sonnet and roughly equal on Opus. - The mechanism is the argument: agent work is mostly cache reads, so a discount on the other rates barely reaches the bill. Verify against the vendor price page. **Tags**: #xref #articleseed  #### [H2] Max effort sets a reasoning floor > [!quote] [00:12:18] > from X high to max you get, and I'm not exaggerating, a 1,500% increase in token usage **Context & use**: - 00:12:18, from Theo's own benchmark runs, contrasted with a 5 to 8% change from low to xhigh. - Testable claim. Keep as a hypothesis until a public measurement replicates it. **Tags**: #xref  #### [H3] Faster models can still finish slower > [!quote] [00:18:01] > So, it is faster, but it's also not very token efficient. **Context & use**: - 00:18:01, just before the fish game timing: 43 minutes on Sonnet versus 36 on Opus. - Wall-clock time is tokens divided by speed, so a token-hungry model loses its speed lead. Compare on time per task, not tokens per second. **Tags**: #xref  #### [H4] A cheap model belongs under an orchestrator > [!quote] [00:21:38] > We shouldn't be calling Sonnet directly. It should be used as one of many things that Opus or Fable will orchestrate **Context & use**: - 00:21:38, the thesis stated after the price and token problems, and backed by the orchestrator benchmark at 00:23:02. - Evidence is one custom benchmark judged by an automated panel. Treat the routing pattern as a hypothesis to test on your own tasks. **Tags**: #xref #articleseed  ### 🧠 Atomic Split Pad #### [P1] Cost per task beats price per token **Claim** (one sentence, Courtney's voice): Price per token misleads for agent work, so I compare models on cost per completed task, including cache reads and the extra tokens a cheaper model emits. **Evidence pointer**: [[#H1]] · [[#H3]] **Scope & conditions**: Holds for long-context agent loops that re-read a large cache. Breaks for one-shot prompts where output tokens dominate, and depends on the vendor's current cache-read rate. **Links**: - [[AI cost architecture]] — because extends - [[Reading vendor benchmarks]] — because contrasts on score-per-dollar charts versus tokens-per-task - [[Agent workflow cost tracking]] — because applies-in measuring spend from invoices  #### [P2] Route read-only research to the cheapest capable model **Claim** (one sentence, Courtney's voice): When a stronger model plans and a cheaper model gathers context, the pair can beat either alone on cost and time, but only if I measure it on my own tasks. **Evidence pointer**: [[#H4]] **Scope & conditions**: Applies to read-only analysis subtasks. Doesn't apply when the subtask needs design taste or high-stakes judgment, and the evidence here is one benchmark judged by its author's automated panel. **Links**: - [[Orchestrator and sub-agent patterns]] — because extends - [[Effort levels as reasoning budgets]] — because contrasts on where cost control is applied - [[Large pull request triage]] — because applies-in splitting a giant change into landable pieces  ### ✏️ Fleeting Queue - [F1] Check Anthropic's pricing page for the real cache-read rate on Sonnet 5.5, Opus 5.5 and Fable 5.1, because the video's speech has 2x versus 2.5x and 60% versus 95% slips: https://www.youtube.com/watch?v=8WbW_n95wc4 #fleeting #inbox - [F2] Identify which models the transcript calls "Soul" and "Astra" before citing the ranking at 00:22:20 #fleeting #inbox - [F3] Open question: does a max-effort floor lower accuracy on my tasks, or only raise cost? #fleeting #inbox - [F4] Article seed: "Read the cache-read row" as a short guide to vendor pricing tables #fleeting #inbox #articleseed - [F5] Compute my own cache-read share from the last 30 days of agent invoices #fleeting #inbox ### → Things 3 (max 2 actions) - [ ] Verify Sonnet 5.5, Opus 5.5 and Fable 5.1 cache-read prices on Anthropic's pricing page before citing the video — research, 15 min #things - [ ] Compute cache-read share of my agent spend from the last 30 days of invoices — AI cost review #things  📄 **Full text:** [[Full Document Contents/Articles/OpenAI Should Be Scared of This One.md|OpenAI Should Be Scared of This One by Theo - t3․gg]] ## Highlights - [00:10:09] out to under 5% with fable. It's closer to 20% with Opus and it's closer to like 50 plus with Sonnet 5.5 because that number is disproportionately large when you look at the other numbers. It would have been really nice if they could have knocked cash read down to like 15 cents or god forbid 10 cents. That would have been insane. But they didn't. Which means that the costs for using this model don't end up being as much cheaper as you might think when you look at these numbers because normal input tokens are barely touched with agentic work because we tend to read from cash and output tokens are a very small portion of the overall costs. I feel obligated to call this all out because I've looked at the numbers for doing like forlike work with opus versus sonnet for real code tasks and sonnet [00:10:53] consistently comes out as if not more expensive than opus does. In fact, with the artificial analysis intelligence index, Sonnet 5.5 cost roughly the same as Fable 5.1 for realworld tasks. Flashbang warning since I know a handful of you like those. We're going to artificial analysis. Yeah, this is insane. Sonnet 5.5 is neck andneck with Fable 5.1 for the most expensive run they've ever done on artificial analysis. That should kill any reason to use this model entirely, right? Kind of. It does kill one thing. Give you a hint. It's one particular word you can see here. starts with M and ends in max. Don't use it. I'm trying to explain why I hate max mode so much, and I don't want to make this whole video about it, but I'll do my best to explain here. ([View Highlight](https://read.readwise.io/read/01m3psg1xj994btff0znypwwvj)) - Note: [H1] Cache-read share erases half-price model discounts - Tags: atom/literature/h1 - [00:11:36] Reasoning levels aren't really levels. You're not saying you should reason this much. They're budgets. They are allowing the model to reason up to a certain amount. So, when you set low, you're saying, "I want you to do this with a small amount of reasoning tokens." When you set medium, high, or x high, you're saying, "I'm okay with you using more reasoning tokens up to a certain point." But you're not necessarily increasing the amount of tokens used. I've had benchmarks where the gap between low and X high was like 5 to 8% tokens. Like it's not a big difference if the tasks are simple. But if the tasks are complex, then having the extra budget can absolutely help. So what is the issue with max? The issue with max is that it's not really increasing the ceiling for how many reason tokens are [00:12:18] allowed to be used. It's more so increasing the floor for how many have to be used. It's effectively telling the model it's not done until it does a certain number of reasoning tokens. And the result is that I've had benchmarks where from low to X high you get a 5% increase in token usage. And from X high to max you get, and I'm not exaggerating, a 1,500% increase in token usage. Literally 15 times more tokens for that one little bump at the end because you're no longer letting it be done with easy tasks. And this can often actually hurt performance because if the model is told to keep overthinking the thing, it's going to overthink the hell out of it. It's going to start second guessing itself and is going to start getting wrong answers ([View Highlight](https://read.readwise.io/read/01m3psg4f3wqcxvw0php9nwxzz)) - Note: [H2] Max effort sets a reasoning floor - Tags: atom/literature/h2 - [00:17:17] surprisingly good value that it's hard for me to justify using much else. I also can't help but notice that all of these are score to cost and none of them are score to number of tokens. There's a reason for that. If we hop back over to Cursors Bench and click tokens, you'll see that Sonnet does have yet another greatest of all time score, their token usage in Cursor Bench, where they did 271,920 tokens per task, putting it ahead of even Opus at 218K and Gemini 38 Flash at 162K. It's almost double the number of tokens to Gemini 38 at Flash, which is insane. It is more than 5x the number of tokens that GPT56 soul and also six Astra used. So keep that in mind. This [00:18:01] is not a token efficient model, which should be made up for by the speed, right? Cuz it's so fast. More bad news sadly. According to Open Router, the TPS for using Sonnet 5.5 through anthropic is around 94 tokens per second, which sounds insane when you're used to something like Astra going at 30. They also report Opus 5.5 at around 70. For what it's worth, for my numbers using things through the official subscriptions, I have seen around 100 TPS on average for Opus 5.5 and around 150 for Sonnet 5.5. So, it is faster, but it's also not very token efficient. And result here is that when I used it for something like fish slop, it ended up taking quite a bit longer than Opus did. Sonnet took 43 minutes to finish and Opus only took 36. And when you ([View Highlight](https://read.readwise.io/read/01m3psg712jgvv5kgqnxyy5d84)) - Note: [H3] Faster models can still finish slower - Tags: atom/literature/h3 - [00:21:38] Opus. Not necessarily the types of tasks I would send a model after, but absolutely the types of tasks that Opus and Fable would. Its strengths come as a tool for our other models to use. We shouldn't be calling Sonnet directly. It should be used as one of many things that Opus or Fable will orchestrate when it's trying to break up real complex work. I kind of made a bench for this for my Grock review where I was trying to find the strengths of Grock 4.7. And I did it by having all of these models do a really big deep audit of T3 code, specifically this giant orchestrator v2pr that's been iterated on for far too long. It's hundreds of thousands of lines of code. I gave all of these models a pretty detailed prompt asking them to break up all the things being [00:22:20] added to orchestrator v2 to propose a strategy where we can get these things landed into main sooner rather than doing it all through the single giant PR. And I found that at the time Astra was by far the best at doing these breakdowns. Fable was not great at it. Grock actually was outperforming Fable. And then Opus, Sonnet, and Soul didn't do particularly well. I have overhauled this since and updated the way that it is judged. And those changes have brought Fable up to a meaningfully higher score. Still below Opus and still far below Astro, which by far had the best plan on how to do these things. I still find that Astra is just like uniquely willing to dig into the details for things. So, it performed really well ([View Highlight](https://read.readwise.io/read/01m3psg9f5apvdvgtxvwhdr4s3)) - Note: [H4] A cheap model belongs under an orchestrator - Tags: atom/literature/h3