What Comes After Attention? This Startup Says It Already Knows.  🔗 **Source:** [What Comes After Attention? This Startup Says It Already Knows.](https://share.snipd.com/episode/43d7a883-a29d-4363-9b8f-c127fc800a15) ## Highlights - **Casual MeetinG Sparked The Interview** - The conversation began casually when the host ran into Alex Whedon in San Francisco and noted Subquadratic had just launched from stealth. - This set a relaxed tone and framed the interview as an on‑the‑street follow-up to earlier reporting. Transcript: Host (Interviewer) I'm walking around here, San Francisco, and I run into Alex Whedon of Subquadratic. And I wrote about you guys a few weeks ago when you launched your, well, really launched a company, I think, right? Yep, came out of self. Yeah, yeah. So tell me a little bit of what you actually do, because I think the name gives it away, but not everybody will get the name, I think, right away. ([Time 0:00:00](https://share.snipd.com/snip/64580e4d-3873-4e3a-ac34-3dc54714ef17)) - **Linear Scaling Enables Million Token Contexts** - Subquadratic's models change the scaling law from quadratic to near-linear by using sparse attention, making very long contexts (millions of tokens) computationally feasible. - Alex Whedon claims SubQ achieves up to ~1,000x compute savings at extreme 12 million token windows by avoiding redundant token-to-token comparisons. Transcript: Alex Whedon It away, but not everybody will get the name, I think, right away. Sure, yeah. I mean, so we are a foundation model company that fundamentally believes that the model architectures that we're using today are not going to scale. And that's with regards to compute, memory, and sample efficiency. Transformers have quadratic scaling laws. You double the input size, you quadruple the compute that you need instead of simply 2x. And so we have built LLMs that scale linearly. So instead of getting increasingly less efficient as they see more data, they remain equally efficient. And that enables us to process a lot more data with the same compute during both training and imprints. Explain that a little bit. Why? Because everyone is using the transformer model right now. Host (Interviewer) Why are you doing something different and why are the others not doing that? Sure. I mean, there are a lot of reasons. One is it's very hard to do well. Alex Whedon It's not a novel idea to use sparse attention or subquadratic algorithms, but it is novel to do so without quality trade-offs, in particular on long context retrieval. And we've made a show that we can do long context retrieval up to 12 million tokens even. And with significant computer reductions, almost 1000x that scale. ([Time 0:00:20](https://share.snipd.com/snip/0eed768e-d840-404f-bf33-417b2b8699b6)) - **Most Token Relationships Are Unnecessary** - Subquadratic's Sparse Selective Attention (SSA) selects only important token relationships instead of comparing every token pair, reducing redundant computation inherent in transformers. - They show over 99% of token pair relationships have low 'tension' scores, so dynamic selection preserves quality while cutting cost. Transcript: Alex Whedon Other models that are out there right now that everybody's using right now? Sure. So in the case of subquadratic sparse detention specifically, which is one of a couple model architectures we've worked with, the idea is that not all of the token relationships matter. Token relationship compute is why you see this quadratic scaling law. There are 1,000 possible two token relationships in a 1,000 token input. Explain that. Oh, sure. So the reason why, so Transformers are like a brute force approach to the problem of text modeling. You could do a simple cheat pass over all the tokens individually, or you could say, I'm going to compare every single individual token to every other possible individual token. That's what Transformers do. Very brute force, very naive, kind of assumes that, you know, the first needs to look at the second, third, the fiftieth, and the five thousandth. And it's just, it just doesn't, like, that's not how humans read text right right so uh reality is to say you know at times our minds go back to like earlier portions of of a book that we read Host (Interviewer) As we were going along but it's not an equal repetition of everything because not every word is as important as every other word is basically what it comes down to right yes in particular Alex Whedon Also not every word is as important to every other word in comparison. Yeah, yeah. Because that relationship is kind of what a model, you know, dense model. Host (Interviewer) Exactly. Model. Alex Whedon Yeah, in fact, that's part of how what we're doing is dramatically different than RAG. Because with RAG, you're discarding entire tokens, like entirely. They're not getting attended to it all. They're not being seen by the model at all. In our case, every token in the text is being seen by the model um but it's just not being like redundantly compared to every other token of the text how can you still get a good result with Host (Interviewer) That though because you're almost picking random not really random relationships but it's you're missing out on something, probably at some point, if you're not modeling every relationship. Alex Whedon Yeah, it's a good question. I mean, so empirically, we can show that even in full temps and tension, the relative importance of most tokens, of over 99% of tokens, is like very low. Tension scores are below 0.1. We actually show this in our model car. Like, clearly, we're just wasting compute most of the time. And in fact, we're actually maybe making the modeling task harder. ([Time 0:02:10](https://share.snipd.com/snip/e6a72fc7-467a-4843-9225-6d15c66b7592)) - **Selection Mechanisms Can Hide Quadratic Cost** - Dynamic selection methods can themselves be expensive if implemented with quadratic components; Subquadratic built a selection mechanism that avoids reintroducing quadratic scaling. - Benchmarking with GLM 5.2 showed selection dominated prefill latency (58% at 1M tokens), motivating a cheaper selector design. Transcript: Alex Whedon They show that you could dynamically select relationships without a significant quality tradeoff. However, they did so by redundantly using a smaller but still full attention model that ends up using the vast majority of the compute at scale. In fact, we recently did some benchmarking with GLM 5.2, found that at one million tokens, 58% of the pre-filled latency or the the reading latency, the import reading latency, comes From that selection mechanism. So that selection mechanism, which is supposed to be seen as cheap, actually dominates the compute because that's a quadratically scaling component. ([Time 0:05:00](https://share.snipd.com/snip/ec5b2a18-c6fd-435c-a948-6b448f48b816)) - **Long Multi-Million Token Pretraining Extends Context Generalization** - Pretraining on very long multi-million-token sequences extends a model's ability to generalize mid-context to extremely long post-training contexts. - Subquadratic published ablations showing multi-million token pretraining improves long-horizon performance, an underreported technique. Transcript: Alex Whedon So we talk about pre-training and how really long multi-million token pre-training actually extends the model's ability to generalize mid-context post-training to super long Context versions of that task. That's an insight we haven't seen published anywhere. Nobody's talking about multimillion token pre-training. And so we published some of these insights in our model card. And it took a while for us to A, to finish the level of ablation that we would have been comfortable with like publishing about some of these insights. ([Time 0:09:19](https://share.snipd.com/snip/536d1ff8-8949-44b8-82d4-966282661d62)) - **Small Model Punches Above Its Size For Enterprises** - SubQ 1.1 Small is under 100B parameters but competes with mid-tier models on capabilities while being cheaper and faster for long-context tasks. - Alex positions this as an enterprise wedge for processing massive datasets without frontier-model costs. Transcript: Alex Whedon B, because we will release larger models and we are working on larger models. Host (Interviewer) How small is small, did you say? Alex Whedon What we posted publicly was fewer than 100 billion parameters. I would venture to say that our model is smaller than any of the models offered by Open AI Anthropic. But our next model will not be so. It's going to be a relatively decent size and will be a significant leaping capabilities from the small model. Host (Interviewer) SubQ, just to put a point on that, it's kind of in the, some benchmarks on the Sonnet 4.6 kind of range. Sometimes a bit more, sometimes a bit less, you know, not Opus, not GPT 5.5, but pretty good. Alex Whedon Exactly, but with cost and size that is smaller than Haiku or GPT 5.1 Mini. And speed. Exactly. Yeah, so that's, so we think that's a pretty interesting enterprise offering. We've seen a lot of people in the enterprise space talking about using the mid-tier models as opposed to the front tier for large data processing tasks, which is exactly where we're Trying to plug in. ([Time 0:10:46](https://share.snipd.com/snip/24528aa3-2036-455e-aff4-7399fd0c4f50)) - **Pursuing Zero Attention For Better Compression** - Subquadratic is researching 'zero attention' or non-attention architectures that use more continuous, compressed representations inspired by world models. - The goal is better information compression and new objectives to surpass transformers' limits in efficiency and capability. Transcript: Host (Interviewer) Because everything we're building right now is based on the attention paper after all, right? All you need is attention. Alex Whedon What is zero attention yeah so um attention is almost kind of like rag in a way because you have queries keys and values to represent um information about tokens let's just do that over Oh stop airplane yeah that's gonna be possible Cool. Do you want to just start with your answer? Sure. I'll plug in some graphic in the middle. Awesome. So a lot of people are familiar with RAG, so it may be helpful to talk about attention kind of in a similar way. So attention is kind of similar to RAG, in that you have queries, keys, and values that represent information about the tokens that you're processing. And you are kind of like creating this graph of this information and enabling the model to retrieve information about the tokens to generate the output. So there's this discreteness of representation, where everything is represented within these nice little boxes. And that's super convenient, it's easy to build a brute force solution around that, but it also means that your ability to compress information is limited. So if you had a more continuous way, an abstract way of representing the information, then you could actually compress the information further which means you can either mix like smaller Models etc or you could just scale things up again to create another leap in intelligence what would that abstraction look like um so i would say like the stuff that we're doing takes A lot of inspiration from uh world models okay uh we're not talking about the video modality in this case, but some of the stuff Gian Lacoon is talking about, rethinking how to represent Long-range dependencies, how to keep a long-range state, how to rethink the objective function modeling, things like that. ([Time 0:12:00](https://share.snipd.com/snip/572d8110-b407-429c-af66-bcd9fd2409b7)) - **Use Enterprise Partners To Validate New Models** - Validate novel model architectures with enterprise design partners before broad release to align features, safety, and product-market fit. - Subquadratic is onboarding enterprises with large token spend to co-shape product and red-team behavior before general access. Transcript: Alex Whedon Totally. Yeah, we could have maybe messaged the launch a little bit differently. There was some debate about how we were going to message it when we did first launch. Product versus research announcement. So we're giving access to the model to design partners now, and these are mostly enterprises, largely with eight to nine figure token spend. This is a core market that we really care about. It has been since day one. I spent a lot of time working with these partners before I started this company. And so I know how we can create wedges in these applications. And the idea here is to validate with them that we're doing what people want to see from a model of our type and the applications where we want to plug in. And then use that as validation that we're ready to roll out to a broader audience. A lot of the people that we're partnering with are strategic, interested in the long-term trajectory of what we're trying to do. And so they're great partners to help us kind of like co-shape the products. ([Time 0:14:31](https://share.snipd.com/snip/95c0e250-bf9e-4e7c-925c-516cda9dcf57)) - **Optimize For Topology Not Just GPUs** - Remain hardware-agnostic while optimizing for topologically aware training; current NVIDIA GPUs work but other chips with different SRAM/DRAM tradeoffs may be advantageous. - Subquadratic evaluates multiple chip topologies to maximize efficiency across platforms. Transcript: Alex Whedon So, I mean, obviously, so our modeling is being on NVIDIA chips today. I think that even transformers and NVIDIA chips are like not the best pairing today. Like transformers are not trained in a very, very topologically aware way. And so we think that there are ways to innovate as well with regards to topologically aware training. And so we're evaluating which topology is best for us. So there are chip types other than NVIDIA that offer differentials in SRAM, DRAM relative to flops, but are quite advantageous. And so I think the future is open for us on like what, which, like what the distribution of chip platforms is to use. ([Time 0:16:09](https://share.snipd.com/snip/c4605f89-9507-478d-92e7-89de55a96510)) - **Stage Releases To Capture Feedback And Safety Signals** - Roll out access in stages: demos, more design partners, limited individual access, then general release to capture feedback and harden safety. - Subquadratic plans staged demos and limited individual releases before broad availability while increasing design partnerships. Transcript: Alex Whedon Yeah, so we will, we're going to start to release more demos. A lot of people have been asking for that, and we want to show people in practice what sort of difference long, cheaper, longer, cheaper, faster, or higher intelligence context can Make within an application or a use case, what types of use cases and locks that are not currently being addressed. So that's next. We're going to incrementally increase the number of design partnerships that we're doing. And then before we do a general access release, we'll also do a limited access release that's not necessarily enterprises or even startups, but rather individuals, but on a smaller Scale. ([Time 0:17:04](https://share.snipd.com/snip/93017d96-4803-4c67-acb0-2082186494b5))