# How We Teach AI Models > [!question] Lingering Questions > - Robinson's "colleague" analogy explains capability layering but says nothing about reward hacking, judge bias, or data quality — where would those failure modes actually show up in his framework? > - If live chats don't retrain the model, how much of a product's perceived "learning" is actually just accumulating external memory (rules, skills, retrieved context) rather than model change? > [!tip] Article Ideas > - A short explainer translating "pretraining → SFT → RL → eval" into a WordPress-plugin-review analogy for a non-ML audience — the vault's recurring move of grounding AI abstractions in a domain readers already know. > [!link] Cross Reference > - [[3. Resources/Permanent Notes/Long-Horizon AI Shifts the Bottleneck From Model Capability to Memory and Effort Scaffolding]] — same claim from a different angle: capability lives partly outside the weights > - [[3. Resources/Permanent Notes/Raw Provenance and Distilled Guidance Serve Different Memory Jobs]] — extends the "memory lives outside weights" point into a design distinction > [!quote] Vocabulary > - *model specification* — a written statement of desired model behavior that a team can inspect, test, and revise, distinct from the model's weights themselves > - *held-out evaluation* — a test set the model was never trained or reinforced against, used to distinguish genuine generalization from memorization ## Bibliographic capsule Lee Robinson, X/Twitter thread (2026-07-24), a short explainer comparing model training to onboarding a new colleague: pretraining supplies broad knowledge, supervised fine-tuning demonstrates desired answers, reinforcement learning rewards behavior against a model specification, and held-out evaluations test whether the behavior generalizes. ## Author's thesis (as I read it) Capability and alignment are separable: a model can be broadly knowledgeable (pretraining) yet still behave badly against a product's actual rules unless that behavior is explicitly specified, demonstrated, and reward-shaped. Persistent "memory" in deployed agents is mostly an external-scaffolding problem, not a training problem — live conversations don't retrain the model. ## Key arguments (≤3) 1. **A model specification externalizes desired behavior** so a team can inspect and revise it, rather than leaving "good behavior" implicit in training data. 2. **Held-out evaluations are what separate genuine generalization from memorized demonstration** — without them, SFT examples could just be pattern-matched back. 3. **Deployed agent memory (rules, skills, retrieved context) is architecturally distinct from the model's trained weights** — a conversation changes what the agent can retrieve, not what the model knows. ## Claim atoms ### A model specification externalizes desired behavior for inspection and revision > [!quote] ID:rw_01kya53zhyh08qrjejce9ecmtg_a1 > "Model specifications externalize desired behavior so teams can inspect and revise it." Where/why: the thread's central framing move — behavior becomes a document, not a black box, once it's written into a spec that reinforcement learning is scored against. ### Held-out evaluations separate generalization from memorization > [!quote] ID:rw_01kya53zhyh08qrjejce9ecmtg_a2 > "Held-out evaluations separate generalization from memorized examples." Where/why: the mechanism claim explaining why eval sets must stay outside training data — otherwise "the model got better" and "the model learned to answer this specific test" are indistinguishable. ### Deployed agent memory usually lives outside the model's weights > [!quote] ID:rw_01kya53zhyh08qrjejce9ecmtg_a3 > "Current agent memory often lives outside model weights in rules, skills, and retrieved context." Where/why: the practical takeaway for anyone building on top of a model — persistence is an application-layer design problem, and a live chat does not change the underlying model. ## Permanent candidates - **A model specification externalizes desired behavior so a team can inspect and revise it** — new, distinct enough from the vault's existing memory-architecture notes to warrant its own atom. Promoted to [[3. Resources/Permanent Notes/A Written Model Specification Turns Desired Behavior Into an Inspectable Artifact]]. - **Agent memory lives outside the model's weights** — already covered by [[3. Resources/Permanent Notes/Long-Horizon AI Shifts the Bottleneck From Model Capability to Memory and Effort Scaffolding]] and [[3. Resources/Permanent Notes/Raw Provenance and Distilled Guidance Serve Different Memory Jobs]]. Connect, don't duplicate. ## Processing checklist - [x] Connect into the relevant MOC — [[3. Resources/MOCs/AI & the Future of Knowledge Work MOC]] - [x] Link existing memory-architecture Permanent notes rather than re-deriving them - [ ] If promoting the spec-as-artifact atom further, find a primary source (e.g., a published model spec) rather than relying on a single thread's framing ## Links - [[3. Resources/MOCs/AI & the Future of Knowledge Work MOC]] — the training/alignment layering this source describes is background for any claim about what AI can reliably do in a knowledge-work pipeline - [[3. Resources/Readwise/Tweets/How We Teach AI Models]] — source document