How We Teach AI Models  🔗 **Source:** [How We Teach AI Models](https://twitter.com/leerob/status/2080467752897146898/?rw_tt_thread=True) > [!note]+ Document Notes > --- > tags: [model-training, reinforcement-learning, ai-evaluation, model-specifications, agent-memory] > author: "Lee Robinson" > source: "https://x.com/leeerob/status/2014651429009101109" > reference: "01kya53zhyh08qrjejce9ecmtg" > date created: 2026-07-24 > --- > # 🏷️ Title: How we teach AI models > ## 🧭 Table of Contents > Summary; main content and notes; references; paraphrase; newsletter; action items; hypotheses; methodology; questions; future research; implications; learning, accessibility, DEIB, logic, DARVO, conclusion; refactor appendix; template library. > ## 🔗 References > - Source: https://x.com/leeerob/status/2014651429009101109 > - Reader document: 01kya53zhyh08qrjejce9ecmtg > ## 📝 Summary > AI systems become useful through layers: pretraining supplies broad knowledge, supervised fine-tuning demonstrates desired answers, and reinforcement learning rewards behavior that meets a model specification. Held-out evaluations test whether behavior generalizes. Current conversational memory is normally stored outside model weights in rules, skills, and retrieved context. > ## 📖 Main Content > The source compares a model to a colleague who needs knowledge, examples, practice, and clear expectations. A model specification states behavioral goals. Supervised examples teach response patterns; reinforcement learning supplies outcome feedback. Objective benchmarks work when answers are known, while open-ended work often uses rubric-based model judges. The author separates intelligence from alignment and notes that live chats do not retrain the model. > ### 🗒️ Notes > - Claim: Model specifications externalize desired behavior so teams can inspect and revise it. > - Mechanism: Held-out evaluations separate generalization from memorized examples. > - Verification focus: Current agent memory often lives outside model weights in rules, skills, and retrieved context. > - Scope: This analysis covers the captured source body. It does not treat linked claims as independently verified unless the source supplies direct evidence. > ## 📚 References > - Primary source: https://x.com/leeerob/status/2014651429009101109 > - Follow linked primary documents before using numerical, legal, medical, or policy claims in publication. > ## 🗣️ Paraphrase > Models first absorb broad patterns, then copy good demonstrations, then practice against reward signals. Tests must stay outside training data. A capable model can still be unhelpful if its behavior does not match the product’s rules. Persistent agent behavior therefore depends on external instructions and memory stores. > ## 📰 Newsletter > **Why this matters for WordPress:** AI systems become useful through layers: pretraining supplies broad knowledge, supervised fine-tuning demonstrates desired answers, and reinforcement learning rewards behavior that meets a model specification. Held-out evaluations test whether behavior generalizes. Current conversational memory is normally stored outside model weights in rules, skills, and retrieved context. For teams building open web software, the useful move is to convert the source’s central claim into a small test, publish limits, and keep the people affected by the decision in the review loop. [Read the source](https://x.com/leeerob/status/2014651429009101109). > ## ✅ Action Items > - DevRel: turn claims into testable demos; OSPO: track governance and dependency effects; Community: invite affected groups into review; WordPress — Core: assess platform impact; Core Performance: measure cost and latency; Core AI: define model-specific tests; Core Program Team: coordinate milestones; Mobile: test small-screen use; Playground: build a reproducible demo; Community: gather contributor feedback; Training: create a lesson; Marketing: state limits; Support: collect failure patterns; Design: review workflows; Documentation: record behavior and caveats; Photos: not applicable unless visual assets are needed; TV: consider a short explainer; Polyglots: plan translation context; Accessibility: test WCAG 2.2 AA; Meta: review site and API effects; CLI: expose repeatable checks where relevant; Hosting: test deployment constraints; Tide: add automated quality checks where applicable; Openverse: review open-asset implications; Themes: test presentation effects; Plugins: publish compatibility guidance. The WPOCC: consider community-governance effects. FAIR: note implications for federated open-source collaboration. > ## 🧪 Hypotheses > 1. If model specifications externalize desired behavior so teams can inspect and revise it., teams will observe a measurable change in decision quality or failure rate. > 2. If teams test the mechanism in a bounded pilot, results will differ by task complexity and user expertise. > 3. Publishing limits beside outcomes will reduce overgeneralization in later decisions. > ## 📊 Methodology > Use a preregistered comparison with a baseline, clear unit of analysis, task-level outcomes, failure categories, cost and time measures, and interviews with affected users. Preserve raw inputs and report null results. For a short announcement, first obtain primary documentation and reproduce the stated behavior. > ## ❓ Questions > - Which claim would change the decision if it proved false? > - What is the correct unit: task, job, team, organization, or ecosystem? > - Which groups bear costs that the source does not measure? > - What evidence would distinguish adoption from dependable outcomes? > ## 🔮 Future Research > Replicate across organizations and user groups; compare short- and long-term effects; measure distributional impact; test alternative explanations; and track whether early gains survive operational use. > ## 💡 Implications > Practice: use bounded trials. Policy: require traceable evidence for high-impact claims. Security: inspect failure paths and dependencies. Education: teach evaluation, not tool loyalty. Funding: tie continued spending to measured outcomes and public limits. > ## 🧠 Learning Styles Assessment > - Visual: add a claim-to-evidence diagram. > - Aural: discuss the argument and counterargument aloud. > - Verbal: rewrite the thesis in plain language. > - Kinesthetic: run the proposed pilot. > - Logical: map premises, mechanism, evidence, and conclusion. > - Social: compare interpretations in a small group. > - Solitary: keep a prediction and error log. > - Naturalistic: examine resource and environmental effects. > - Reading level: approximately grade 10–12; technical or institutional terms need definitions. > ## 📊 Accessibility Assessment > - Perceivable: provide alt text, captions, sufficient contrast, and text equivalents (WCAG 1). > - Operable: support keyboard access, visible focus, and adequate target sizes (WCAG 2). > - Understandable: use clear headings, definitions, and consistent navigation (WCAG 3). > - Compatible: use semantic markup, valid names, roles, and values (WCAG 4). > - Cognitive: shorten dense passages and expose the argument structure. > - Motion and media: avoid forced motion; provide controls and transcripts. > - Testing: combine automated checks with keyboard, screen-reader, zoom, reflow, and cognitive review against WCAG 2.2 AA. > ## 🌍 DEIB Assessment > - Cultural breadth: identify the source’s geographic and institutional scope. > - Inclusive language: separate groups from stereotypes and avoid universal claims. > - Neurodiversity support: provide summaries, headings, examples, and predictable structure. > - Representation: seek evidence from people affected by the proposed change. > - Psychological safety: invite dissent without requiring public disclosure of harm. > ## ✅ Logical Validity > The layered training explanation is coherent and distinguishes capability, evaluation, and alignment. It simplifies contested details such as data quality, reward hacking, judge bias, and post-deployment adaptation. The colleague analogy is explanatory, not proof. > - Strong reasoning: the source offers a clear thesis and at least one mechanism. > - Fallacy check: watch for hasty generalization, selection bias, false cause, motive attribution, and analogy used as proof. > - Fix: add comparison data, definitions, uncertainty ranges, and counterevidence. > - Tone: explanatory and persuasive. > ## 🛡️ DARVO Analysis > No complete DARVO sequence is established by th  How do AI models learn new skills and behaviors? The process is surprisingly human and easy to understand, even if you don’t have a machine learning background. #### Helpful colleagues AI models, increasingly used through agents, are similar to a helpful colleague. What traits do the best people you’ve worked with share? Maybe you thought of coworkers who were great communicators, or who had the judgment to know when to ask for help versus when to figure it out themselves. There are also more human qualities, like being personable, empathetic, and kind. But I particularly enjoy working with people who can take ambiguous problems and find ways to solve them. So how do we get a model to act like that ideal colleague? We need to first write down our version of the “correct” behavior. This document is called a [model specification](https://model-spec.openai.com/2025-12-18.html), [principles](https://ai.google/principles/), [constitution](https://www.anthropic.com/constitution), or similar. It's used both for internal alignment and as a guideline for testing model behavior (i.e. evals) during training. #### How do we get better at our jobs? My over-simplification is to try hard things, fail, learn, and get better through repetition. If you’re onboarding a new teammate, you don’t expect them to be productive on day one. They need to learn how to use your systems and tools, and how your team works together. Humans, with all our imperfections, are quite good at learning new skills (and remembering them). Let’s say you want to become the best basketball player in the world. Which approach should you take to improve your skills? 1. Read everything ever written about the game of basketball. We’re going to ignore being limited to 24 hours in a day. 2. Play 10,000 games of basketball. The same time-bending rules apply. You learn through trial and error, reviewing film, grading your decision making, and improving during practice. Knowledge and experience are two different stats, and ideally you want to max both. It reminds me of how people talk about “book smarts” versus “street smarts”. The best players have each. How do we learn things faster? With a coach! The ideal coach will watch you play, tell you what to fix, and always push you against your current limits. They’ll teach you how to make high-quality decisions when you’re on your own, nudging your brain weights toward better decisions over time. #### How do models learn? Models don’t learn exactly like humans, but human learning provides a useful analogy. These models start by learning to predict patterns from an enormous library of other people’s experience (i.e. pretraining). They actually *can* read every book on basketball! After pretraining, you have a base model. The model may be very “book smart” (e.g. it knows everything about ancient history) but it is imperfect and has some quirks. It doesn’t feel great to talk to and can make mistakes. You then show the model many examples of great behavior you want it to mimic (i.e. supervised fine-tuning, or SFT). Think about this like studying film of great players. This gets you far, but you can’t become great only by copying someone else’s moves. You need to watch the model do real work so you can help it learn and improve. To shape its behavior, you let the model play simulated games and reward it when it does well (i.e. reinforcement learning, or RL). This reward is like a coach’s feedback, where you tell the model if the decisions it made were good or bad, nudging it to improve with more attempts. [New research](https://alignment.openai.com/beneficial-rl/) also suggests that more generalizable qualities like truthfulness can be learned during RL. If you teach the model to be more honest when answering questions from one domain, values like truthfulness can generalize when answering *any* question. #### Grading the model We measure whether models are improving in two main ways: practice tests where they can review the answers, and final exams where they’ve never seen the questions before. It would be hard for humans to know if we’re improving at calculus, for example, without testing for understanding. If you take a test and fail, you know exactly where you need to improve, but this only works when the domain being tested (e.g. math) has an answer key. One of the main challenges for training large AI models is creating a diverse set of “tests”. These tests could be on anything from math to coding. If you can create an answer key, then the models can try the test many times and learn to improve. What about measuring intelligence across many different tasks? Think about AP exams, which cover subjects from calculus to history. Many of these exams combine multiple-choice questions scored by correctness with free-response questions graded against a rubric. This is similar to how we evaluate models with benchmarks. Some benchmarks measure objectively verifiable things (e.g. did the tests pass?) and others ask a separate model to grade the results using a rubric (i.e. LLM-as-judge). These results give you a reference to understand if the model is actually learning and improving during training. Critically, benchmarks are intended to test unseen or “held-out” tasks. Otherwise, it would be a bit like memorizing all the answers to your test, and then showing up and writing them down from memory, which isn’t a true measure of intelligence. #### Intelligence alone isn’t enough If you’re a genius, but rude and socially unaware, chances are people won’t like you. We can all probably think of someone we know like this. It’s not enough to just be smart! This is why aligning models to act “correctly” (according to the model spec we defined) is incredibly important. Imagine you had a friend who ended every conversation with some now-popularized LLM slop like “Honestly? That’s the tell”. You’d quickly get tired of talking with them. Engineers and researchers training models spend a bunch of time talking to a new model before it’s finished. They document all the quirks and places where it could improve. This could be overuse of stock phrases like “Bottom line:” or sacrificing clarity for a shorter bullet point, leading to some weird unintelligible language with overly complex words. With the issues identified, they can then work to train the model further and penalize bad behaviors, slowly leading to a model with more aligned behavior. Additionally, they can A/B test the model in production and measure whether users preferred responses from the new version and had better outcomes. #### Continual learning You might expect these models to improve and get smarter over time. But that’s not how they work today! Learning only happens when the model is training. Your conversations don’t update the model’s intelligence in real time. Instead, you need to write down things to remember, typically as rules or skills. Agents can then read these files and include them in context when prompting the model. #### This naive approach to memory through files has worked surprisingly well, but there’s more work to do before agents can learn and remember skills the same way a new colleague would. Teaching models to be helpful is a deep topic. Hopefully this high-level overview was interesting enough to inspire you to go learn more. If you want to see more explanations like this, or want me to cover another part in more depth, let me know!, How We Teach AI Models, Lee Robinson, 01kya53zhyh08qrjejce9ecmtg.