Fine-Tuning Language Models with Reinforcement Learning with Michael Albada

Fine-Tuning Language Models with Reinforcement Learning with Michael Albada 🔗 **Source:** [Fine-Tuning Language Models with Reinforcement Learning with Michael Albada](https://www.youtube.com/watch?v=TUynF91MKAU) > [!note]+ Document Notes > tags: [fine-tuning, reinforcement-learning, language-models, machine-learning] > author: Michael Albada > source: YouTube > reference: https://www.youtube.com/watch?v=TUynF91MKAU > date created: 2024-12-19 12:00 > > # 🏷️ Title: Fine-Tuning Language Models with Reinforcement Learning >  ## 🧰 Action Inbox - [H1] Evaluate open-weight models for your use case — Consider cost vs performance tradeoffs #evaluation - [H2] Implement reinforcement learning with verifiable rewards for function calling — Use Glaive dataset as starting point #implementation - [H3] Set up fine-tuning pipeline with production feedback loop — Mine failure modes from production data #optimization ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ### 🔗 References - [Glaive Function Calling V2 Dataset](https://huggingface.co/datasets/glaiveai/glaive-function-calling-v2) - [Hugging Face Transformers Library](https://huggingface.co/transformers/) ## 📝 Summary - Michael Albada presents fine-tuning with reinforcement learning as a method to improve generative AI reliability - Open-weight models are rapidly closing the gap with proprietary models while offering significant cost advantages - Small models (<4B parameters) have dramatically improved, narrowing performance gaps with larger models - Fine-tuning becomes cost-effective at scale and provides strategic independence from API providers - Reinforcement learning with verifiable rewards enables precise control over model behavior for structured tasks ## 📖 Main Content The presentation argues that while proprietary models like GPT-4 and Gemini lead performance benchmarks, open-weight models are lagging by only months in frontier intelligence. This creates opportunities for organizations to achieve near-frontier performance at dramatically lower costs through fine-tuning. The cost of intelligence is approaching zero as small models improve and catch up to larger ones. Fine-tuning offers strategic benefits including model ownership, privacy, security, and the ability to optimize for domain-specific tasks that may not be priorities for large research labs. The speaker emphasizes that fine-tuning works best when combined with production feedback loops to continuously improve model performance on real-world failure modes. ## 🗒️ Notes #### ⏱️ [00:00](https://youtu.be/TUynF91MKAU?t=0) — [H1] Introduction to Fine-Tuning for AI Reliability - Key idea: Fine-tuning is a powerful but challenging way to improve generative AI reliability - Evidence: "Fine-tuning is a really powerful way to do that. Now, it's not the easiest way, but it is one of the most robust and most reliable" #### ⏱️ [02:15](https://youtu.be/TUynF91MKAU?t=135) — [H2] Open-Weight Models Are Rapidly Improving - Key idea: Open-weight models lag proprietary models by only months, not years - Evidence: "The openweight models are never that far behind and they're really lagging by just months on frontier intelligence" #### ⏱️ [04:30](https://youtu.be/TUynF91MKAU?t=270) — [H2] Top Open-Weight Models Compete in Top 10 - Key idea: Open-weight models now place in top 10 performance leaderboards - Evidence: "GPT OSS 120 billion and Quen 3 235 billion are now placing in the top 10" #### ⏱️ [05:45](https://youtu.be/TUynF91MKAU?t=345) — [H1] Small Models Are Catching Up Dramatically - Key idea: Models under 4B parameters have tripled performance since March - Evidence: "These tiny models, less than four billion parameters... were scoring at a level of 15... Look at these leaps up to 40" #### ⏱️ [08:20](https://youtu.be/TUynF91MKAU?t=500) — [H2] Performance Varies Widely by Use Case - Key idea: Expensive models may underperform cheap ones for specific applications - Evidence: "You can spend a lot of money for a really expensive model that doesn't work all that well for your use case" #### ⏱️ [10:10](https://youtu.be/TUynF91MKAU?t=610) — [H1] Strategic Trade-offs: Proprietary vs Fine-Tuned - Key idea: Proprietary models offer convenience but create strategic dependence and commoditization - Evidence: "If all of the companies are depending on a small number of research labs... there's very little differentiation" #### ⏱️ [12:30](https://youtu.be/TUynF91MKAU?t=750) — [H2] Fine-Tuning Enables Model Ownership and Control - Key idea: Fine-tuning provides privacy, security, and performance optimization for specific domains - Evidence: "It allows you to own your own model... get outperformance on your tasks" #### ⏱️ [14:45](https://youtu.be/TUynF91MKAU?t=885) — [H2] RAG vs Fine-Tuning Trade-offs - Key idea: RAG works for dynamic context, fine-tuning for repeated instructions and reasoning - Evidence: "If you find yourself continually adding similar information... move over to fine-tuning" #### ⏱️ [16:20](https://youtu.be/TUynF91MKAU?t=980) — [H3] LoRA Reduces Fine-Tuning Complexity - Key idea: Low-rank adaptation makes fine-tuning more efficient by updating fewer parameters - Evidence: "We freeze the vast majority of the weights in the model and we only learn that low rank adaptation" #### ⏱️ [18:00](https://youtu.be/TUynF91MKAU?t=1080) — [H2] Quality Over Quantity in Training Data - Key idea: A few dozen to hundred examples can dramatically improve domain performance - Evidence: "Just a few dozen to a few hundred examples can really dramatically improve performance on a domain" #### ⏱️ [20:15](https://youtu.be/TUynF91MKAU?t=1215) — [H1] Reinforcement Learning with Verifiable Rewards - Key idea: RL enables precise control over model behavior through grading every output - Evidence: "This is a way for us to tell the model this was correct and this is wrong" #### ⏱️ [22:30](https://youtu.be/TUynF91MKAU?t=1350) — [H3] Group Relative Policy Optimization - Key idea: GRPO is more efficient than PPO by eliminating the need for value models - Evidence: "We're able to do away with the value model and instead we compute multiple rollouts" #### ⏱️ [24:45](https://youtu.be/TUynF91MKAU?t=1485) — [H3] Function Calling Example Implementation - Key idea: Compound reward functions can optimize for JSON validity, correct functions, and parameters - Evidence: "Creating valid JSON outputting the correct function for the current context and additional rewards for each of the arguments" #### ⏱️ [26:00](https://youtu.be/TUynF91MKAU?t=1560) — [H2] Watch for Reward Hacking - Key idea: Models may find ways to achieve high rewards without solving intended problems - Evidence: "Always watch out for rapid spikes in the reward which can indicate reward hacking" #### ⏱️ [27:30](https://youtu.be/TUynF91MKAU?t=1650) — [H1] Fine-Tuning as Competitive Moat - Key idea: Major model providers use fine-tuning internally despite promoting API-only approaches - Evidence: "When it comes time for them to design and ship their own product experiences, most of the time they're doing additional fine-tuning" ## 📚 References - [Glaive Function Calling V2 Dataset](https://huggingface.co/datasets/glaiveai/glaive-function-calling-v2) - [Hugging Face Transformers Library](https://huggingface.co/transformers/) - [Building Applications with AI Agents - O'Reilly](https://www.oreilly.com/) ## 🗣️ Paraphrase - Open-source language models are quickly approaching the performance of proprietary models while offering significant cost savings and customization opportunities - Organizations should start with API-based proprietary models but consider fine-tuning for strategic independence, cost optimization, and domain-specific performance improvements - Reinforcement learning with verifiable rewards provides precise control over model behavior, making it especially valuable for structured tasks like function calling and code generation ## 📰 Newsletter 🚀 **Fine-Tuning: Your Secret Weapon for AI Reliability** The AI landscape is shifting dramatically! While everyone focuses on the latest GPT releases, open-weight models are quietly closing the gap—lagging by just months, not years. Small models under 4B parameters have **tripled** their performance since March alone. **Why This Matters for Your Projects:** - Strategic independence from API providers - Dramatic cost reductions at scale - Domain-specific performance optimization - Enhanced privacy and security control **Getting Started:** Begin with proprietary APIs for rapid prototyping, then consider fine-tuning when you find yourself repeatedly adding similar instructions or need better performance on specialized tasks. **Pro Tip:** Mine your production failures for training data—it's one of the most effective ways to improve model reliability over time. Resources: [Hugging Face Transformers](https://huggingface.co/transformers/), [Glaive Function Calling Dataset](https://huggingface.co/datasets/glaiveai/glaive-function-calling-v2), [Michael's Book on AI Agents](https://www.oreilly.com/) ## ✅ Action Items #### DevRel - Education: Create workshops on fine-tuning vs API trade-offs, LoRA implementation tutorials - Developer experience: Develop fine-tuning starter templates and evaluation frameworks - Community management: Host fine-tuning office hours, curate open-weight model recommendations #### OSPO - Software Inventory: Catalog open-weight model dependencies and licensing requirements - Security Management: Establish guidelines for fine-tuning with production data - Developer Support: Provide training on model evaluation and selection criteria #### Community - Contributors: Guide fine-tuning contributions to model improvement projects - Maintainers: Support governance for community-driven model training initiatives ## 🧪 Hypotheses - Organizations using fine-tuned open-weight models will achieve 70% cost reduction while maintaining 90% performance of proprietary APIs - Production feedback loops in fine-tuning will improve task-specific performance by 40% within 6 months - Small model performance improvements will enable 80% of AI applications to run locally by 2025 - Fine-tuning adoption will accelerate when cost savings exceed 10x for high-volume applications ### 📊 Methodology - Compare fine-tuned vs API costs across different query volumes and model sizes - Measure performance improvements from production feedback integration - Track open-weight model benchmark progression vs proprietary model releases ### ❓ Questions - At what query volume does fine-tuning become cost-effective for different model sizes? - How can organizations systematically identify when fine-tuning will outperform prompt engineering? - What governance frameworks are needed for responsible fine-tuning with user data? ### 🔮 Future Research - Automated fine-tuning pipeline optimization based on production metrics - Cross-domain transfer learning techniques for specialized applications - Efficient methods for combining multiple LoRA adapters ### 💡 Implications - Practice: Teams need fine-tuning evaluation capabilities alongside prompt engineering skills - Policy: Organizations require data governance for model training and deployment - Security: Fine-tuning creates new attack vectors requiring specialized security measures - Education: AI curriculum must include model training alongside API consumption - Funding: Budget planning needs to account for infrastructure vs API cost trade-offs ### 🧠 Learning Styles Assessment Reading level estimate: Flesch-Kincaid Grade Level 12 #### 👀 Visual learners - Present: Performance charts, cost comparison graphs, architecture diagrams - Improve: Fine-tuning workflow visualizations, decision tree for model selection #### 🔊 Aural learners - Present: Full video presentation with clear narration - Improve: Audio summary of key decision points, podcast-style discussion #### 🗣️ Verbal learners - Present: Clear definitions of LoRA, GRPO, technical terminology with examples - Improve: Analogies for reinforcement learning concepts, story-driven examples #### 🧩 Kinesthetic learners - Present: Code examples, Hugging Face implementation guides - Improve: Interactive Jupyter notebooks, hands-on fine-tuning lab exercises #### 🧠 Logical learners - Present: Performance benchmarks, cost-benefit analysis, algorithmic comparisons - Improve: Decision matrices for model selection, ROI calculators #### 🧑‍🤝‍🧑 Social learners - Present: References to community datasets, collaborative training approaches - Improve: Fine-tuning study groups, peer review processes for model evaluation #### 🧘‍♀️ Solitary learners - Present: Self-contained code examples, independent evaluation criteria - Improve: Self-assessment rubrics, personal fine-tuning project templates #### 🌿 Naturalistic learners - Present: Evolution of model capabilities over time, ecosystem interactions - Improve: Model lifecycle thinking, sustainability considerations for training ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Clear heading structure, descriptive chart alt text - Improve: Detailed descriptions of performance graphs, table summaries #### Keyboard navigation - Present: Standard video controls, accessible slides - Improve: Transcript with timestamp navigation, keyboard-friendly code examples #### Contrast and readability - Present: High contrast slides, readable font sizes - Improve: Ensure chart colors meet accessibility standards #### Text resizing and reflow - Present: Scalable slide content, responsive code formatting - Improve: Test at 200% zoom, ensure no content clipping #### Media and captions - Present: Full video with clear audio - Improve: Complete transcript, visual description of charts #### Forms and interactions - Present: Standard video interface - Improve: Accessible code exploration tools, interactive examples ### 🌍 DEIB Assessment #### Cultural breadth - Present: Technical focus with global applicability - Improve: Examples from diverse industries, multilingual model considerations #### Inclusive language - Present: Gender-neutral technical language, defined jargon - Improve: Plain language summary, avoid assumptions about infrastructure access #### Neurodiversity support - Present: Structured presentation, consistent formatting - Improve: Clear section breaks, summarized key points #### Representation - Present: Technical presentation focused on methodology - Improve: Diverse use case examples, varied industry applications #### Psychological safety - Present: Encouraging tone about starting with simple approaches - Improve: Acknowledge learning curve, provide beginner-friendly entry points ### ✅ Logical Validity #### Strong reasoning - Deductive strength: Logical progression from cost analysis to fine-tuning recommendations - Inductive strength: Performance trends support future improvement predictions - Causal reasoning: Clear links between training approaches and performance outcomes - Analogical strength: Effective comparisons between LoRA and full fine-tuning #### Fallacies - Appeal to authority: Relies heavily on benchmark performance without discussing limitations - False dichotomy: Presents proprietary vs fine-tuned as binary choice rather than spectrum - Hasty generalization: Small model improvements may not apply to all domains equally #### Tone of voice - Overall tone: Enthusiastic and encouraging while maintaining technical precision --- ## Full Transcript [00:00] I am happy today to be presenting on [00:02] fine-tuning models with reinforcement [00:04] learning. And I think one of the main [00:06] desires here is to find ways to improve [00:09] the reliability of our generative AI [00:11] applications and systems. And [00:14] fine-tuning is a really powerful way to [00:17] do that. Now, it's not the easiest way, [00:20] but it is one of the most robust and [00:22] most reliable, and it's one that I think [00:24] is going to be receiving more attention [00:26] moving forward. So, that sets us up [00:28] great. And let's go ahead and dive in. [00:32] So I want to start with a little bit of [00:34] context. So open weight models are [00:37] improving rapidly and so much of the [00:40] excitement over the last couple years [00:42] has been led by these incredible [00:44] research labs releasing better and more [00:47] compelling frontier models month after [00:49] month. And we're seeing incredible leaps [00:52] and bounds on a wide range of [00:54] benchmarks. And while there's a gap [00:56] between benchmarks and our real world [00:58] applications, we are seeing improvements [01:00] in hands-on applications as well. But [01:04] there's a quieter story that's also [01:06] happening that I don't think is [01:07] receiving enough attention and I think [01:10] has very profound implications. [01:13] We have seen openweight models also [01:16] improving by absolute leaps and bounds. [01:20] And when I say open weight, I mean [01:22] models that have been trained and the [01:25] full artifact has been released publicly [01:28] onto the internet. They are free for you [01:31] to download and for you to host, you to [01:33] serve. And you can see on this chart [01:36] that even though the proprietary models [01:38] have stayed out and ahead, the [01:41] openweight models are never that far [01:44] behind and they're really lagging by [01:45] just months on frontier intelligence. [01:48] And so when you add that up over time [01:50] and you consider that compounding, this [01:53] means that we have access to [01:55] nearfrontier performance and [01:57] intelligence for an absolute fraction of [02:00] the cost. The other incredible benefit [02:03] to this is these openweight models are [02:05] accessible for us to be able to [02:07] fine-tune. So you can start with a [02:10] really high level of performance and [02:13] then apply fine-tuning on top of that. [02:18] And [02:19] these openweight models are highly [02:21] competitive and I'll admit they are not [02:24] the top performing models today. And so [02:26] this is pulled from a very recent uh [02:28] leaderboard. And you can see that GPT 5 [02:30] and Gro 4 and Gemini 2.5 still dominate [02:33] the leaderboard. But I want to [02:35] illustrate the GPT OSS 120 billion and [02:39] Quen 3 235 billion are now placing in [02:42] the top 10. So this is an incredibly [02:45] competitive space and while we [02:48] anticipate the proprietary models to [02:50] continue to lead and continue to really [02:52] push the frontier, we should expect [02:55] these openweight models to also continue [02:57] to improve. And so this puts an [02:59] incredible amount of intelligence at our [03:01] fingertips and we really are approaching [03:03] a point where the cost of intelligence [03:06] is plummeting and is gradually [03:08] approaching towards zero. [03:12] And the other story here is that small [03:15] models are catching up. So yes, the best [03:18] performing openweight models I just [03:20] showed you are 120 billion parameters [03:22] and 235 billion parameters. Those are [03:25] large. Those are not easy to host. [03:28] But I'd like you to take a look at this [03:31] chart and you will see that the gap that [03:33] we used to see between these large and [03:36] small models has really closed. And the [03:41] improvements that we've seen on these [03:42] large models over the last couple years [03:44] is really impressive. But look at what's [03:47] happened since even March of this year. [03:50] These tiny models, less than four [03:53] billion parameters, used to be really [03:55] hard to work with, right? They were [03:56] scoring at a level of 15 on this [04:00] composite intelligent index. Look at [04:02] these leaps up to 40. That's really [04:05] where the largest models were performing [04:08] in January of this year. So that gap is [04:11] narrowing and this is a whole sequence [04:13] of of innovations that are continuing to [04:17] drive this. We're getting better at [04:18] distillation. We're getting better at [04:20] quantization. And as we are continuing [04:23] to advance the frontier models, we're [04:25] finding better ways to then extract that [04:28] and condense that into smaller and [04:30] smaller form factors. And this is [04:31] exciting. This is moving towards more [04:33] intelligence that we can deploy on the [04:35] edge on mobile devices on laptops and so [04:39] expect this trend to continue. [04:42] Now this does not mean that we're [04:45] getting to universal high performance [04:48] across our models and that your price [04:51] performance and value may vary widely. [04:54] And so this is my buyer beware slide. [04:57] You can spend a lot of money for a [05:00] really expensive model that doesn't work [05:02] all that well for your use case. You can [05:05] also find really affordable models that [05:08] will work really well for your use case. [05:10] And similarly with the openw weight [05:12] models, I'm not saying that all of them [05:14] are good. Many of them are not. And so [05:17] you do need to be a little cautious. But [05:19] if you are willing to pay attention and [05:22] track the leaderboards and take some of [05:23] the most promising models and assess [05:25] them on your use case, you can find high [05:28] performing models that will work very [05:30] well for your problems and your context [05:33] at a tiny fraction of price. And so this [05:36] is good. This is going to put more [05:38] competition on the labs to be able to [05:41] keep providing higher quality at lower [05:43] price. And remember that benchmarks do [05:46] not tell the whole story. and many of [05:48] the labs are perhaps overoptimizing for [05:51] the benchmarks. So make sure that you [05:53] evaluate for your use case. You might [05:55] actually see widely different uh [05:57] performance across the different models [05:59] due to different training recipes and [06:01] different training data. So evaluation [06:04] will be your friend. [06:06] So let's stop and think a bit about [06:09] choosing a proprietary versus going [06:12] after a fine-tuned model. So I recommend [06:15] you always start with proprietary [06:18] models. We now have access to frontier [06:20] intelligence with a simple API call. You [06:23] don't need to worry about deployment. [06:24] You don't need to worry about training. [06:26] And what this is doing is enabling [06:28] greater capabilities and intelligence [06:30] for a much wider range of teams and [06:33] companies. You no longer need a giant [06:35] machine learning organization and all of [06:39] that infrastructure that it used to take [06:41] to be able to do machine learning at [06:43] scale. This is bringing intelligence and [06:45] making it much more accessible. [06:47] But it does come with some costs. It [06:49] comes with this strategic dependence and [06:53] you do at least get lower costs at [06:55] scale. I'd also like to refer back to a [06:58] fantastic point that Chip Wyn made in [07:00] her opening keynote that if all of the [07:04] companies are depending on a small [07:06] number of research labs and a small [07:08] number of APIs, [07:10] there's very little differentiation. [07:12] That's a commoditization layer. And so, [07:15] yes, you can do additional work with [07:17] context engineering and rag to better [07:19] personalize those results. But if [07:22] increasingly the model is the product, [07:24] then choosing to take that significant [07:27] of strategic dependence, choosing to to [07:30] accept that degree of commoditization is [07:32] a really important strategic question [07:34] that each organization needs to think [07:36] through carefully. [07:38] Now, making that decision to move over [07:40] to fine-tuning. It comes with a lot of [07:42] benefits. It allows you to own your own [07:45] model. It allows you to adjust and [07:47] modify it to make sure that it works [07:49] well for your use case. It comes with [07:52] some ongoing training responsibilities [07:54] and deployment overhead of course, but [07:56] you also get more privacy and more [07:58] security if you handle it appropriately. [08:01] Uh it also allows you to get [08:03] outperformance on your tasks. It's [08:05] possible that you're working on [08:06] something that is not that important to [08:09] the big research labs. If you need [08:11] better performance on a domain specific [08:13] language, if you need better performance [08:15] on your particular area, going to [08:18] fine-tuning can allow you to really [08:21] focus and drive performance on that task [08:23] that's more important to you than anyone [08:25] else. It gives you much greater control [08:27] over knowledge and style and it can give [08:31] you lower costs at higher scale. Again, [08:34] if you're starting out small number of [08:36] API calls, stick with the big models. [08:39] But as you start feeling the pain of [08:42] cutting larger and larger checks out to [08:44] the big model providers, then it might [08:47] be time to start making the investment [08:49] into openw weight models and fine-tune [08:52] models that might work better for your [08:54] use case. [08:56] Now, let's talk about the difference [08:58] between rag and fine-tuning. Rag is a [09:01] fantastic technique because it allows [09:03] you to adapt very quickly. It allows you [09:05] to bring in your dynamic context, your [09:08] dynamic information, and it allows you [09:11] to pull quick API calls, say pull in [09:14] information about that particular case, [09:16] that particular customer. It allows you [09:18] to pull in real time data. And so that [09:21] adaptability is really great and it [09:23] comes with no upfront additional [09:25] training investment. It's great for [09:27] lower query volume and it's also very [09:29] good for knowledge heavy tasks. allows [09:31] you to augment the domain the knowledge [09:34] and awareness that is uh been trained [09:37] into those model parameters. [09:40] But if you find yourself continually [09:42] adding similar information, similar [09:45] instructions, um I think this is [09:48] probably an underappreciated aspect of [09:50] these uh language model and generative [09:52] AI applications and systems is in [09:54] practice I see prompts grow over time. [09:58] we tend to add more instructions, more [10:00] context, more details. We're not as good [10:03] at going back and cleaning up and [10:05] removing that information. And so the [10:07] longer your prompt gets, that's more [10:10] computation, that's higher cost, it's [10:11] higher latency. [10:13] If you move over to fine-tuning, you can [10:15] take that information that you want to [10:17] add every single time and through [10:20] fine-tuning, update those parameters and [10:23] really bake that additional information [10:25] more deeply into your model. And so this [10:27] works much better with higher volume and [10:30] when you need to reason more deeply over [10:32] that information that you've pulled. [10:34] So where exactly is this cost break [10:37] even? It depends on a few factors. It [10:39] depends on the size of the model that [10:42] you need. That depends on the complexity [10:43] you're working with. I know Nicole spoke [10:45] at some length about choosing large [10:47] versus small models. Fine-tuning works [10:49] for any size and scale, but of course [10:52] it's going to be faster, it's going to [10:53] be cheaper, and it's going to be easier [10:55] to do this on smaller models. And then [10:57] it's going to be more affordable to [10:58] serve. And one of the benefits of [11:00] working with fine-tuning is allows you [11:03] to really find and calibrate which model [11:05] is going to work best for your use case [11:07] and it's going to give you that optimal [11:09] performance. [11:12] One other big algorithmic innovation [11:14] that's come through recently is low rank [11:16] adaptation. So rather than needing to do [11:18] full fine-tuning on every single [11:21] parameter in your model which measures [11:23] in the billions, we can use this [11:26] technique uh called Laura where we use a [11:30] a dimensionality reduction technique and [11:33] effectively we add in this uh matrix [11:37] approximation. So we freeze the vast [11:40] majority of the weights in the model and [11:44] we only learn that low rank adaptation. [11:46] It's a much smaller number of parameters [11:48] that we can tune over. So it's more [11:50] computationally efficient and you can [11:52] also uh efficiently [11:55] host and serve multiple Laura adapters. [11:58] And so you can even reduce your cost for [12:00] serving multiple models this way. But [12:03] it's a really useful uh technique that [12:05] allows you to get more performance with [12:07] lower costs. And of course it's tunable. [12:09] So that inner dimension what uh [12:12] dimension you are uh shrinking down to [12:16] you can keep that higher in order to get [12:18] better performance or you can make it [12:20] smaller to uh to further compress and [12:22] make it faster to train. [12:24] So thinking about your data requirements [12:27] let's say you've now decided that you [12:28] want to move to fine-tuning. What I want [12:30] to say is the volume that you need to [12:34] fine-tune has come down. Now [12:35] pre-training of course requires an [12:37] enormous amount of information and data. [12:40] Supervised fine-tuning at the really [12:41] large scale also requires quite a bit of [12:44] work. But now when you want to adapt in [12:46] just your particular domain, quality is [12:50] much more important than quantity. All [12:52] things being equal, more quantity will [12:54] help. But I've been very surprised to [12:56] see how much just a few dozen to a few [12:59] hundred examples can really dramatically [13:01] improve performance on a domain that [13:03] that model is not great at working with [13:05] in the beginning. The other thing is if [13:08] you can mining from production is a much [13:11] more uh effective manner. I've heard few [13:14] a few of the presenters talk about this [13:16] earlier of how do you close that [13:18] flywheel? How do you close that loop? [13:20] How do you learn from failure modes? [13:22] Well, this is a great opportunity to [13:23] extract out those cases that are not [13:25] working, really specify what type of [13:28] behavior you hope to see, and then send [13:30] those back in for fine-tuning. This only [13:32] works for some cases. I know certain [13:34] domains are very sensitive about [13:35] training, but if you work in a domain [13:37] where you can feed those samples back [13:40] in, can be a really powerful technique [13:42] to allow you to solve this problem at [13:44] the model level rather than having to [13:46] spend the work tweaking prompts and uh [13:50] trying to to maintain that overtime. [13:54] So many of the big model providers work [13:58] through a few different training steps [14:01] and I'm going to move a little quickly [14:02] here. I want to make sure I uh I uh save [14:05] us some time. But typically, you need to [14:08] go through pre-training, instruction [14:09] fine-tuning, preference fine-tuning, and [14:11] reasoning fine-tuning. And that [14:13] reasoning fine-tuning is typically with [14:16] reinforcement learning. And this is the [14:18] technique I'm going to focus on because [14:20] it is the one that has led to some of [14:22] the most significant advancements. If [14:24] you think back to that significant leap [14:27] up we saw around last year from the [14:29] proprietary model that came from 01 that [14:31] came from the shift of adding in a [14:34] greater based on verifiable rewards. So [14:36] supervised fine-tuning is great if you [14:39] have a clean set of input output pairs. [14:41] It's a great place to start. When you [14:43] shift over to reinforcement learning you [14:46] now need a grader. You now need an [14:47] ability to assess every single type of [14:50] output that comes from the model during [14:52] training. But if you can do that grading [14:54] effectively, it allows you to really [14:56] extract a lot more information. You get [14:59] more juice from every training sample. [15:01] Comes at more computational cost, but [15:03] allows you to typically build more [15:05] robust models that can handle a wider [15:08] range of inputs more reliably. [15:12] Now, preference tuning is still [15:14] incredibly important and reinforcement [15:16] learning with human feedback and direct [15:18] preference optimizations are fantastic [15:21] uh techniques to do that. Those tend to [15:24] be better when you have training data [15:26] that consists of multiple outputs and a [15:29] preference coming uh from typically your [15:31] users of A is better in this case, B is [15:34] better in this case. When we move to [15:37] reinforcement learning with verifiable [15:38] rewards, this is a way for us to tell [15:41] the model this was correct and this is [15:44] wrong. Now, this is incredibly powerful [15:46] for shaping the outputs and for ensuring [15:51] that our model is actually solving the [15:53] problem in the correct way and in the [15:55] way we want it to be solved. It's [15:57] incredibly useful for code generation. [15:59] It's great for structured outputs. It's [16:01] incredibly good for tool use. Uh, [16:03] Chipwin also mentioned that choosing [16:06] which functions to call and how to use [16:08] tools is a valuable use case. That sets [16:11] me up very nicely. That's exactly the [16:12] problem that I'm going to focus on here. [16:16] And in reinforcement learning with [16:18] verifiable rewards, we take our training [16:20] data and we pass those into our current [16:22] model here. That's uh represented as the [16:25] policy box. From that model policy, we [16:29] generate multiple completions, also [16:31] known as rollouts. We grade every single [16:33] one of those and that goes through this [16:36] verifiable reward box. It returns zero [16:39] if it's a failure or alpha which is some [16:41] reward if uh if otherwise. Now you [16:45] probably want a scale of multiple [16:48] types of rewards. You can use binary but [16:51] having intermediate rewards can really [16:53] help with more stable training. I [16:55] sometimes get asked if you need an upper [16:58] limit if this has to be zero to one. I [17:00] typically use zero to one because that's [17:02] easy for me to reason about, but given [17:04] the way the the optimizer works, it [17:07] really is unbounded because the rewards [17:10] are relative. So if you want to use much [17:12] larger numbers, that's actually fine and [17:14] won't interfere with the training [17:15] process. [17:17] We typically use group relative policy [17:19] optimization. This is a relatively new [17:21] but much more efficient new algorithm [17:24] for doing reinforcement learning based [17:26] training compared to the prior version [17:28] of a proximal policy optimization which [17:31] came from more traditional reinforcement [17:34] learning for optimizing sequences using [17:36] a value model to help compute the [17:39] advantage. With group relative policy [17:42] optimization, we're able to do away with [17:46] the value model and instead we compute [17:48] multiple rollouts and then we compute [17:50] the advantages uh relative to the other [17:53] rollouts. And so typically you need a [17:55] few more rollouts for this but you're [17:58] saving trainable parameters and you're [18:00] saving memory which is really important [18:02] for fine-tuning these these larger [18:04] models. [18:06] So I'll walk you through an example [18:07] here, but I've taken something called [18:09] the Glaive function calling V2 data set. [18:12] This is one of the best function calling [18:13] data sets out there right now. And it [18:15] comes along with a sequence of [18:18] interactions with user interacting with [18:20] the assistant with ground truth examples [18:22] of what function should be used, what [18:25] parameters it should be used with. So I [18:27] took this data set and I implemented a [18:30] greater uh based on uh reinforcement [18:33] learning with verifiable rewards and [18:35] I've put together a compound uh greater [18:38] function that operates over [18:42] uh creating valid JSON outputting the [18:44] correct function for the current context [18:47] and additional rewards for each of the [18:50] arguments and then some additional [18:51] penalties if it's missing a function [18:53] call at all. It's not particularly [18:55] complex, but I think what this [18:57] illustrates is it gives you a lot of [18:59] control to specify exactly what behavior [19:02] you're looking for and then to shape [19:05] those model outputs to get exactly the [19:07] reliability that you determine that that [19:10] you need for your use case. But you [19:12] still get to benefit from the the vast [19:16] number of parameters and the deep [19:18] ability to understand context that these [19:21] language models come with. I then uh [19:23] specify our training configuration here. [19:25] I'm using the hugging face transformers [19:27] library. It's a a great library. It's [19:30] entirely open source and my source code [19:32] also includes some scripts in order to [19:34] set up the environment and then uh train [19:36] your model. Here is my uh reward curve [19:39] and you can see we go from very poor [19:40] performance at the very beginning to [19:42] getting consistently uh answering the [19:45] function calls as well as getting most [19:46] of the parameters correct over time. [19:48] Always watch out for rapid spikes in the [19:51] reward which can indicate reward [19:53] hacking. And this is the model finding a [19:55] way to get a lot of reward without [19:57] actually solving the model with solving [19:59] the problem that you intend. Don't be [20:01] surprised if it happens. These models [20:02] are really good at getting high rewards [20:05] even if it's going around your [20:07] intention. I will uh skip over the [20:10] hyperparameters here just in the [20:11] interest of time. Well, what I'll leave [20:13] you with is always start with uh [20:17] proprietary models and start with [20:18] context engineering and rag. It's [20:20] faster, it's easier, it gets you to a [20:22] product experience quickly. But really [20:25] consider moving over to fine-tuning as a [20:27] way to improve performance on your use [20:29] case, reduce cost, and give you more [20:31] control. Remember, the big model [20:33] providers are telling us to only trust [20:36] them and rely on their models and their [20:38] APIs. But when it comes time for them to [20:41] design and ship their own product [20:43] experiences, most of the time they're [20:45] doing additional fine-tuning to get [20:46] better performance and better results [20:48] because they know that's an important [20:49] part of their moat. And so I encourage [20:51] all of you to think about what your moat [20:53] is on your team and think about the data [20:57] and the training process and the greater [20:59] all of this is something that can help [21:01] give you better performance and it's [21:03] something that can create enduring value [21:06] and create that better performance over [21:08] time. a new better model comes out. [21:10] Great. Now you can start with that [21:12] higher foundation and still pass it [21:13] through that same training regime that [21:15] you've built. [21:17] And one last thing, if you've enjoyed [21:19] this presentation, my book building [21:22] applications with AI agents, uh this is [21:24] my first book is now available on the [21:26] O'Reilly platform and it is also [21:28] available on Amazon.