Fine-Tuning vs RAG  ## Metadata - Author: [[Changelog Master Feed]] - Full Title: Fine-Tuning vs RAG - Category: #podcasts - URL: https://share.snipd.com/episode/07a46953-cacb-40e4-9fea-f2e179a2b2b7 ## Document Notes ## Highlights - **Fine-tuning Misconceptions** - Fine-tuning LLMs is often misunderstood compared to fine-tuning diffusion models like Stable Diffusion. - Retrieval augmented generation (RAG) better serves use cases like customizing responses based on company emails without costly GPU fine-tuning. Transcript: Speaker 1 Oh, yeah. That's so good because it does feel like there are use cases that are becoming very clear on what LLM shine in and what they're not good at. And then there's also the stack that's forming. And we did a survey probably. That's another thing that we did in the community on top of all the fun other stuff. We did a survey and we surveyed people that are actually using LLM's or even not using LLM's. And we asked them why you're not using them and we went through a bunch of stuff or why you are using them. What are some big pain points? And it was becoming very clear that there are certain use cases that people are using LLM's for and we can get into that in a minute. And then there's also this stack that is forming. And the stack was probably the most interesting. I know you guys mentioned it with the A16Z article and Rajko is one of the authors of that and he helped me with the report when I wrote it too. So he's behind the scenes like moving the puppets, the puppeteer. The cool thing with the stack is you kind of have, if I break it down for those who haven't seen the diagram that we put together, it's like you have the foundational model. And then you have some kind of vector database, which is like the hero, the champion in this whole LLM scene. You've got, if you need to do some fine tuning or model building, you have that component to it, but we can get into that. I'm very opinionated about the fine tuning part and I'll tell you why in a bit. And then you have stuff like developer SDKs and this is, you know, like your Lum Index or Langchains. And then on top of that, you have like the monitoring or experiment tracking, prompt tracking, those kinds of things, like a port key, a prompt to my port prompt layer. There's all kinds that are coming out. And so what we didn't have in that moment that I think are starting to emerge more now. And I'm really excited about is like, how are people actually evaluating these models? And is it coming with your different tools? Or are you doing extra stuff on top of it? So that was the inspiration behind a whole nother survey that we're doing right now on evaluation. Yeah. I can say personally, I'm doing extra stuff on top. That's my like short and very short answer, which is of course much more involved. But yeah, I don't know what, what is your sense of that? Am I just intuition wise? Am I out of the norm or in the norm with that? No, you are completely in the norm. And I think the hardest part, and this is why we wanted you to do a survey around it is because this is one thing that is super unclear. Nobody really knows if they're doing it right. And they don't really know what the best practices are. And so you also don't really know what you're evaluating. Are you just evaluating the model? One thing's for sure. All these benchmarks are complete bull that we all know, right? That is very clear. How do you really feel about it? Yeah, and they are interesting in certain ways. But the types of evaluations that are going on there. So like I think they serve a place maybe, but they don't translate into like, OK, now I have this use case, right? If I take the model on the top of that leaderboard, I am very much not guaranteed to have like the quote, the best results for my use case. And I think that's what's confusing to a lot of people. 100%. That's exactly it is that these models and the use case that you have, who knows how it's going to match up against one another, right? And then it's not only that, but how are you monitoring or evaluating for toxicity or the ability for it to do this one thing that you care about? I mean, I don't care if it's and also it just kind of feels like a lot of marketing at the end of the day when you see the newest model comes out and everybody loves Sota. Sota. This is, you know, state of the art. Sota beats chat GPT on all these different metrics. And I just kind of laugh because it feels like I've desensitized to that these days. Yeah, I think also it's kind of often kind of funny to me that even like chat GPT is being used as a static baseline for these things when it's not even like chat GPT isn't a model, right? It's a product that has layers on top of it for, you know, the handles, all sorts of things. And so that's another misconception that I've seen is like, well, is it really fair to compare a models output to the output of a product that has a lot of kind of functionality built around It and with it. Yeah. And that also kind of lets people know when that's drawn out, it lets people know that, hey, the LOM here is not your application. There's this whole layer on top of it, which I know you were talking about retrieval based augmentation or retrieval augmented generation as we are gearing up for this episode. There's of course, like whatever your opinion about prompt engineering is, there is an engineering element to how you call these models and chain things together. And like you say, evaluate things, validate things, filter things for whether it be toxicity or sexuality or or ([Time 0:05:06](https://share.snipd.com/snip/5729fc33-ffd8-4744-8a65-e0c9786f3eb5)) - **Vector Databases as Stack Hero** - Vector databases are the central component in the evolving LLM stack for tasks like semantic search. - Developer SDKs and monitoring tools build on top of vector databases to form the orchestration layer of generative AI systems. Transcript: Speaker 1 LLM scene. You've got, if you need to do some fine tuning or model building, you have that component to it, but we can get into that. I'm very opinionated about the fine tuning part and I'll tell you why in a bit. And then you have stuff like developer SDKs and this is, you know, like your Lum Index or Langchains. And then on top of that, you have like the monitoring or experiment tracking, prompt tracking, those kinds of things, like a port key, a prompt to my port prompt layer. There's all kinds that are coming out. And so what we didn't have in that moment that I think are starting to emerge more now. And I'm really excited about is like, how are people actually evaluating these models? And is it coming with your different tools? Or are you ([Time 0:06:25](https://share.snipd.com/snip/0e4fa0b6-c92a-4df7-aaf6-27191bb65e77)) - **LLM Benchmarks Mislead Use Cases** - Benchmarks for LLMs often mislead since top leaderboard models aren’t guaranteed best for specific use cases. - Evaluations must consider use-case specifics like latency, toxicity, and required capabilities beyond raw benchmark scores. Transcript: Speaker 1 Also don't really know what you're evaluating. Are you just evaluating the model? One thing's for sure. All these benchmarks are complete bull that we all know, right? That is very clear. How do you really feel about it? Yeah, and they are interesting in certain ways. But the types of evaluations that are going on there. So like I think they serve a place maybe, but they don't translate into like, OK, now I have this use case, right? If I take the model on the top of that leaderboard, I am very much not guaranteed to have like the quote, the best results for my use case. And I think that's what's confusing to a lot of people. 100%. That's exactly it is that these models and the use case that you have, who knows how it's going to match up against one another, right? And then it's not only that, but how are you monitoring or evaluating for toxicity or the ability for it to do this one thing that you care about? I mean, I don't care if it's and also it just kind of feels like a lot of marketing at the end of the day when you see the newest model comes ([Time 0:08:13](https://share.snipd.com/snip/636da729-3ef7-45a9-be4d-c94c8f24e9c9)) - **Keep LLM Chains Simple** - Simplicity helps debug complex LLM stack issues like empty outputs. - Sometimes writing direct Python logic for chains and exception handling yields more reliable results than using complex frameworks. Transcript: Speaker 1 Like if you import like whatever chain from Langchain and then whatever model, right? And then whatever vector database and then like you put it all together, you run the thing and then you get like an empty string output, right? Like you're saying like, where is where did it go wrong? Right. And how do you like divide and conquer debug that? Right. So I think a lot of what I've just in my own applications that I'm working on for clients and other things. A lot of times, to be honest, for me, it's a lot simpler to write out my chain of my LLM reasoning in just regular Python logic, add like whatever exception handling I want, like make the Call to the vector database, like manually and like create some logic around that. So I feel that that sort of Python DIY side, it's like less convenient. But I usually end up getting there still, you know, again, it's part of the maturity of this field, I guess, and partly like how things are advancing quickly. Speaker 2 You know, they'll have a fix for that. You know, in the spirit of you may have heard, like, you know, when you have a problem with Facebook, what's the answer? More Facebook? Well, in that spirit, I'm sure Meta will put out debug llama for you in no time. And it will solve all of your problems right there. It's funny too, because it feels ([Time 0:13:32](https://share.snipd.com/snip/80ebaf72-3e11-41e2-8a81-e960c8e94172)) - **When to Fine-Tune Models** - Fine-tune models only when teaching fundamentally new tasks or outputs not known by the base model. - Use RAG for personalizing responses based on domain documents without expensive and complex data collection and labeling. Transcript: Speaker 1 I think this is the time for the hot take officially declaring this practical AI and friends episode. So it's a safe space to declare your hot take. Speaker 2 Safe space. I love it. Speaker 1 Well, I know I just think that. I hear a lot about fine tuning and I don't know if people who throw around the idea of fine tuning something really understand what you fine tune a model for, especially like LLMs. If it's a stable diffusion model, that's a whole different story. And I think sometimes these diffusion models, they give us the wrong idea of what fine tuning in LLM will do. So undoubtedly you saw the rise of like lensa or photo AI or all of these fine tuning. Basically what these companies were doing in the background is they were running some kind of diffusion model and you would upload your selfie or your selfie or a picture of your dog And you would be able to bring it into the world of AI art. And if you take that concept over to LLMs, you think, oh, well, if I just fine tune an LLM on all of my emails, then the LLM will know how to write emails like me. But it's not like that. There's the misconception that it's not like equal in that regard. Fine tuning. You don't fine tune something so that it can understand you more and you can call it out and say, now, right? Like Demetrius because what you want to fine tune for for that case, let's just be clear. That's where retrieval augmented generation shines because you just say, hey, here's a database or a vector database of all of Demetrius's emails and the most you can do some few shot Prompting and say, right like this, here's like five styles of Demetrius writing a response to this. So make a sixth one and you're golden. You don't need to go through like burning a lot of cash on GPUs and GPUs are scarce these days to fine tune some model that may or may not work after you fine tuned it. So I think it's worth calling out like when you should be fine tuning things because I don't I don't want to say like never fine tune it. I just want to say like I've seen a lot of people talking about it and also a lot of companies starting that will tote how easy it is to fine tune and how you should be fine tuning and use your Company's data to fine tune. It's like fine tuning. I think the best way that I heard it talked about was in a recent MLOps community podcast that I had with Shao Les, he created Raga's, which is like an evaluation framework for RAGS. That's awesome. By the way, go check it out. But he's also very big in fine tuning and he is also one of the main people that does the OpenInstruct project, which is a whole open source LLM project. And so he was mentioning how you want to fine tune when you have some new function or some way, some new output, something that you need to teach the LLM to do that it doesn't necessarily Know how to do. So a perfect example of this is OpenAI's functions. So like chat GPT or GPT for functions. This makes it much easier for you to get very, very clean data or data that's outputted in a certain way, right? And it gives you the structured data. And that is a whole reason that you would want to fine tune something. Or the other example, I think is, uh, llama, the code llama. But I think where code llama falls down is that if the original base model doesn't see a lot of examples of code, then no matter how much you fine tune it, you're not going to get a good coding Model out of it. And some people who are using code llama and loving it, like come tell me because I haven't found anybody yet. So that's my rant about fine tuning versus RAGS. And I have the last thing that I will say is like if you're looking at fine tuning and you've been brainwashed by the society out there or the community at large and you think you need to Do it, like, let's just remember why ML was so hard before LLM's like collecting data is not so easy labeling data, cleaning data. ([Time 0:16:34](https://share.snipd.com/snip/73dccb75-275a-44ea-b42e-1f98b9392113)) - **Avoid Fine-Tuning on Raw Text** - Don't fine-tune LLMs on raw company documents for question-answering tasks. - Instead, prepare extensive instruction prompt datasets with questions and answers for effective fine-tuning. Transcript: Speaker 2 An LLM. Speaker 1 So I've been in countless conversations where the idea is, oh, while we're wanting to do like question answering on our documents or something like that, right? And we would love to fine tune a model on our internal company documents, and then it's going to be better at question answering. So what I generally tell those people is like, hey, like when you're fine tuning one of these models just on that raw unstructured text, in the best case scenario, what you're creating Is a better auto complete model to auto complete your type of documents, right? What you're not doing is creating a better question answering model. Those are two way separate things, right? So if you wanted to fine tune a model like that, I'm not like you're saying there may be cases where that's useful. Speaker 2 Maybe it's a domain thing. Maybe it's a very like you want very structured type of output answers like you're talking about. Speaker 1 But in that case, what you want to do, you actually don't want to fine tune just on that raw text data. You want to create your own set of instruction prompts likely that are fed in with various questions and you have the answers and you have everything like set up and you have thousands And thousands of these examples, which that I think when you frame it that way, then people are like, oh, so it's like a lot of work to create that sort of data set. And yeah, like turns out it still is, right? ([Time 0:22:00](https://share.snipd.com/snip/3841a50b-b79a-4866-832d-c209098700bf)) - **Learn RAG Step-by-Step** - Take the free, self-paced MLOps Community course to build a retrieval augmented generation pipeline. - Learn to preprocess data, ingest into vector databases, and produce accurate semantic search with LLMs. Transcript: Speaker 1 Because I need to point a couple of my coworkers to it. Well, if you just want rants from me about how you shouldn't fine tune, then you can find us on the MLOps community podcast. But the easy one is we have learn dot ML ops dot community. And that should get you to the like the learning section of our website. And yeah, as I said, it's exciting. It's a little bit like nerve-wracking because we plan on doing two styles. This one that we just released is go at your own pace, you get it, and then you can go through the lessons. And hopefully you can get your company to pay for it because it's within the learning budget. And it actually is for work, right? So that should be an easy sell. But if your boss is not interested in paying for it, just DM me and I will give you some amazing copy that you can send to your boss. A nice little email that will hopefully convince them to change their mind. But the other pieces that we're going to start doing like cohort based courses. ([Time 0:29:42](https://share.snipd.com/snip/c4cf1798-3dda-4b36-8cc4-4e3b467f70c4)) - **Company Size Influences LLM Choice** - Small startups and very large companies tend to avoid OpenAI for LLM use, whereas mid-size companies adopt it more. - This reflects differing priorities on speed, privacy, and in-house resource capabilities at different company sizes. Transcript: Speaker 1 I'm really fascinated by how many people are using open source versus using open AI. And one of the most hated visuals of the whole report is this one on like page 10. And it's talking about who's using open AI and what size they are. And people were like, this does not explain anything. What you're trying to say and the visual do not match up at all. And so again, I am not gardener. I am one random guy who has never written a report, never did a survey. But for some reason, I felt compelled to do this three months ago, five months ago now. And I almost regret it. But seeing the final product, I'm very happy that I did it. And so yeah, you should be proud. There's always things to look back on, but yeah, it's a nice report. Speaker 2 It's a good one. Exactly. Speaker 1 And we were able to move fast on it. And so I'm sure gardeners going to put something out soon. But that's the beauty of the community that we can move a little bit faster. Anyway, back to this visual that people did not like. And I mean, like a lot of people did not like it. So the whole idea was, are you using open AI? And we found that there's a bit of a correlation between people that are in super small startups, like zero to 50. ([Time 0:38:27](https://share.snipd.com/snip/fda8bb0b-fac2-4f30-8b1e-75d0b0ef67a3)) - **Startup Speed Beats Vendor Lock-In** - AngelList’s lead for LLM adoption prefers quickly incorporating many LLM features with OpenAI for speed. - He postpones concerns about vendor lock-in to focus on rapid product feature rollout first. Transcript: Speaker 1 And he's running the LLMs at AngelList. And I asked him, hey, so are you worried about that vendor lock-in type thing because you're only on OpenAI? Have you messed around with even just anthropic or cohere? And I thought his answer was fascinating because he told me, look, you know what? There are so many other pieces of surface area that I would like to cover, so many other features that I would like to implement with these LLMs and be able to use in our product that if I'm Stuck on one feature and trying to figure out what the best model is, then that's going to slow me down. I just want to go and get as many features plugged in as possible. And I know that chat GPT works really well. And so I'm just going to go as hard as I can and incorporate these features because I have a laundry list of them that I want to do. And then once I get all of that out of the way, then I can start going back and saying, okay, let's figure out, should we bring a model in house? ([Time 0:45:35](https://share.snipd.com/snip/b0ff904d-f346-463b-a28b-5bb8df60bc88)) - **LLMs Democratize Product Innovation** - Lowering the entry barrier with LLM APIs allows product people to rapidly innovate with AI features. - This knockdown of traditional ML complexity fuels a surge in creative AI-powered product ideas. Transcript: Speaker 1 What he has to say. Well, for those that are just listening, I am wearing tie-dye. So it is all peace and love here. And I am excited because right now, anyone who wants to mess around with machine learning and AI, they can. I have seen so many scenarios where a product person has said, you know what? Let's try and throw some AI with this. And they've been able to create an enormous amount of value for their company by adding some features that just call chat GPT or cloud or whatever. And that for me is really enticing because the barrier to entry has just been destroyed, you know, it almost was like the last couple of years, the first couple of years of the MLOps community. I mean, we still are. There's still a lot of traditional machine learning where I put it in quotation marks because it's not that old, but there's a lot of really hard stuff happening with quote unquote traditional Machine learning. But with the advent of LLMs, a lot of that has become really easy. And so all these NLP tasks that were really hard up until 2022, they're not as hard. And so you're seeing the creativity of people being able to put LLMs into their products. I love that. That is something that now I didn't realize how much I enjoyed product and the idea of speaking with product people until chat GPT came out because now I'm like, oh, I want to talk to more Product people. The product owners are really great people to talk to because they have these wild ideas and they know how to figure out if this is actually a success or not. So there's that piece and dovetailin on that. So we're having another LLMs in production conference on October 3rd. ([Time 0:49:46](https://share.snipd.com/snip/470db3c8-4bc5-436a-b62c-90412afc631d))