Al Engineering 101 With Chip Huyen  ## Metadata - Author: [[Lenny's Podcast: Product | Career | Growth]] - Full Title: Al Engineering 101 With Chip Huyen - Category: #podcasts - URL: https://share.snipd.com/episode/72193086-bbeb-4936-b7a1-7939dc322048 ## Document Notes ## Highlights - **Prioritize Users Over Hype** - Talk to users and iterate on feedback instead of chasing the latest AI hype. - Focus on data, prompts, UX, and reliability to improve AI apps more than swapping models. Transcript: Lenny Rachitsky So it's this very simple table you it of what people think will improve AI apps and what actually improves AI apps. What people think will improve AI apps, staying up to date with the latest AI news, adopting the newest agentic framework, agonizing about vector databases to use, constantly evaluating What model is smarter, fine-tuning a model. And then you have what actually improves AI apps, talking to users, building more reliable platforms, preparing better data, optimizing end-to workflows, writing better prompts. Why do you think this hit such a nerve with people? And just if you had to boil it down, what do you think people are missing about building successful AI apps? Chip Huyen What I'm saying that got asked a lot and a lot is that, how do we keep up to date with the latest AI news? And I'm like, why do you need to keep up to date with the latest AI news? And I was very kind to intuit it, but there's so much news out there. A lot of people also ask me questions like, how do I choose between two different technologies? Like maybe like recently, like MCP versus like agent to agents, right? Like protocol. And it was like, which one is better or like this or that? And I think it's a silly question. It's awesome. It's like, first, like if, how much of the improvement would you get? Like from like optimal solutions versus non-optimal solutions. Right. And sometimes they were like, actually it's not much. Right. And I was like, okay, if it's not much improvement, why do you want to spend so much time debating something that doesn't make much difference to your performance? And another question I asked is like, if you adopted a new technology, like how hard it would be to switch that out to another? And sometimes I was like, oh, I think it would be like a lot of work switching it out. And I was just like, hmm, let's say here's a new technology. It hasn't been tested by a lot of people. And if you adopt it, you would be stuck with it forever. Like, do you actually want to adopt it? Right. Maybe you want to think twice about, like, overcommit to, like, new technologies that hasn't been better tested. Lenny Rachitsky I love your just broader advice is just simple, like, talk to, to build successful apps, talk to users, build better data, write better prompts, optimize the user experience versus Just like, what is the latest and greatest? What's the best model to use right now? ([Time 0:04:52](https://share.snipd.com/snip/21303725-e2ae-4d3e-9581-3f47225f2dcd)) - **Pre-Training vs Post-Training** - Pre-training builds broad language statistics while post-training tailors behavior for tasks. - Fine-tuning is post-training and should be used when task-specific adjustments are necessary. Transcript: Lenny Rachitsky What's the simplest way for someone to understand? What is the difference between pre-training and post-training? And then just how fine-tuning fits into that, just what fine-tuning actually is. Chip Huyen Disclaimer, don't have like phone visibility into like on what like this big secretive like frontier labs are doing uh but right from what i heard right so so i think it's like one is um Like supervised fine tuning we have demonstration data and you have like a bunch of like um experts like okay here's a prop right and here is a what the answer should be like. And you just train it to simulate, emulate what the human expert would be like. And that's also what a lot of people with open source models are doing. They do it by distillation. So instead of having human experts to write really good sounding, great answers to prompts, They get very popular, famous, good models to generate a response to it and getting these Trained smaller models to emulate. So sometimes you see people just like, so that's because I really appreciate open source community, by the way. But going from being able to train models that can emulate an existing good model is very different from like being able to train a good model like an output for existing good model. So it's a big step there. So yeah, so like we have my supervised fire tuning. And another thing that's like very big, I'm not sure you have guests talking about it already, but like reinforcement learning is like everywhere. Lenny Rachitsky Let's pause on that because I would definitely want to spend time on that. And that's such a cool topic that's merging more and more in my conversations. But just to even summarize the things you just shared, which I think is really, really important stuff. So the idea here is a model, essentially this algorithm piece of code that someone writes and say the frontier models are feeding it just like the entire internet of content. And basically it's trying to test itself on predicting across all that data the next word. Essentially, token is the correct way of thinking about it, but a simpler way to think about it is the next word in text. And as it gets it wrong, it adjusts these things called weights, essentially. Is that a simple way to think about it, even though that's just very surface level? Chip Huyen So I think of language modeling as a way of encoding statistical information about language, right? So let's say that we both speak English. So we kind of get a sense of like what is more statistically likely. Like if I say my favorite color is, then you would say, okay, that should be another color. Like the word blue would be much more likely to appear than the word like uh end of table right because statistically blue is more like each capital infinite color is so so it's just like Get this um it's it's it's a way of encoding statistical information so like when language modeling ministry and a large amount of data like it you see a lot of languages a lot of domains. So it can tell, like, okay, you may say this sentence. Then it uses a prompt, and it would come, like, with the next most likely token. So, by the way, it's not a new idea. Actually, it's an idea that comes very, very old, like, from the 1951 papers, like English Entropy. I think it's like Claude Shannon. It's a great paper. And I think there's a story I really like. It's from, did you read Sherlock Holmes, by the way uh yeah i read a few sherlock holmes books yeah yeah so so this is a story of like when sherlock holmes was using this statistical information To like have shown a case so he was getting um so this is this story uh there is somebody left a message uh with a lot of like. So, Schroederholm was like, okay, he knows that in English, the most common letter is E. Then the most common stick figure must be E, right? And then he goes, he starts like that. He was really saw the code. So, I think that's language. So, in a way, it's like simple language modeling, right? But instead of like at a word level, he does it as like character level. And token is something in between, right? A token is not quite a word, but it's bigger than a character. So let's say we say token because it helps us like read what helps us reduce vocabulary because with character is like smallest amount of like vocabulary, right? So Amphibet has like 26 characters, but words can have millions and millions. Whereas tokens, you can be able to get the sweet spot between the two. So let's say that we have the new word, how to say, podcasting. Let's say it's a new word, but you division it podcast and ink so people understand okay podcast we know the meaning we know that ink is like uh like a verb like gerund whatever it is so we Know the word like podcasting so that's why the token comes in but yeah uh that's like the pre-tuning is basically like encoding statistical informations of language to help you predict What is most likely um i think that most likely is a more simple way of doing it because it's more like building a distribution of like okay so the next token could be like more like 90% of The time it could be like a color, like 10% of the time could be something else right. So it's basically distribution so language would like pick like depending on your sampling strategy like do you want it to always pick the most likely token, or do you want it to pick Something more creative? So I think my sampling strategy, I think, is something extremely important. It can have your boost of performance in a huge way, and very, very underrated. Okay, awesome. Lenny Rachitsky So essentially, a model is just code with this whole set of weights, essentially the statistical model that has learned to predict what comes next after certain words and phrases. Chip Huyen Yeah. Lenny Rachitsky And then post-training and fine-tuning specifically is doing that same thing. So pre-training, you get like GPT-5. Fine-tuning is someone taking GPT-5 and doing the same sort of thing, adjusting these weights a little bit for specific use cases on data that they find is necessary to do their very Specific use case. Is that a simple way to think about it? Chip Huyen Yeah, I think you like width as like functions, right? So let's say just like you have maybe the functions of like maybe Lenny's height is maybe like 1x, like 1x plus something, like 2x, so like 1 and plus something is the weight, right? So you change it until you fit the correct data, which is like my height and your height, right? So you can take it as a weight, it's just like a weight, like they function. So you like change adjust the width so they can fit the data, which is the training data. Lenny Rachitsky Awesome, okay. So we were talking about pre-training, post-training, fine-tuning. ([Time 0:07:24](https://share.snipd.com/snip/aa72e10a-e71c-4706-a033-84545dbf8d79)) - **How RLHF Improves Model Behavior** - RLHF trains models by using human comparisons to build a reward model and optimize outputs. - You can replace human labels with AI or verifiable signals, but the mechanism remains rewarding better responses. Transcript: Chip Huyen And it produce an output, right? You want to buy, like you want to reinforce, encourage the model to produce an output that is better, right? So like now it comes to like how do we know that the answer is good or bad, right? So usually people realize on like um signals so one way to get like a first one good or bad is like human feedback right it happened you have two responses you can okay this one's better Than the other um and we do that is because like as humans uh we tend to it's very hard to give like concrete score but it's easier to do comparisons right like if you ask me okay give this Song a score i'm not a musician like and don't know like how hard it is like it's like yeah i don't know like what like how 10 i'm going to do six you know and then if you ask me again a month from Now on i completely forgot and say okay maybe now seven only four i don't know but then if you ask me okay here are two songs and which one would you prefer to play for the birthday party i Was like okay i can't prefer this song so like comparison is a lot easier. So you have human feedback and then you use this human feedback to treat a reward model. So you might tell like which, and then the reward model will help you like, okay, it's a model that produces response. It's a reward model can score. Is this good or bad? And you try to buy us who are like producing better model, the better responses. Another is like you can instead of using a human you can use like ai right like because the response is like yes good good or bad right or another thing is like people are very big on nowadays Like verifiable rewards which is like natural um so basically they give it a math problem and then math solutions uh like it's a model output a solution is you know okay so expected response Should be in a 4a2 and it doesn't provide 4a2 and it's then it's wrong right now it's not a good response um so so yeah so like a lot of time people like using this um human labor like human Um human laborers should like produce like math like um how to say expert questions and i say expected answers and in the ways that like design systems that like verifiable so that the, The models can, can be trained on. Yeah. Lenny Rachitsky Okay. I'm really glad you went there. This is essentially RLHF reinforcement learning with human feedback, which is exactly what I wanted to also talk about. Right. Chip Huyen Yeah. So I think it's like, it's general. It's like, it's a way of learning. It's like training is controversial learning. ([Time 0:16:18](https://share.snipd.com/snip/1194470e-9ac1-4732-b150-c950e2d7cea7)) - **Use Evals Strategically** - Build evals for critical paths and operate by return on investment, not perfection. - Use targeted evals to uncover specific failure modes and guide product improvements at scale. Transcript: Lenny Rachitsky This is the other piece of data content these companies share that AI labs really need. Can you just talk about what an eval is, the simplest way to understand it, and then how this helps models get smarter? Chip Huyen So I think if people approach eval, I think they're like two very different problems. One is an app builder, right? And like, can I say have an app that do like maybe a chatbot. Very simple. I know it's the first thing that came to my mind. And I want you to know if a chatbot is good or bad, right? So I need to come a little way with like if I let the chatbot. Another thing is, I think of this as a task-specific event design. So let's say I'm a model developer and I want to make my model better at current writing, right? And I was like, okay, but how do I even measure career writing? So I even need someone to understand career writing and think about what makes a story good and then design the whole data set and then criteria to evaluate career writing. So I think it's like more like evolved design that is very interesting. How to do it. And then also like train people, like how to do it effectively. So I guess in a case, I think evolved is really, really fun because it's extremely creative. I was looking at like different evolved we were building and was like, wow, like, it's not dry at all. It's just like super, super, super fun. Lenny Rachitsky We had a whole podcast on evals with Hamel and Shreya and that's exactly what they talked about. It's just, it's actually really fun to create evals for companies especially. So let's still dig into that one a little bit more. There's this kind of debate online that I don't know how big of a deal this debate is, but it feels like people spend a lot of time thinking about this. This idea of do we need evals for AI products? Some of the best companies say they don't really do evals. They just go on Vibes. They're just like, is this working well? Can I feel it or not? What's your take on just the importance of building evals and the skill of evals for app ai apps not the model companies you don't have to be like absolutely perfect at things to win you Chip Huyen Just need to be like good enough and being consistent about it okay this is not the philosophy i follow but like i have worked with enough companies to see that play out so when i say that Why come you don't evaluate let's say you are like an executive right and you want to have a new use case. So here's a use case you started out, you built, and it's like it works well, right? The customers are somewhat happy, you don't have the exact metric for it, but like, so the traffic keeps increasing, like people seem happy, people keep buying stuff, right? And now here's an engineer coming like, okay, we need EVA for it. So and it's like saying it was like okay how much effort do we need to put into eva and they were like okay maybe like two engineers as much as much and they could maybe would improve that And was like okay so how much expected gain can i get from it and the engineer would be like oh maybe you could improve it from like 80 to like 82 percent 85 percent right and i was like okay But if you take that two engineers and be going to launch a new feature then it could give me like so much more like improvement right so I think it's like one of them is like EVAL, sometimes People think of EVAL is like okay this is good enough just don't touch it like if you do spend a lot of energy on EVAL it would like only incremental improvement where it spends energy on Like another use case and maybe it's good enough that you can just vibe check it. So I do things that's like maybe that's what the debate is about. I do things that's like a lot of time people just like get things to the place when it's like, okay, good enough. People run. But of course, there's a lot of risk associated with it because if you don't have a clear metric, you have good visibility to how the applications, the models are performing. It might do something very dumb or it can cause you like, I don't know, something like crazy can happen. So, yeah, so I do think evolve is very, very important if you have, if you operate a scale and where failures can have catastrophic consequences, then you do need to be very tyrannical About what you put in front of the users, understand different failure modes, what could go wrong. And also maybe in a space when it's a feature, the product is a competitive advantage. You want to be the best at it. You want to have a very strong understanding of where are and like where you are with the competitors but it's just something that's like more like a low-key okay it's like something It's like okay that's not so core or like it helps with our users then maybe you don't need to be so so obsessed or like theoretical about it is okay that's good enough for now and if it fails Then it fails like okay i know it's like it's so terrifying but like yeah um yeah i think it's only about the question of return on investment. I'm a big fan of evals. I love reading evals. And I say it's like I understand why some people would choose to not focus on evals right away and choose bringing on new functionalities instead. ([Time 0:22:29](https://share.snipd.com/snip/727ab87b-0ced-4ee6-bb4b-b5af1ea10ed2)) - **Win RAG With Data Preparation** - For RAG, invest heavily in data preparation before debating vector DBs. - Chunk size, summaries, metadata and rewriting docs for QA dramatically improve retrieval quality. Transcript: Chip Huyen So RAG stands for Retrieval Augmented Generations. It's also not a specific to JNF AI. So the idea is just like for a lot of questions, we need context to answer. So I think it came pretty, I think it's from the paper of 2017. So someone was like, so they realized it's like for a bunch of like benchmark, when the question answering benchmark, they realized it's like, okay, if we give the model information About the questions, then the answer can be much, much better. So what they do with us is they try to retrieve information from Wikipedia. So for question about topics, it's like retrieve that and then put it into context and answer it just much better. So I feel like it sounds like a no-brainer, right? I mean, obviously. So I think that's what Racket, as a simplest sense, is just like providing the model with a relevant context so that it can answer the questions. And that's where things get really more interesting because traditionally when it started out, Rack is mostly text. So we talk about a lot of ways of how to prepare data so that the model can retrieve effectively. Let's say that not everything is a Wikipedia page, right? Like Wikipedia page is pretty content and you know, okay, everything about it is about a topic. But a lot of times I have documents extremely long, right? And like they have a weird way of like structures and documents. Let's say that you have documents about Lenny podcast, right? And in the future, in the beginning of documents, like from now on, podcast wouldn't refer to Lenny's podcast, right? So let's say somebody in the future is like, okay, tell me about Lenny, right? Lenny's work. And because the rest of the document does not have the term Lenny, you just don't know, you might not retrieve it. And the document is long enough that it's chunked into a different part. So like the second part doesn't have the word Lenny, so you cannot retrieve it. So you have to find a way to process data so that makes sure it's like, it can retrieve the information that's relevant to the query, even though it might not immediately like obvious That it's related. So people come up with like only thing if I think like contextual retrieval, like giving a chunk of the data that's relevant, like maybe in a summary, metadata, so that it knows. All the same people use it as a hypothetical question. It's very interesting. For a heap of a chunk of documents, I must get a bunch of questions that the chunks can help answer. So that when they have a query, it's like, okay, does it match any of the hypothetical questions? So it can fetch it. So it's a very interesting approach. Okay, so maybe before I go to the next thing, I just wanna say this, data preparations for rack is extremely important and i would say just like in the a lot of the companies that i have seen So it's like the biggest performance in their rack solutions coming from like better data preparations not agonizing over like what very database to use uh because very database of Course it's very important to care about things like latency or if you have very specific access patterns, like read heavy or write heavy. Of course, it's like it matters. But in terms of pure quality answers, I think the data preparation is just hands out. ([Time 0:32:04](https://share.snipd.com/snip/18c7a923-7dc9-4ff2-8a42-7f1e8776a8bf)) - **Make Docs AI-Readable** - Rewrite documentation and add AI-focused metadata so models can retrieve facts accurately. - Create summaries, hypothetical Q&A, and explicit scale/units to remove ambiguity for AI readers. Transcript: Chip Huyen One way is just mentioned as in like you have like chunks of data so we can think about like how big of each chunk should be right because if it's like sort of thing about like if the context You want to maximize maybe you can it's a very simple example right now you want to retrieve like a thousand words right so if each chunk of data is too long then so if a beta chunk is long Then it's more likely to contain more relevant metadata so you can retrieve more but if it's too long like then you have a thousand words and so chunk is like a thousand words you get a rich One chunk so it's not very useful but if it's too short then you can retrieve more relevant information like also it can retrieve a wider range of, like, documents and chunks. But at the same time, a chunk is too small. It contains relevant information. So you have, like, very nice, like, chunk design, like how big a chunk should be. You add, like, contextual information, like summary, metadata, hypothetical questions. Somebody was telling me, like, a very big performance they got is that from um rewriting their data in the question answering format so like instead of having like so they have a podcast Right instead of just chunking the podcast you just like reframe rewrite it into like here's a question here's answers um like and produce a lot of them it can use ai for that as well so That's one example of data processing a lot of example we i see is like for people helping like using ai should have like specific uh tool news and documentations right and a lot and we Write documentation usually to document documentation today is written for human reading and ai reading is different because it's different because humans we have like common sense And we kind of know what it is so so one things that are like uh human for human experts they have the context that ai doesn't quite have so somebody told me is that like what's a big change They have is like let's say that um you have a you have a function a document uh documentation for this maybe this library as a library says okay the output of this one is like maybe talking For like i know some crazy term, maybe there's some temperature or something on the graph. It should be like one, zero or minus one. And as a human expert, maybe understand the scale, like what one in the scale mean. But like for AI, it just really doesn't understand what that means. So actually have like another annotation layer for AI. It's like, okay, what temperatures equal one means like that. It's not like it's an absolute temperature. It's like associated with the scale over there. So it's just saving that all this data processing to make it easier for AI to retrieve the relevant information to answer the questions. ([Time 0:35:13](https://share.snipd.com/snip/a64cd5ff-7d86-431e-bb17-bddffb05428d)) - **Randomized Trial On Coding Tools** - One friend's randomized trial split engineers into high/medium/low performers and gave half in each group Cursor access. - The highest-performing engineers showed the largest productivity boost from the AI coding tool. Transcript: Chip Huyen So, I have different reports. One team would tell me that like um one of the people tell me okay amongst all his engineers he thinks it's like senior engineers would get the most output like would be more productive Because it's like okay so that person very interesting so so he actually divided his team to like three buckets but he didn't tell them, obviously. He was like, okay, here's more like currently like best performing, average performing and lowest performing. And then there's a randomized trial. So they give like half of each group access to like cursor. And then he was noticed like over time, he was like, okay, something funny, like the group that get the biggest performance boost, like in his opinion, so it was very close in his team. There's a biggest performance boost, like the senior engineer, so it's the highest performing. So the highest performing engineer get the biggest boost out of it. And then the second group is just like the average performing. So his opinion is like, okay, the highest performing engineers, they're so normal, proactive. They would say, no, I just solve problems. So AI helps them solve problems better. Whereas the people who have the lowest performing, they only don't care much about work, right? So like it's easier to just like go on autopilot, get it to like generate like bad code and just like do it. Or just don't know how to do it. Another company, however, they tell me that's like actually senior engineers are the one most resistant to like using ai as this tooling because they said it's like okay but ai because They are more opinionated and they have very high standard it was like okay but ai code jitter code just sucks so just like very very resistant in using this so i don't know i haven't quite Be able to reconcile very different reports on that yet. Lenny Rachitsky This is so interesting. So just to make sure I'm hearing the story. So there's a company I work with that did a three-bucket test with their engineering team where they created three sorts of groups. The highest-performing engineers, mid-performing engineers, lowest performing engineers, and gave some of them, so they gave some of them access to say cursor. Was it cursor or what did they give them access to? It was cursor. Chip Huyen I think by the way it was cursor. Okay, cool. I didn't work with them. It's more like a friend company. Lenny Rachitsky Okay, it's a friend's company. So did they give like half of the higher performing engineers cursor and half not or how did they do the split? Chip Huyen Yeah, so like they give like half of the entire company, but like half for each bucket. Yeah. And then they observe the difference in like productivity. I see. Lenny Rachitsky Yeah. So how did they even do that? They're just like, okay, you get cursor, you don't get cursors. How did they do that? That's so interesting. Chip Huyen Yeah, I didn't get into the mechanics of it, but I was like, I respect you for doing a randomized trial. That is so cool. Yeah. Okay. Lenny Rachitsky How large was this engineering team was it like hundreds of people it's not that large it's about like maybe um 30 to 40 yeah 30 to 40 okay yeah wow okay so they found that the highest performing Engineers had the most benefit from using ai tools and then behind them was the middle tier engineers and the worst performers were the lowest performers. Yeah. Chip Huyen But it's still not the same everywhere. ([Time 0:45:55](https://share.snipd.com/snip/8d5b7c62-9310-46f1-b70b-5a42aa81dd5a)) - **System Thinking Gains From AI** - Senior engineers who think systemically gain the most leverage from AI tools. - System thinking and architecture will matter more than rote coding as AI automates implementation details. Transcript: Chip Huyen Yeah, I definitely really appreciate, as you see companies, we appreciate engineers who have a good understanding of the whole systems and be able to have good problem-solving skill, Thinking holistically instead of locally. When our company have seen the way they work as they told me is they work completely different now and like so they actually restructured engineering org so that like they get more senior Engineers should be more in the peer review because they like to get like uh sort of writing guidelines on like what is a good engineering practices um what is the process would be like Maybe like okay so they write like a lot of like on how to work well. And then they have more junior engineers just produce code and submit PR. But senior engineers more in the reviewing case. So I think it might be prepared for the future. So another company actually told me something very similar. So they kind of prepare for the future once you only need a very small group of like very, very strong engineers to like create like processes and like reviewing code to get into production, But like get like AI or like junior engineers to like produce code. But then the question becomes like, how does one become a very strong senior engineer? Lenny Rachitsky Right, that's right, that's right. That's the problem. Chip Huyen Yeah, so I don't know what's the process. Who was thinking about like, yeah. Lenny Rachitsky No one's thinking about it. It's just, it's a problem. We won't have any more in 10, 20 years. There'll be no more engineers because no one's hiring junior engineers. Although I could make the case, junior engineers, people just getting into computer science right now are just native, AI native. And in theory, you could argue they will become really good really fast if they're curious, aren't just, you know, delegating learning and thinking to AI, but learning how to actually, Using it to learn how to code well and architect correctly. Like you could argue they will be the most successful engineers in the future. Chip Huyen I do think that what I mentioned is loading to architect. I think I grouped that in like system thinking. I do think it's a very important skill because I think AI can help automate a lot of like distributed skills. ([Time 0:49:39](https://share.snipd.com/snip/f18aa591-ef84-4879-8f66-24d1a6ce32e1)) - **Teams Will Become More Integrated** - Organizational boundaries will blur as product, engineering, and eval work must integrate closely. - Cross-functional teams and shared ownership of metrics will become critical for AI products. Transcript: Lenny Rachitsky I'm curious just if you had to think about in the next two or three years, just where things are heading. What do you think? How do you think building products will be different? How do you think companies working will be different if you had to think of maybe the biggest change we expect to see in the next few years in terms of how companies work? Chip Huyen I think in a lot of organizations, they don't move that fast, right? But at the same time, they have to move faster than I expected. Because, again, I think it's like bias. Like I don't work with dinosaur companies. I don't care. I think a lot of executives who come to me are like very forward-looking so maybe for me I'm very biased towards organizations just like move fast. So yeah so I think one big change I see is like in organizational structure. I think there's a lot of value placed in like, so before, right, we have like a lot of disjointed team. Like we have very clear like engineering team, product team. But then there's a question of like, who should write Eva, right? Like who should own the metrics? And it turns out it's like Eva is not a separate problem. It's a system problem, right? Because you need to look into different components, how they interest each other. You need to use the behaviors because you need to know what users care about so that you can, so that you can, like, write Eval, like, reflect what users care about. So, so all of that, like, you can sort it from, like, you look into different component architectures, place guardrails and stuff. So it's just engineering, but understanding users is, like, what product, right? So, so because of, like, a lot of things, Eval extremely important. So like the kind of bring product team and like engineering team, even like marketing team, like user acquisition, like very close to each other. So, so yeah, since in a way, so if you go, structuring, so there's more communications between like previously very distinct functions. ([Time 0:57:20](https://share.snipd.com/snip/52f1a741-5bde-470d-9c7b-d844f6effe5f)) - **Boost Quality With Test-Time Compute** - Spending compute at inference (test-time compute) — e.g., sampling multiple answers or more internal 'thinking' — can boost perceived model performance. - This improves outputs without changing the base pre-trained model. Transcript: Chip Huyen Versus the perceived performance. So let's say it's like, Amashin thought about, are you familiar with the term test-time compute? Lenny Rachitsky I don't think so. Chip Huyen Help us understand. So the idea is it's like, okay, like you have a fixed amount of compute, right? So you're going to spend a lot of compute on pre-training or training the model. Pre-training, and then a lot of some compute on like five-training. And the ratio of pre-training to post-training compute is like crazy very different different lab um and also like since then it has to spend compute uh on like generating inference When I have a trained and fighting a model now you want to like survey to users so I might type of questions or prompt and it's like jared like do inference like and that requires a compute And i guess i feel about discussion of like uh should i spend more compute on like pre-tuning or fire tuning or inference right because like inference and people found out as like test Time compute so like spending more compute on inference is like called like test time like uh compute uh like as a strategy of like just allocating more resources, compute resource To generate inference. When I should bring better performance, how does that do it? Like, let's say, let's say you have a math question, right? And maybe instead of just generating one answer, I can generate like four different answers and say, okay, whichever is the best according to some standard. Or like, okay, I have four answers and then maybe like three of them say 482 and one of them said like 20. Okay three of them in the in agreement so the answer should be 482 right so like just people shouldn't generate a bunch of it or another thing is like a lot of time like reasoning uh thinking It's just like people should like generate more thinking tokens and spend more time thinking before showing the final answers uh it's like require more compute but it's like give it More and more better performance. So, yeah, so I think it's like from the user perspective, right? Like when the model spends more time exploring different potential answers, thinking longer, it can give you much better final answers. ([Time 1:06:10](https://share.snipd.com/snip/c36c2ed4-f6c0-4f8a-9a2e-bc0821d30075)) - **Find Ideas By Noticing Frustrations** - To find AI product ideas, watch your daily frustrations for a week and build micro-tools to remove them. - Start with small niche automations that improve your own workflow and iterate from there. Transcript: Chip Huyen And usually what we think of is like, okay, like one tip is like, go look from the last week, right? Like for a week, just like pay attention to what you do and what frustrates you. And when something frustrates you, I think like, is there anything we can do? Is there like, can it be done a different way? So it's not frustrating. And I can talk like people can swap, accept subnobes or teams. And if you see they come on frustrations, maybe just something you can think about just to build something around that. So yeah, so I feel like just like notice like how we work, thinking of like ways to like constantly ask questions, like how can it be better? And then I just build something to like address the frustrations. I think it's a good way to just like learn and adopt AI. ([Time 1:10:06](https://share.snipd.com/snip/63db9ef7-bca0-48c9-9c64-d71073b3b4e7))