A Developer's Toolkit for SOTA AI  ## Metadata - Author: [[Changelog Master Feed]] - Full Title: A Developer's Toolkit for SOTA AI - Category: #podcasts - URL: https://share.snipd.com/episode/f1c9a695-5ddf-47f1-af2e-63e4b75a27c5 ## Document Notes ## Highlights - **Nuro Experience Shapes Codeium** - Varun and Anshul both worked at autonomous vehicle company Nuro, where they faced large scale ML workload challenges. - Their shared background in GPU and ML infrastructure led them to found Exa Function and later build Codeium focusing on generative AI. Transcript: Speaker 1 Varun, if you want to kick off, and then Anjul afterwards. Speaker 2 So, maybe I can get started. Actually, it starts in 2017. I started working at this company called Nuro that does autonomous goods delivery. So, it's an AV company. There I sort of worked on large scale, offline, deep learning workloads. So, as you can imagine, an autonomous vehicle company needs to run large scale simulation. They need to basically be able to test their ML models at scale before they can actually deploy them on a car. And sort of in 2021, I left Nuro and started X-A-Function, which is the company that is building out this product, Codeium. And X-A-Function started out building GPU virtualization software. So, you can imagine, for these large scale deep learning applications, one big problem is GPUs are scarce, they're expensive, and also hard to program. And sort of what X-A-Function started building was solutions and software to make it so that applications ([Time 0:02:09](https://share.snipd.com/snip/6f3f8fab-db73-4bbc-bec2-2d06cd45bc13)) - **GPU Virtualization Challenges** - GPUs are scarce and expensive, and unlike CPUs, cannot be easily virtualized to run many workloads. - Efficient GPU usage is crucial since modern high-end GPUs cost tens of thousands of dollars each. Transcript: Speaker 2 Maybe I can take it a step back of why these GPU workloads are just a little bit annoying compared to CPU workloads. One of the really sort of unique things about GPUs is that unlike CPUs, it's kind of tricky to virtualize. Like one common thing that we have with CPUs is you can put a bunch of containers on a single VM and then you can kind of make use of the CPU compute effectively, right? You can basically dump 10 applications onto a CPU and it's perfectly fine. For GPU, it's a little bit more messy because the GPU doesn't have a ton of memory. So you can't just load up infinitely many models on there. Like let's imagine you have a GPU with 16 gigs of memory and each of these models takes like 10 gigs. You can't really even put two applications on there. So then that already becomes a big issue. And that's sort of what a lot of these large deep learning workloads were struggling with. So when I was at NURO, one big problem we had was we had around like tens of models, but we had these workloads that needed hundreds of GPUs, some of them even thousands of GPUs. And we struggled to basically make it so that we were even able to use the hardware properly. And then you can imagine the complexity then stacks with now we're in a state where companies have trouble even getting access to 10 GPUs because of NVIDIA sort of scarcity issues. ([Time 0:05:26](https://share.snipd.com/snip/2043e4f4-c1cd-4f4a-a96d-f944724b1c07)) - **Explosive Growth of AI GPU Workloads** - AI workloads on GPUs have grown exponentially in size and scale since 2018. - Managing over 10,000 GPUs in a single cloud region shows the rapid growth and importance of deep learning infrastructure. Transcript: Speaker 2 So when we started, like you can just imagine everything was a lot more smaller scale, right? The hyperscalers of the cloud providers just didn't have nearly as much GPUs. Like if you ask them like what fraction of cloud spend is GPU spend is probably like very small single digit percentage points, maybe even less than that at the time. So this is like a very small workload for them when we sort of started, both me and also started in NURO in like 2018. But then over time, this grew a ton, like we could see it from the training workloads. These were no longer like even single node training workloads. Like back in the day, a single GPU node that had maybe like eight few hundreds or something was like considered a lot of compute. And suddenly now we were able to witness the fact that this was slowly becoming eight eight one hundred nodes. And then more than eight of these nodes were necessary than even to train these models. And similarly, to prove out that these models were capable, like in an actual production setting, you needed to run offline testing at massive scales, like on the order of like 5000 To 10,000 T fours scales, which is like kind of incredible in terms of raw flops. So we were able to see this hockey stick happen in front of us. And then that's sort of what made us want to start exa function in the first place. We realized that there were going to be large deep learning workloads. One interesting fact is for us, like for just the exa function GPU virtualization software that we ended up selling to enterprises, we ended up managing over 10,000 GPUs on GCP in a single GCP region. So we ended up managing more than 20%. And we realized that, that, Hey, this was only going to keep growing. ([Time 0:08:10](https://share.snipd.com/snip/f942906a-83b8-4405-a5ab-4ca17b906c26)) - **Codeium Is a Developer AI Toolkit** - Codeium extends beyond autocompletion, offering an AI toolkit including IDE chat and natural language search. - This aims to address the entire software development lifecycle, unlike Copilot's focus on autocompletion. Transcript: Speaker 1 Can you talk a little bit about what co-pilot's not doing for generative AI or how you're approaching it that allows you to show people this as a better way forward from your perspective. I mean, we have tons of respect for the co-pilot. I'm just going to start with it. As we said, we were all early users of it. Definitely not putting you into conflict with them. That is a starting point for people. Absolutely. Yeah. I think the way we kind of view this and I kind of like alluded to this earlier is that you're writing brand new code, right, with autocomplete is really just one small task that we do as Engineers, right? We refactor code, we ask for help, we write documentation, we do PR reviews. And so kind of our general approach has always been, let's try to build an AI toolkit rather than an AI autocomplete tool. Got it. So we can get more into this into the weeds here, but autocomplete is just one of our functionalities that we provide. We provide an IDE chat. So I think chat GPT, except integrated with the IDE, natural language search over your code base using embeddings and vector stores in the background. So we're really trying to expand, how can we address the entire software development lifecycle? ([Time 0:14:02](https://share.snipd.com/snip/ffa159c0-3030-4600-92e6-f3d52b06154f)) - **Free Plan Drives Infrastructure Efficiency** - Codeium offers an unlimited free individual plan, forcing efficiency in infrastructure and providing large scale usage data. - This scale allows optimized deployment to enterprises, resulting in a superior product experience. Transcript: Speaker 2 Like if we have better infrastructure, that's great. But if that makes a product that's the same, no one should care. They just assume that you should. Yeah. So what we started with is we set a very high bar for ourselves. Codeium is an entirely free product. So like for the individual user, it's something that they can install and use immediately for free that are unlimited and there's like no limits at all. So like when it comes to autocomplete, you can use it as much as you want. And this is by the way, forced us to do things where infrastructure is as efficient as possible. Just to give you a sense of the numbers we're talking about here, we process over 10 billion tokens of code a day. That might sound like a large word. It's like over a billion lines of code a day that we process for our own developers. We're forced to do this entirely for free. And then on top of that, we probably have one of the world's largest chat applications also because it's in ID as well. And all of this put together has allowed us to build a very, very scalable piece of infrastructure such that we're the largest users of our own product. We're the largest user of our own product. We learn the most from our users and we can then take those learnings and deploy in a very cost effective, very efficient and optimized way to our own enterprise users. So one of those things where we force ourselves to learn a lot from an individual plan and then take all those learnings and actually bring them over to the enterprise. And a lot of the learnings we were only able to make because we place like very, I would say like annoying infrastructure constraints on ourselves by saying, hey, you guys got to do this Entirely for free basically. And we're committed to building, codeium is going to be a free product forever. Actually, the individual plan will always be free. And it's one of those things where our users are just always like, how are these guys even doing it? Like what are they even doing to make this happen? And most of our users by the way are users that have turned off of Go pilot. We have spent very little, if not anything on marketing. So it's just one of those things where our users are like, how do we make this free? We take the approach of we think some of the best products in the world are free, like products at Google, right? They're entirely free. Google doesn't tell you all the time that they have the best infrastructure, but they do have the best infrastructure. It just so happens to be the case that that shows itself off in the best product. And we could talk a little bit more about how we take our sort of focus on infrastructure and make a much better enterprise product as well. But like, that's the way we sort of look at it. It's like, how do we deliver a materially better experiences with our infrastructure? Speaker 1 And our users shouldn't care that we actually did that. You've brought it up. You got ([Time 0:18:20](https://share.snipd.com/snip/cc061cb5-3fab-4239-a7ab-f7903b7a97d2)) - **Self-Hosting Enables Fine-Tuning** - Self-hosting enables fine-tuning LLMs on a company's proprietary code, vastly improving relevance and reducing hallucinations. - Codeium achieves local fine-tuning and inference on the same hardware without extra provisioning, enhancing privacy and efficiency. Transcript: Speaker 1 I mean, yeah. So I think, you know, that as Varys said, there's a lot of things that we do from like the individual infrastructure point of view so that we can do crazy things like make it all free for All of our individual users. But once we actually self-host, there's actually a lot of things that you can do that, you know, just any other tool can't do without being self-hosted. One of the ones that Vary just mentioned is the personalization, right? If you're fully hosted in a company's tenant, you can use all of their knowledge bases to create a substantially better product, right? I think the way we generally think about is that you have a generic model that's good. It's learned from trillions of tokens of code on the public corpus. But if you think about any like individual company, they have themselves hundreds of millions of tokens of code that has never seen the light of day. And that's actually the code that's the most relevant for them if they want to write any new code. Think of like all the internal syntax, semantics, utility functions, libraries, DSLs, whatever it might be, in a model like a copilot or a codeum by the nature of having to be low agency Can only take about 150 or so lines of code as context, right? So this is not like one of those like, you know, chat GPTs or GPT4s where you're like putting in files and files of context, like it's really small where you can put in. And so there's really no way for a single inference to have full context of your code base without actually fine tuning the base model that we shipped to them on all of their local code. And so we've actually done a bunch of studies and we're like on a how this actually massively reduces like hallucinations and all these other things that you always hear coming up with LLMs. But you know, things like this, things like providing more in-depth analytics, all these things actually come up by being self-hosted. And as we mentioned, these are all at the core to some degree an infra problem, right? How do you actually do fine tuning locally in, you know, companies tenant? That's actually an infra problem that, you know, we're happy to talk more about maybe I'll just I'll pass it back to you, Chris. I actually am about to ask a follow-up about that because you've got me really thinking about some of the use cases in my own life on that. And so with the self-hosting model and you're able to now kind of like, you know, open AI I said, you know, with JAT GBD4, there's only so far we're going to go because we've kind of we've Used the public corpus of knowledge out there on the internet, you know, so there's only so much more vertical scaling you can do on the model learning. And so, you know, you're touching on the fact that there's so much hidden IP in code, hidden information in code that is of huge value, particularly to the company that it's in because It's representing their business model and the way their business has evolved over time. And so if I'm understanding you correctly, you're basically saying that your solution can take advantage of that on their behalf and really, really honed against it. What are some of the limits on privacy? Are they able to do that? Because that's a big topic. We've actually talked about it on the show before about, you know, in this generative AI age with IP concerns and privacy concerns and, you know, getting the lawyers involved. Are you able to do the training on their site and keep it to the customer entirely or do they have to let their IP out and stuff? How do you approach that problem? Yeah. I mean, so one of the answer to any question of like, does any IP leave coding for enterprises? The answer is always no. So in pretty much every like part of the system, like, you know, our guarantee is to actually be able to deploy this whole thing fully air gapped. We've even deployed in places like, you know, AWS, Cov cloud, which is like entirely, you know, doesn't even have connection with the internet kind of scenario. So nothing ever leaves there to address some of the points you brought up there. It's like, yeah, I mean, we're not like the only ones who are like saying like, Oh, no, the data that a company has privately is like super important. And it's potentially even more important than the size of the model. I think, you know, a good example, this is actually meta. Instead of using like a GitHub co-pilot or any generic system, they decided in, you know, I guess, classic meta fashion to like train their own auto complete model internally using All of their code. And they actually, you know, published a paper, I think a few weeks back. And their model was like, in terms of size, I think 1.3 billion parameters like slaw in respect to the LLM world. And it just massively outperformed GitHub co-pilot on pretty much every task. There's some like corroborating evidence to, you know, what we're saying about fine tuning that doing this actually does lead to materially better performances for, you know, the User in question. Is that meta model going to be good for everyone else to code? Probably not, but that's also not the whole point. Right. And in terms of like being able to fine tune locally, yeah, we're able to, you know, do this completely locally. And again, it comes down to like, you know, scale of data. Our base model has been trained on trillions of tokens of code. Right. That's a lot. That's why we need this like, you know, multi node GPU setup to do all this training. But an actual company, you know, if they have like, say like even 10 million lines of code, it's about a hundred million or so tokens, there's like a huge order of magnitude difference Still between this pre training and the fine tuning, which is why we can do this kind of locally on actually surprisingly, whichever hardware they choose to provision for serving their Developers. So again, this comes to some of our like, and they'll in front of the actor added all the stuff that we know how to do. We actually can do fine tuning and inferences on that same piece of hardware. So we don't actually ask, you know, companies to provision more hardware and even more like critically, we are able to do fine tuning during any idle time of that GPU. So whenever that GPU is not being used to perform an inference, it's actually doing like, you know, backdrop steps to like continuously improve the model, you know, fine tuning is just One aspect of like a larger kind of personalization system. ([Time 0:21:58](https://share.snipd.com/snip/0865cf5f-9f7f-43c4-8ac6-af4e3742c9ac)) - **Streamline Workflow in IDE** - Use IDE-integrated chat features for streamlined code exploration instead of copying code to external tools. - Apply generated code changes directly with 'apply diff' to avoid merge conflicts and improve workflow. Transcript: Speaker 1 Of trying to figure it out. We're still in very early days of these AI applications. And this is our learnings of trying to become a product company. We're actually taking like the U.S. Quite seriously, right? And this is actually what the individual plan is, because it's great to get feedback on. I think very, you know, concretely, I think a lot of people have that frustration of like having to copy a code block or reach out GPT right out of full prompt and like, you know, remember The exact prompt that they typed in before that gave them a good result and then copying the answers back in and then making modifications like that workflows clearly kind of broken. So when we actually built our chat functionality into the IDE, we're like, okay, what are all the parts here that can get totally streamlined? Right? And so we actually did things like, you know, on top of every function block, there's a little like code lenses that are just these small buttons that someone can like click like explain This function. It'll automatically pull in all that relevant context, open it up in the window, you're not copying anything over. And it's like writing, you know, get out in a human text. Or if you say like refactor a function or had doc strings, right, or write a unit test. Do you know, just like small little buttons or, you know, preset prompts that you can just then click, do this generation on the side. And then we even have a way of clicking like apply diff. And because we know where we pull the context in, we can apply diff right back into the context, right? And so you're not copying things back and trying to like resolve merge conflicts, like all of these things are done kind of automatically. So there's a lot of really cool things you can actually do when you start bringing these things into the IDE where developers are. ([Time 0:29:40](https://share.snipd.com/snip/d3b8c6ed-e60f-4abf-bc17-3fd21582543f)) - **Autocomplete Dominates AI Usage** - Autocomplete accounts for 90-95% of AI tool usage by developers due to continuous passive assistance. - Exploration mode, where developers write comments to guide code generation, is key for discovering new APIs and functions. Transcript: Speaker 2 So interestingly, just a little bit about multi modality, I think we're maybe a little bit far from leveraging, I guess, other modes beyond text for code. I think maybe that that'll happen. But I think there's not enough evidence right now yet for auto complete just to be open about the sort of the functionality we have, we have auto complete, we have search and we have code Base aware chat. So for we recognize right now that of the usage, auto complete accounts for more than 90 to 95% of the usage of the product. It's because chatting is not something people do like even every day potentially. They might open it up once every couple days, but auto complete is something that's like always on very passively helpful and people get the most value out of it, which is kind of counterintuitive. I think people don't recognize that immediately. But when people are doing auto complete, we've recognized there's like two modalities right of the way people type code. There's a modality of accelerating the developer, which is like, Hey, I kind of know what I'm going to type and I just want to tab complete the result. And then there is also an exploration phase, which is like, I don't even know what I'm trying to do. But based on that, I write a comment. This is like a classic thing where like my behavior writing code has materially changed because of tools like code, where I'll write a comment and I kind of just hope and pray that it pulls In the right context so that it gives me the best generation possible. So in my mind, for the acceleration case, code is like very helpful, right? It can like auto complete a bunch of code. But to make the exploration case, that's where the true magical moment comes in where I had like no clue at like how I was going to use a bunch of these APIs. And that's sort of what we're focused on trying to make really better, whether that be in chat as well as with auto complete. How do we make it so that we can build the most knowledgeable AI that is maximally helpful and also minimally just like annoying? The interesting thing about code, you know, as a product or these auto complete products is they get a little bit of getting used to, but even despite the fact that they write wrong things, It's not very annoying because you can very easily just say, I don't want this completion or it didn't like write an entire file out and you need to go and correct a bunch of functions. It was like a couple of lines or maybe like 10 lines of code. You can very easily validate that it's correct, right? That comes back to then what Anshul was saying, which is how do we make sure we can provide always the maximally helpful sort of AI agent. The answer is have the best context possible. And a couple of nitty gritty details we do is currently our context and we'll write a blog post about this is double what copilot says. We allow double the amount of context for auto complete than what they do. The second thing is we're able to pull context throughout the code base. And this is actually that same piece of technology that is pulling context throughout the code base through search and all these other functionalities is getting used as part of chat For code base aware chat, which is something that copilot doesn't even have today yet. The third piece is finally for a large enterprises, how do we make it so that these models actually semantically understand your code, which is where fine turning comes in. It's like for us, context gets us a lot of the way, but it doesn't get us all the way because you can just imagine even with double the context, let's say we can pass in 1000 lines of code for A company with 10 million lines of code, we're scratching for orders of magnitude less code than the company actually has. So this is where our vision is like we want to continually ramp up the amount of knowledge these models have and the ways in which they can be helpful. ([Time 0:31:58](https://share.snipd.com/snip/dc88ca45-2807-4161-a2d4-9c49b95249a6)) - **AI Assists, Doesn't Replace Engineers** - Varun expects AI to assist developers by automating boring, repetitive tasks, not replacing human control anytime soon. - They focus on practical, useful AI features rather than chasing hype about fully autonomous coding agents. Transcript: Speaker 1 Stuff like that. And you're having to stay out front. Where's your brain taking you at night? You know, when you when you stop and you chill out and have a glass of wine or whatever you do and you're kind of just pondering, what does the future look like? And I'd like to know both from your own specific personal standpoints in terms of your product and that, but just the generative AI world in general, how do you see it going forward? I'd love your insights. Yeah, I think the classic question and then the grand scheme of things like, oh my God, is like gender day. I just can like totally get rid of my job or completely like invalid. And I think for us, we will be the first people to say that we know we do think like AI would just be like the next step in a series of at least in code or a series of tools that have had made like Developers more productive, right? That have led them to be able to focus on more kind of interesting parts of software development. And you know, we're an assistant, right? All these tools are called AI assistant tools, I think for a reason. You know, we're definitely not at a place yet. I don't think for a while where there is it going to be like a human in the loop, like in control, you know, guiding the AI and what to do. So from that kind of respect, like the doomsday scenario, I'm, I don't want to speak for it, but I think we're like pretty far from that mentality. But we do think like, I think, you know, we wouldn't have gotten into code. And if we didn't genuinely think that there was just so many things that we do as a day to day as engineers that are just a little frustrating, boring, kind of take us out of the flow state, You know, slow us down. Those all seem like very prime ripe things to like try to address with AI, right? And I think that's kind of our general goal, right? I think there's a lot more capabilities to build, right? I don't think search chat, these are going to be the last, I guess, like building blocks that we build where we have more capabilities coming up that we're super excited about. But yeah, it's also like, you know, going to be a thing where, as you said, this is moving super quickly, right? Like we have like research, open source, like applications all developing at the same time at pre-expede. And so I think part of what we're also looking forward to is like how can we also just like educate like, you know, at least software developers on the best way to use AI tools, how to like Best make the most use of it so that they are part of the way, right? And that they also can get a lot of value. Well said. Varun. Speaker 2 Yeah, maybe if I was to just say like you were asking me what the big worry is, for me, the big worry is there's going to be a lot of like exciting new demos that people end up building. And obviously for us as a company, we need to make strategic bets on like, hey, this is a worthwhile thing for us to invest in. For instance, I think a couple months ago, there was an entire craze on agents being able to write like entire pieces of code for you and all these other things. For us though, we had lots of enterprise companies that were sort of using the product at the time and recognize that the technology just wasn't there yet, right? Like take a code base that's like 100 million lads of code or 10 million lads of code. It's going to be hard for you to write C++ that's like five files that compiles perfectly. And then also like uses all the other libraries when you have context that's like, you know, five files. It's not going to be the easiest problem. And I think that's maybe an example. But for us, we've currently, I would say, just a pat on the back over the last eight months, iterated like significantly faster than every other company in this space, just in terms of The functionality. But we need to make strategic bets on what the next thing to sort of work on is at any given point. And we need to be very careful about like, hey, this is like a very exciting area. But is it like actually useful to our users, right? Like is it actually useful in that, hey, like maybe we could do something where a great example is given a PR, we generate a summary. And I think co-pilot has tried building something like this. And we tried using the product that co-pilot had and it was just wrong a lot of the times. And I think that would have been an interesting idea for us to pursue and keep trying to make work. But then there is a diminishing returns. And I think Anshul and I have seen this very clearly in autonomous vehicles where we had a piece of technology that was kind of just not there yet. Like it needs a couple more breakthroughs of machine learning to kind of get there. And the idea of building it five years in advance, right, you shouldn't be doing that. You just 100% shouldn't be building a tool when the technology just says there yet. And that is something that keeps me up at night is like, what are the next things we need to build while keeping ([Time 0:36:30](https://share.snipd.com/snip/16fbaa12-3ee8-4c94-abf9-0ee12fd884ff))