Automating Code Optimization With LLMs  ## Metadata - Author: [[Changelog Master Feed]] - Full Title: Automating Code Optimization With LLMs - Category: #podcasts - URL: https://share.snipd.com/episode/afec0f71-fd18-4bd3-b150-ab912c0be1c1 ## Document Notes ## Highlights - **Code Optimization Importance** - Code optimization improves speed, memory, and CPU usage, impacting cloud costs and energy consumption. - It's crucial for scalability, but often overlooked by developers using high-level languages. Transcript: Speaker 1 Time? We are in an area nowadays that we see more and more applications consuming a lot of cloud resources and a lot of resources. And everybody is trying to optimize the performance of their code. Now, when we're talking about code optimization and in particular performance, typically people would like to optimize things like application being faster, maybe memory consumption. We know everybody complaining about Chrome using too much memory, for example, the past or CPU usage, which is connected very much with the energy that the different software is using. If we talk about mobile phones, application there, we would all like to be more efficient and consume less energy. Practically, that's the area that we have been focusing in my research and our group in our research and in the company. Speaker 2 Maybe you could talk a little bit about some of the history of that research. I'm sure that that has just like everything else been impacted by this kind of latest wave of AI technologies and generative AI. But I know that the company and yourself have been involved in research prior to and all during the development of these things. So could you give us a little bit of background on how you first started thinking about these problems and how it kind of developed over time? Speaker 1 Yeah, code optimization is not a new thing. If you read the research papers 20, 30 years ago, everybody would like to optimize and make the code efficient. Like there are a lot of tools like profilers that help developers find hotspots. The biggest problem in this area is, okay, we profile our code, but how we can automatically improve it so we make it faster. Typically, this is a very, very manual process and majority of the people that work on these areas are super specialized. And it's more and more difficult nowadays to find people that know how to optimize the performance of their code because the programming languages are becoming higher and higher level Like now people write more in language like Python, JavaScript, TypeScript. So nowadays, the real companies that deal with code optimization are companies in the lowers from the hardware space like big Intel, NVIDIA, and those, they write specialized software That take advantage of their hardware, where they show the hardware it outperforms or technology companies that need the scalability. But majority of developers will not necessarily bother about the performance of the code as an immediate first thing that they need to optimize. ([Time 0:02:22](https://share.snipd.com/snip/0deebf53-6d37-4546-b631-236bb81ad491)) - **Code Optimization History** - Traditional code optimization is manual, relying on specialized profilers and compiler options. - AI-driven optimization automates this, making it accessible to more developers. Transcript: Speaker 1 It kind of developed over time? Yeah, code optimization is not a new thing. If you read the research papers 20, 30 years ago, everybody would like to optimize and make the code efficient. Like there are a lot of tools like profilers that help developers find hotspots. The biggest problem in this area is, okay, we profile our code, but how we can automatically improve it so we make it faster. Typically, this is a very, very manual process and majority of the people that work on these areas are super specialized. And it's more and more difficult nowadays to find people that know how to optimize the performance of their code because the programming languages are becoming higher and higher level Like now people write more in language like Python, JavaScript, TypeScript. So nowadays, the real companies that deal with code optimization are companies in the lowers from the hardware space like big Intel, NVIDIA, and those, they write specialized software That take advantage of their hardware, where they show the hardware it outperforms or technology companies that need the scalability. But majority of developers will not necessarily bother about the performance of the code as an immediate first thing that they need to optimize. And we have tried and we have a platform that we helped practical engineers to automate this kind of process, to make it easier for developers to identify place in their code that are Slow and then to optimize it without necessarily having the knowledge that they need but also do it automatically. Now the history of code optimization, as I said, it was very, very manual process at the very beginning. Very few people know how to optimize code for specific hardware. You know, they had in the past, I guess they had to read books and compile their options. Eventually, then you had better and better compilers. So you can use compilation option to optimize your code and tune the options of the compilers, etc. And then you have a lot of profiling tools that help developers also optimize. But still, all these processes, most of the time, has been manual or semi-minual, right? And that's where we see the advances of AI helping this process. And to give you a bit of context, how we started our startup, we started after we published a paper in 2018, I think, that was in the foundations of engineering conference, where we showed That we could automatically help developers choose better data structures by taking a code, looking at data structures and optimizing it by giving variations. For example, sometimes in languages like Java, for example, you don't need to use an R-List if you potentially can use a Link List in a scenario that may be better. So we try to do these small changes and show that we have good performance impact. But majority of people at that point were manually translating code, like rule-based transformation, like regular expressions, if I see this pattern, convert it to this pattern. ([Time 0:03:49](https://share.snipd.com/snip/510eb4e2-ba51-459b-9208-fd3be4616556)) - **AI Code Optimization in CI/CD** - Integrate AI code optimization into your CI/CD pipeline, alongside testing and security scans. - Use it to get performance improvement suggestions on pull requests. Transcript: Speaker 2 Speaking of that, kind of getting into this latest wave of maybe what I'll just refer to as developer tools that are AI-related, again, people might be familiar with code generation Type of tools or explainers or something like that. There've also been, I've seen kind of agentic type of tools that will write up PR for you to do like you say, hey, I want to do this thing. And then there's a PR generated here, given the focus on code optimization, could you draw out some of just for people that are maybe kind of getting into this? Could you draw out some of when they would want to use this kind of tool versus some of these other maybe generative applications or applications of generative AI to code, I guess? Yeah. How does it fit into that ecosystem? Speaker 1 So the way we present the code optimization tool currently is part of your CI-CD process. So you make a pull request, then you will run some unit tests, you will have your integration testing, potentially you'll have a scanning, security scanning tool like SNIC or checkmarks, Et cetera. And then the next step is depending on where your application is deployed to analyze your code. And we tell you make those changes because you can have this 20% improvement in CPU and execution time, et cetera. That is as the way we present it currently as a CI-CD tool in the tool chain of the developer tools. However, if you think about the technology underneath and all dev tools, the way I see it are going to be using AI and they will take advantage of LLM-based solutions. From the moment you're using LLM, if you generate code or you translate code, it's a kind of the same kind of approach. It depends on the data that you will apply the LLM. So code generation tools, practically they see, when you say for the PR example, the way those LLMs have been trained is that they see that there is some comments below the code. So you say, if I give you those comments, can you predict the code? So in the code translation, where you don't know about the speed of the code, you say, I have seen this C++ code and the equivalent of Python code and LLMs do it. Like for example, copilot or charge PPT, they can translate code. Like if it's perfect, probably not yet, but that is a fundamental technology. So code optimization is on the same set of tools, but it says this is a slow code that I have seen. Now I have seen a variation of faster code. So I can recommend you this faster code. Eventually you can expose LLM, like any other tool that is built on VS code or LLM based, you can expose it in a editor and then the developers from our LLM will get suggestion for faster Code. That doesn't mean that will be beautiful code necessarily, but it will be faster for the hardware that you need to run it. ([Time 0:07:23](https://share.snipd.com/snip/ad0dc9e2-fd11-4a29-9ddc-5d5f8a877981)) - **LLM for Rapid Prototyping** - Mike Basios observed a backend engineer quickly create a UI with an LLM, despite not knowing the language. - He believes LLMs significantly boost developer productivity. Transcript: Speaker 1 So I was one of the very first believers of LLM assistive code generation tools. Like I tried to get bad access to GitHub, copilot, even try to use GPT-3 to see all this kind of technology. I'm very big believer because I have seen members of our team using LLM to build things much, much faster to code. For example, we had the backend engineer, we needed a prototype for a front end, he just used one of the close source LLMs and he could in one day he did a new UI that he didn't even know the Language, he didn't know the types of code. So I see that they are very, very promising. I see more users right now for good developers that already know, you know, the basic of computer science. I'm not a believer of, hey, you don't need to code, you will use this LLM for you. That are some videos, potentially they are over promoting that. But if you are a good enough developer, you know what you need and you use, they can practically help you dramatically to build easy applications, to generate tests, to generate comments About your code. And of course, the performance of, from our experience, I believe GitHub, copilot, and GPT are still outperforming the other models. But we see more and more open source models starting to become very, very good into the different languages. Like we have been trying Lama2, we have been trying CodeGen, we have been trying all of this and we even expose them to our platform so people can compare the results of those. Those tools will need to become a bit easier for developers and VS Code tools to use. Because by default, people, this API exposure that some of the close source models are giving is solving a lot of headaches for a lot of developers. That's why a lot of developers still have preference for this. But definitely open source models, they are very good. And I see them becoming even better if they are fine-tuned or specific language or specific context. I'll give you an example. If you want to do translation from SQL to an SQL, let's say we want to optimize SQL queries that people are doing on a data basis, you can take one of the open source models and fine-tune It on SQL and you probably will outperform GPT 3.5 or 4, all the context that you have. But I'm truly a very big believer. It's a bit of psychology between developers that, hey, but I am a believer that people that will use, they have advantage over people that don't use those tools currently. ([Time 0:17:38](https://share.snipd.com/snip/4c0b3aea-3deb-4a9e-b543-ee670b3d3295)) - **Enterprise Code Optimization** - Enterprises prioritize custom or on-premise LLMs for code optimization due to IP concerns. - Fine-tuning open-source models on proprietary data offers a secure solution. Transcript: Speaker 1 Things? It's very, very important topic and we have got a lot of experience with this and I will mention an example where we went to a client, very big technology firm, super big one of the best And we said, hey, you know, this is our platform, we have these LLMs, you can use any LLM of your choice like GPT-4 or open source LLMs like LAMA-2, et cetera. In the beginning, they said we don't have any approval for OpenAI because first they don't know the IP issue, a second they don't want the code to go outside, right? We're talking about proprietary code that a lot of sales companies. So the solution there was, okay, you can use the custom open source LLM on your data, we do not see anything, so it's a custom solution on premise and while they're using a product, then Practically our platform allows them to generate their own training data set. So for example, they use Artemis, they optimize code, they see sometimes the code is optimized, it's not optimized, but they generate the data and those data we cannot see and the client Should have those data for fine tuning their own model. Then through the platform, they can further fine tune their own model. That is how the industry will go, especially financial sectors and technology sector or defense companies, they will never give their code outside on an open source level. But LLMs are super powerful on that. With one client, they said, I'm not sure if I can get a recommendation, if you give me a recommendation from an LLM who is liable if that code doesn't work, right? What is IP issue? I said, how do you solve it now? They have the IP checkers, et cetera. I say you can just use IP checker for this moment, the same code, the same process that you would do, but also LLMs and we have added this functionality can do very good similarity search. If you have other code bases and similar functions in your code base, you can fine tune nicely or you can look like the same way people are building chatbots on your documents, you can Build your own chatbot on your code. It's practically similar to search. Then we could recommend, we even identified that three teams had implemented the same functionality a bit different. You even save time. Still, the underness technology, if you know how to use it properly, you can take advantage of it. The biggest example is Databricks about Mosaic ML because Mosaic ML, they acquired Mosaic ML that helps organization fine tune their own LLMs of their data because no organization Will give that data to open AI or any other company to fine tune on top of them, especially big organization that there is value on their data. The same thing applies for coding. That's how we see and that's why we adopted the product according. ([Time 0:26:43](https://share.snipd.com/snip/178965b1-7109-4f20-a924-0182f4b8ab9b)) - **Building LLM-based Applications** - When building LLM-dependent applications, start with closed-source APIs for easier deployment. - Address LLM hallucinations with unit tests, gradual code changes, and developer validation. Transcript: Speaker 1 If you're building applications on your application dependent on LLM output, then you need to make sure that as a first stage, I would recommend you something like a close source API Usage because it will solve you the headache of deploying your own LLM, having good CPUs. That is a problem that you cannot scale. At this moment, most teams don't know how to do it. Of course, there are a lot of startups, a lot of companies that are working on this on how to build your own LLM in a scalable way, but that can be a nightmare to build. If your business is not how to deploy an LLM and the value somewhere else, if somebody provides it as a service, it makes sense to use it. In our side, we say you can import with an API key and a secret key your LLM that you have access, then your application becomes much easier. Then you have the problem that you need to solve if the client okay, if those data are used on OpenAI or because there is a data that goes there. This in financial sector or currently cannot be accepted. Then our product will have to deploy LAMA2, have to deploy it. So then you have to build it yourself for use of service. So we had to build it because we were one of the early adopters, but there are tools. Hugging phase provides a very nice API for you to deploy. Now they change the license, I think, but I think majority of people can use it. So the speed of LLM, this is a big problem for scaling application definitely. Then other issues about LLM, which were a bit at the beginning, now they are trying to be fixed this token size. Every time you ask and the result may be incomplete, if there is a then how do you deal with the previous context and all this kind of, you need to spend time with it to do it properly. I am expecting more and more tools and open source are solving like Langchain and those tools also open source are doing some of the things. And of course, the biggest problem that a lot of people talk is about hallucination of the models. You cannot trust necessarily the models and you cannot just say generate code and execute that code in your backend because somebody may do SQL injection, similar to where people are Doing SQL injection in the past, especially for coding, you can have LLM kind of injection. So you need to be very careful on exposing the prompt to the end user because somebody can really damage. So those are ([Time 0:30:42](https://share.snipd.com/snip/59f44293-83b7-4feb-8482-321d0ac49e1e))