Why Kubernetes Cost Optimization Keeps Failing  ## Metadata - Author: [[The New Stack Podcast]] - Full Title: Why Kubernetes Cost Optimization Keeps Failing - Category: #podcasts - URL: https://share.snipd.com/episode/3970e8f4-96ae-435c-b323-a0603c40890c ## Document Notes ## Highlights - **Why Kubernetes Cost Optimization Fails** - Kubernetes cost optimization is hard because applications are dynamic with changing loads requiring static resource allocation. - Thousands of applications with different needs and many stakeholders make manual optimization fail, leading to up to 80% resource waste. Transcript: Speaker 2 First of all, why is it so difficult to optimize costs for Kubernetes? Speaker 1 Yeah, so it's a huge challenge, actually. Eventually, today, we live in a cloud-native world where applications today are really dynamic. The load always changes. And for every application, you have different business contexts. Every application has different needs. In addition to that, the load, the resource consumption of every application is really dynamic and can always change. And you have the application owners, these are the engineers, the application owners, the ones who are responsible for the application, they need to set a static size of how many resources They should allocate to every single application. And it's really difficult because you need to make sure you always have enough capacity to handle every spike, while on the other end, you don't want to spend and waste too much money On resources that are under provision. Now, in a production cluster, you have not a single application and a single owner. You have thousands of different applications where each one of them has, again, a different need and a different behavior. And you have so many stakeholders that always need to make sure that every application is always optimized and adjusted. And obviously at scale, it just doesn't work. And it leads to a lot of application causing performance issues, and also a lot of cost and a lot of waste. Usually we see 80% or 70% waste on over-provisioned resources and actual waste, and companies just fail to manage it because the environment today are static, and there are so many Stakeholders. ([Time 0:01:54](https://share.snipd.com/snip/e15672ba-35f8-469b-a203-60730be17ff1)) - **AI Workloads Increase Resource Strain** - AI workloads increase resource demands, especially for GPUs, CPUs, and memory. - The fundamental problem remains static resource allocations that need manual adjustments. Transcript: Speaker 2 Another thing that occurs to me about what companies are dealing with right now is AI, and AI workloads, and presumably that would have some impact too, right? I mean, because that seems to be, when people are running AI workloads, they often are kind of unpleasantly surprised when they get their bill for cloud, and is that also a factor now? Speaker 1 Yes, so AI workloads would be, we see the growth of more and more workloads of AI requiring GPUs, CPUs, and memory. And again, the problem remains the same. The resource allocation of those workloads is static, and someone needs to manually adjust and set the resources in a static way. So we see the growth of those, but the problem remains the same for all different types of workloads. ([Time 0:03:49](https://share.snipd.com/snip/83ed9558-26c6-44fc-9c59-56970372b2c2)) - **Avoid Static Recommendations** - Do not rely on static recommendations for Kubernetes optimization as workload dynamics make them quickly obsolete. - Avoid manual application of recommendations that often are not trusted by application owners prioritizing performance over cost. Transcript: Speaker 2 What are some of the most common solutions or fixes for optimizing Kubernetes cost? Speaker 1 Yeah, so up until now, the best way, the things that companies try to do, they try to work with the tools that provide the recommendation, that provide visibility to the problem and provide Every single workload a recommendation that someone needs to apply the recommendation. The problem with those solutions is that, again, the resource consumption is dynamic. It always changes. And recommendations that were relevant in the past would not necessarily be relevant for the future. You might have some unpredictable spike that does not match the recommendation, and then it would lead to performance issues. It can even lead to downtime. So what we see, we see that companies just don't apply those recommendations because they are afraid of those. Not only that, the ones who are responsible for the application, the ones who need to apply the recommendation are the application owners. And they usually don't care less about the cost, they care about the performance. They want to make sure that their application would run properly. So they tend to not trust the recommendation, and it requires repeated manual work of always applying the recommendation as the recommendation always change. Again, we live in a cloud-native environment where applications are dynamic. ([Time 0:04:51](https://share.snipd.com/snip/ec39cb5e-699c-45bc-9565-d88572e1d075)) - **Static Limits of Kubernetes Automation** - Kubernetes auto-scaling configurations are static and require manual threshold setup. - Dynamic demand changes require automation that adjusts scaling triggers in real time for optimal resource allocation. Transcript: Speaker 2 Yeah. So, I mean, it's, let's talk about automation. Like, one of the benefits of Kubernetes, it can scale up, it can scale down. You're having a spike in demand, you can, you know, you can scale up or you can go back down. What are some of the limitations of automation, especially at the enterprise level? Speaker 1 So Kubernetes is an amazing tool, eventually. Companies can set it with different configurations. You can define when do you want to do horizontal scaling, when should we have an additional node that would be provisioned to the cluster? You can configure plenty of stuff. But these eventually, all the configuration that you can do within Kubernetes are static configuration that someone needs to set. And as the cloud is so dynamic and the demand always changes, the problem with those triggered, the thresholds that you set for the scaling, they are static. Someone needs to define them. And as the application always change, the demand always change, you need something that will make sure that the trigger, the thresholds, are always being up to date with the relevant Things that are relevant for what exists today in the cluster. What is relevant for the spike we're having right now. You need something that would be dynamic. ([Time 0:06:26](https://share.snipd.com/snip/6d4b80e9-94cb-4299-bd05-65a9944d20a2)) - **Focus on Full Automation** - Focus on fully automated, dynamic resource allocation per application to balance performance and cost optimally. - Automation at container level can eliminate friction between application owners and DevOps seeking cost reduction. Transcript: Speaker 2 What should people be focusing on, in your view, in terms of optimizing costs for Kubernetes? Speaker 1 Yeah, so funny enough, today most companies are focusing on the visibility and the recommendation. But the problem with those solutions is that it shows you the problem, but it shows you that there's a huge waste and there are many things that you can do to optimize. But they don't provide companies with how to achieve those savings, how to get the value. So, in my view, in the way what we get from the market and what we see from customers, The biggest problem today in cloud resources is making sure that at any given time every application Would get the right amount of resources in a fully automated way and actually, free the biggest friction that exists today in organization, between the application owners who want To make sure their application gets the right amount of resources in terms of performance, while you have the DevOps and the platform teams that also want to reduce the cloud spend. So optimization with full automation for every single container can free this burden and eliminate the biggest friction that exists today. Okay. ([Time 0:08:00](https://share.snipd.com/snip/6d57cb57-0ac6-4906-b1b3-ca97cf906569)) - **ScaleOps Dynamic Resource Management** - ScaleOps manages resources dynamically at the container level in real time to meet demand and reduce waste. - This approach shifts Kubernetes resource management from static to dynamic allocation, improving performance and cost efficiency. Transcript: Speaker 2 And free them from that manual having to set things, having guess, basically, what it's going to need. Exactly. How does ScaleOps tackle this problem? Speaker 1 Perfect. So, ScaleOps, what we do, we manage the resource allocation at the container level in real time in a fully automated way. What we do, we make sure that every application gets exactly what it needs. We know to monitor the resource allocation and many other different business metrics to always make sure that the performance is performing best, and we actually also improve the Reliability and the performance of every single application, while we eliminate all the waste that exists, all the over-provisioning. We actually make shift Kubernetes from a world where all the resource allocation is static, we change it to be dynamic. So the pod level, every container would get exactly what it needs at any given time. And then companies can just forget about all the process of always chasing recommendation, chasing the application owners to set their resources properly. With ScaleOps, we just automate all this process and let companies focus on innovation and building stuff. ([Time 0:09:22](https://share.snipd.com/snip/a5d23568-47d3-4c9c-9961-dbcb79a01937)) - **Origin Story of ScaleOps** - Yodar Shafrir founded ScaleOps after seeing partial solutions fail at Run AI for managing diverse workloads. - He partnered with Guy Baron who experienced firsthand the pains of manual cloud resource management. Transcript: Speaker 2 How did you decide to start ScaleOps? What's the origin story of ScaleOps? Perfect. Speaker 1 So before founding ScaleOps, I was at a company called Run AI, where what we did was to better orchestrate AI workloads on GPUs and eventually optimize those GPUs and help companies To reduce the cloud cost. I was responsible for the Kubernetes team over there, and I got to meet plenty of customers. Eventually, we sold to DevOps. And what I started to realize was that companies really love the solution, but at the end of the day, they see it as a partial solution. They have huge production clusters with all kinds of resources, a lot of CPUs and memory, storage network, and they were failing to manage all types of resources properly. They were looking for a solution, something that would help them to better manage all the orchestration of all types of workloads. So I realized that the market is pretty much broken. Companies are looking for a solution and that's actually the biggest problem that exists today in the cloud. I left RunAI, started to work on ideation, trying to figure out what can I build in those areas. And during this time, I met plenty of potential customers. And that's eventually how you get feedback for different ideas. And that's how I met my partner today, Guy Baron. He's the CTO of the company. Back then, he was the head of R&D at Wix. And I met him when I tried to pitch him the idea of Scalops, the potential customer. So obviously, he came with all the perspective of the customer. His team always needed to worry about adjusting the cloud resources, repeated manual work that they needed to do every week with a lot of waste. So it was a very good sales pitch. And after a while, Guy and I started to know each other and we had a really great connection. And we realized that we perform really good together and we need to start something together. ([Time 0:12:49](https://share.snipd.com/snip/146ba1cc-d1e6-4f64-b014-bafb645c9077)) - **Planned Roadmap for Deeper Optimization** - Optimize nodes by selecting best instance types and running suitable workloads on spot instances safely. - Incorporate pricing plans and optimize GPUs and network resources for deeper savings. Transcript: Speaker 1 Yes, so we are, soon we'll be launching node optimization where we will always make sure that the best instances will be selected. In addition to that, we are going to launch spot optimization. We are going to use the fact that we're application context-aware, and we know exactly what every application needs at any given time, and find the workloads that can run on spot instances, And making sure that you run it the most efficient way in terms of the cost, while also making sure that the interruption of spot instances will not cause any issues. Obviously, we'll do that while taking into account our rise and savings plan. And we are going to launch optimization for GPUs and for network. So these are the things that we can share. ([Time 0:15:33](https://share.snipd.com/snip/ac634e64-b6cf-43a9-a3ca-78a67ffdbe4d))