How Oracle Is Meeting the Infrastructure Needs of AI  ## Metadata - Author: [[The New Stack Podcast]] - Full Title: How Oracle Is Meeting the Infrastructure Needs of AI - Category: #podcasts - URL: https://share.snipd.com/episode/0df02c12-6b0e-43fd-8a1c-6a3741a08e11 ## Document Notes ## Highlights - **GPU Workloads Demand Peak Power** - GPUs are constantly running at peak power during AI workloads, unlike CPU web workloads which have peaks and troughs. - This leads to higher power consumption and hardware failure rates, requiring new infrastructure solutions like large GPU superclusters. Transcript: Sudha Raghavan So right after ChatGPT's release, Gen.AI has been literally taking over the world. It is the fastest adopted technology today. Daryl Taft Yeah. Sudha Raghavan And along with that came this rush of GPU requirement. Everybody wants a GPU, whether they know how to use it or even if it is the right choice for the problem they're trying to solve. We are seeing a lot of GPU demand. Now, GPU demand, as you rightly pointed out, can go from a very small, I want to test something, to hundreds of thousands of nodes in one cluster running one job. Like a big, large, bad job, training job, right? And that space has really not been explored. We have hyperscalers in the cloud. So there have been 100,000, 200,000 nodes running more edge web applications, but all of them acting together and running one job for days and weeks and months to do the training is brand New. Now, that brings up a special set of requirements where we're seeing the power consumption for these things. If you can imagine when for web workloads, you have a peak and a trough. Right. But these GPU batch workloads, there is no trough. Everything is running at peak all the time. Yes. And so the demand of power, the failure rate of the hardware. Yeah. Extremely accentuated. Yes. One node out of 100,000 nodes. So OCI is building the largest supercluster for GPUs yet. We're going to build about 131,000 plus GPU node cluster to run one job for one of our largest customers. Now, can you imagine if one of those nodes goes down and the whole job just stops? What does that mean? Have we lost all the things that have been? Because these are stateful workloads. So the node has to checkpoint so often into a common distributed storage that that one node going off, maybe for a few minutes, you lose what the node was doing because it didn't persist. But it's not for days, it's not for hours. And then when the node comes back up, it needs to pick up where the job is, not where it left it. So again, the state management of the overall job is very critical, very important. And that's something we haven't seen with traditional cloud-native CPU workloads. And again, with that, these nodes, because they have to interact in one job, they want to be placed close. They want microsecond level latencies acceptable only. When customers used to come with these 100 node requirements to us, they didn't tell us place these nodes in the same rack and close to each other. Nobody gave us those physical requirements. Now, because networking is an absolute must, we are running large scale, single workload batch jobs that all need to work together to do one job, they all have to persist state very often Into a common distributed file store. So everything from the PCI bus throughput to your RDMA throughput to the actual networking throughput inside, everything matters now. Everybody is talking about, you know, if you give me a thousand nodes, give it to me in the same data hall. Give it to me on the same rack. I don't want any latencies. We used to talk about latencies in terms of multi-cloud, like hybrid cloud, you know, database to application latency. Can you make sure that that is only a few milliseconds? But now even that is not enough. They're literally saying, place it on the same rack if possible. Daryl Taft Let's go back to the roots of Kubernetes, right? You know, how it was originally architected. It was originally architected for CPUs. Sudha Raghavan And stateless workloads. Daryl Taft And stateless workloads. Sudha Raghavan Yes. Daryl Taft So, now let's fast forward to today, and looking at how Failover was architected at that time. Absolutely. Resiliency was different. Yeah. What was different? And so what are you doing now to help increase the resiliency? It seems like physical location is important. Sudha Raghavan Physical location is important. The mechanisms to checkpoint data have become really, really important. So how do you connect your GPUs to a clustered file storage, distributed file storage service? That's the checkpoint? Yeah, which then allows you to do the checkpoint for the job. Daryl Taft So you're checking, when checkpoint means the check, and what does the point mean then? Sudha Raghavan No, no. So checkpoint basically means if a job is divided into 100 pieces, I tell a central piece of software, storage, whatever you want to call it, I am done with part one. Daryl Taft Okay. Sudha Raghavan So if I go down, nobody else needs to pick part one and rerun it. Daryl Taft Okay. Sudha Raghavan So checkpointing means I am letting the system know how much work I have done. Daryl Taft Is that manual? Sudha Raghavan Hopefully not. It can't be, right? No. When you're talking about like 100,000 jobs, 100,000 nodes running one job. Daryl Taft But that's kind of an addition to the Kubernetes architecture that's had to be made. Sudha Raghavan Exactly. Exactly. So that's one important aspect that people are now realizing how to embed it into Kubernetes natively. ([Time 0:01:50](https://share.snipd.com/snip/dd9056d2-1170-4eba-b035-caaa97b3baf9)) - **Checkpoint and Kubernetes for GPUs** - Implement checkpointing carefully to save progress frequently in stateful AI jobs and enable resumption after failure. - Enhance Kubernetes with GPU-specific features like node placement and workload scheduling for efficient, resilient AI operations. Transcript: Sudha Raghavan And stateless workloads. Daryl Taft And stateless workloads. Sudha Raghavan Yes. Daryl Taft So, now let's fast forward to today, and looking at how Failover was architected at that time. Absolutely. Resiliency was different. Yeah. What was different? And so what are you doing now to help increase the resiliency? It seems like physical location is important. Sudha Raghavan Physical location is important. The mechanisms to checkpoint data have become really, really important. So how do you connect your GPUs to a clustered file storage, distributed file storage service? That's the checkpoint? Yeah, which then allows you to do the checkpoint for the job. Daryl Taft So you're checking, when checkpoint means the check, and what does the point mean then? Sudha Raghavan No, no. So checkpoint basically means if a job is divided into 100 pieces, I tell a central piece of software, storage, whatever you want to call it, I am done with part one. Daryl Taft Okay. Sudha Raghavan So if I go down, nobody else needs to pick part one and rerun it. Daryl Taft Okay. Sudha Raghavan So checkpointing means I am letting the system know how much work I have done. Daryl Taft Is that manual? Sudha Raghavan Hopefully not. It can't be, right? No. When you're talking about like 100,000 jobs, 100,000 nodes running one job. Daryl Taft But that's kind of an addition to the Kubernetes architecture that's had to be made. Sudha Raghavan Exactly. Exactly. So that's one important aspect that people are now realizing how to embed it into Kubernetes natively. Second aspect, these GPUs are super expensive. Traditionally, CPUs have not had a lot of high utilization, but they're still 10x or below cheaper than what a GPU costs. Now people are really wanting to use the GPU for the maximum power. So they're like, you know, if my GPU finishes one job, I have 10 queued up. Don't keep the GPU idle. I'm paying for, this is not a by use. This is not your function. This is not serverless, right? I'm paying for the bare metal. I'm paying for that chip. I better use it the most efficient way. So we have things like queue and multi-queue, which are coming to the Kubernetes ecosystem to help you with that scheduling need. And then there is observability. Who would have thought that I would need to check the temperature of the chip on the GPU to make sure it doesn't fail? I'm trying to preempt failures. If GPU is running really, really hard for a long period of time, and the temperature changes, my goal, this chip is going to die. So checkpoint, reduce the workload, see if the thing recovers, otherwise pull it safely out. So new observability, new types of metrics in the system. Daryl Taft Because you have to safely pull it out and then make sure the data is continuing to be collected on those remaining servers. Exactly, exactly. Sudha Raghavan Right. So it's very, very interesting, the different dimensionality of the problem that GPUs are solving, how new features are being requested from the Kubernetes ecosystem. Daryl Taft And so when we were talking at Oracle Cloud World, you mentioned the need for better metrics. So you can be able to better understand the different GPUs by different backend makers, right? Exactly. Sudha Raghavan Bitterend chip makers, right? And normalize that for the customer. The customer shouldn't need to do a different monitoring. We have standardized CPUs, why not GPUs? And we are trying. Daryl Taft So I think you've been touching on the challenges that come with metrics and monitoring. Absolutely. And how observability changes in this environment because now it's a much more of a physical switchover in many respects. So what are you asking of the big GPU maker? Sudha Raghavan Yeah, so we work a lot with NVIDIA. And NVIDIA gave us a of data center manager, right? They gave us software that a customer on their choice could deploy on these GPUs to help them collect those extra GPU metrics that are so important to guide their jobs to provide the right Business output that these training models are expected to provide. And not only are we doing that with NVIDIA, we're doing that now with AMD. Before even they come out with their MI300, this is going to be norm. It's not an after-the NVIDIA is adding it after. But we're now saying this is standard pattern. Chipmakers have to now provide metrics that they were not providing before because it's important for the software that's running on it. And cloud providers like us try to abstract that from even the Kubernetes layer. So it doesn't have to know, oh, I'm running on an NVIDIA chip versus an NVIDIA chip. Yeah. I can query one operator interface and it's going to tell me what I need to know to run the Kubernetes job. Daryl Taft Is there an operator that's been being developed right now? Sudha Raghavan Yes. Yes. What's it called? So in Oracle Cloud, we added a thing called Node Manager. Okay. Helps all of these shipmakers give plugins to that Node Manager as the single interface that Kubernetes will talk to in order to collect data. So my Node Manager will know what hardware it is, but the Kubernetes infra is abstracted from... Node Manager provides the single-layer API, and Kubernetes infrastructure is now abstracted from the actual chip. So while I spoke about all the hardware problems, if we start exposing every hardware problem to layers like Kubernetes, which is already considered complex, it's not going to work. So we, cloud providers, are trying very hard to not increase the complexity of something like Kubernetes, but providing the functionality that Kubernetes users are accustomed to. ([Time 0:06:06](https://share.snipd.com/snip/328119b1-8811-4caa-9fed-93f35ab60c9d)) - **Abstracting GPU Metrics Uniformly** - Different GPU chipmakers present varied metrics which complicate monitoring. - Oracle's Node Manager abstracts hardware differences into a uniform API for Kubernetes to simplify cluster management. Transcript: Daryl Taft I think you've been touching on the challenges that come with metrics and monitoring. Absolutely. And how observability changes in this environment because now it's a much more of a physical switchover in many respects. So what are you asking of the big GPU maker? Sudha Raghavan Yeah, so we work a lot with NVIDIA. And NVIDIA gave us a of data center manager, right? They gave us software that a customer on their choice could deploy on these GPUs to help them collect those extra GPU metrics that are so important to guide their jobs to provide the right Business output that these training models are expected to provide. And not only are we doing that with NVIDIA, we're doing that now with AMD. Before even they come out with their MI300, this is going to be norm. It's not an after-the NVIDIA is adding it after. But we're now saying this is standard pattern. Chipmakers have to now provide metrics that they were not providing before because it's important for the software that's running on it. And cloud providers like us try to abstract that from even the Kubernetes layer. So it doesn't have to know, oh, I'm running on an NVIDIA chip versus an NVIDIA chip. Yeah. I can query one operator interface and it's going to tell me what I need to know to run the Kubernetes job. Daryl Taft Is there an operator that's been being developed right now? Sudha Raghavan Yes. Yes. What's it called? So in Oracle Cloud, we added a thing called Node Manager. Okay. Helps all of these shipmakers give plugins to that Node Manager as the single interface that Kubernetes will talk to in order to collect data. So my Node Manager will know what hardware it is, but the Kubernetes infra is abstracted from... Node Manager provides the single-layer API, and Kubernetes infrastructure is now abstracted from the actual chip. So while I spoke about all the hardware problems, if we start exposing every hardware problem to layers like Kubernetes, which is already considered complex, it's not going to work. So we, cloud providers, are trying very hard to not increase the complexity of something like Kubernetes, but providing the functionality that Kubernetes users are accustomed to. Whether it's through KubeCTL, all your known interfaces, we will just provide more stuff because now you're running on a GPU. Huh. Right. And another very interesting thing, you know, when Kubernetes ran on CPUs, they didn't, and they said launch a cluster, nobody told you, oh, I need these CPUs to be close to one another, To be next to where my data is. Nobody gave you all of those clues. They just said launch the cluster and whatever VMs you have or bare metals you have. That's part of my tenancy. Now, they're actually telling us. So we are adding inside our Kubernetes interface ability for customers to provide placement information. Daryl Taft Okay. Sudha Raghavan So place me next to this job. Place me like, so next to this cluster. So we're adding all of those. You don't have to, if you're running a small job, you don't have to. Daryl Taft So all these things are being added into the infrastructure software environment. ([Time 0:09:50](https://share.snipd.com/snip/79754839-3881-4bca-8c1e-1fd76ebf3947)) - **Oracle’s Internal AI Studio Journey** - Oracle developed an internal AI Studio platform to manage model training, serving, and data governance. - The platform enables reuse of AI services across thousands of developers, ensuring efficiency before public release. Transcript: Daryl Taft So when you think about DevOps and data science, you have to start thinking about things like CICD, for instance. Sudha Raghavan Yeah, so that's a great question. In fact, you know, inside Oracle, obviously we're building a lot of generative AI services. And so we have this concept of what we call internally an AI studio, which is a platform that allows you to quickly set up everything from your model training infrastructure, serving Infrastructure, data governance piece. Data provenance and governance is very important because you don't want to spread the data. How do you do compliance over this data set? What is your inventory of all the generative AI services? We are an enterprise. We have over 40,000 Java developers. If somebody in the company has built a solution, just reuse that. Don't go rebuild. So we are providing an AI registry for Gen AI registry for all the services that come in. So we are platformatizing and we strongly believe in something like an AI studio. Once we are able to use it internally, it will be a no-brainer for us to release that to the public. But we first always try to build it for our own use cases, dock for it, make it perfect, and then release. ([Time 0:19:04](https://share.snipd.com/snip/00b3e2b9-319f-4bfb-8613-f2a7fe13657a)) - **Fleet Management for AI Clusters** - Manage AI clusters as fleets to support diverse workloads like training and inference on the same infrastructure. - Adapt configurations per fleet to optimize resource use and reliability across multiple simultaneous AI jobs. Transcript: Daryl Taft How are these new AI workloads affecting cluster management and scheduling? Sudha Raghavan So we were just talking to a customer of ours. Cluster management is very, very interesting. Actually, fleet management to be even more specific, right? Imagine you're a company that is running a GPU cluster of a thousand nodes. It's not too large, not too small either. You run one training job on it. The training job completes. It takes whatever weeks, days, whatever it takes, it completes. Now I want to run a bunch of inferencing workloads because that's my business. On the same cluster. I do not want to redo the infrastructure. But then the characteristics of a thousand nodes running one job versus the same thousand nodes, but running a hundred jobs, very different. Daryl Taft Yes. Sudha Raghavan So cluster fleet, if you say cluster management, it's still one cluster. Right. So that's not too different. But if you say fleet management, now, in the previous case, I had one fleet to manage. Now I have 100 fleets to manage. And both the configurations, what I look for when I see failure, they're both different. ([Time 0:21:58](https://share.snipd.com/snip/374e642d-138d-4c80-897f-4f37e2a52ec3)) - **Balance Scale with Business Value** - Sustainable scalability in GPU infrastructure means balancing power consumption and utilization while delivering business value. - Choosing the right AI model size for the actual business problem improves efficiency and reduces costly failures. Transcript: Sudha Raghavan For us, it's all about sustainable scalability. There's so much demand today for GPUs. And as I started this, I was saying, whether you want it or not, you want one just in case. With the market going at that rate, we want to be able to provide customers with the ability to do that, but in a sustainable way. Meaning, the power usage of this thing. Being able to run efficiently. I was talking about how everybody wants to use 95%. How do you get the infrastructure that's always by default doing that? Last but not the least, how do you provide the business value that you're wanting from this Gen AI? Using Gen AI is cool, but at the end of the day, if it's not adding to my P&L, it's not going to work for long. So how do I get from, okay, I got these GPUs, I got these models, they work great, to, oh, now, because of that, here's the value I'm getting. We need to make that cycle very, very quick. Daryl Taft So that's the challenge and how to do it quickly. And then I expect then the trade-off is like, how do you, you know, what do you do to make it quick but not fail? Sudha Raghavan Yes, make it quick, not fail. Be very flexible. So you can run a 10-node GPU cluster without a lot of overhead, but you're also able with the same cloud provider to run a 100,000 node cluster with all the nuts and bolts that you need to Care about because now it's such a big cluster. Daryl Taft So do you have to make trade-offs to your pipelines, to get that velocity but without failure? What are those things that you can safely say, we can trade this out? Sudha Raghavan So in my opinion, the trade-offs to figure out what model to use for what business problem. Do you use a SLM and LLM? If you use an LLM, do you use a 1 billion parameter which can run on CPUs, you don't need GPUs. Do you need 7? Do you need 400? How do you make that decision? I am a business owner. I want to solve this business problem. Don't make me pick what is the right AI model to run for my use case. Daryl Taft So in some respects, the trade-off is just thinking it through a little bit better. Thinking it through a little bit. And not just going with the biggest parameter, you know, LLM. Sudha Raghavan It might not be needed at all. Daryl Taft And if you do that, you've got to understand, it may take a little longer, but if you use an LLM with a lot more parameters, you may face a failure you don't want to have. Sudha Raghavan Exactly. And it's, again, business domain dependent. Text summarization does not need a 405 billion parameter model. Daryl Taft Right. This is an interesting ongoing discussion. Right. ([Time 0:23:26](https://share.snipd.com/snip/eea87756-3ed2-473b-aeb3-99d991a19d1b))