Kubernetes GPU Management Just Got a Major Upgrade  ## Document Notes ## Highlights - **Hardware Triggered Kevin's AI Moment** - Kevin's 'aha' came when GPU hardware matured enough to properly accelerate long-standing algorithms. - He credits hardware advances, not just algorithms, for mainstream AI momentum. Transcript: Kevin Klues Yeah, it's funny. So as Jesse was saying that, I was actually thinking, my kind of aha moment was more once the hardware reached the maturity level that it had. So these algorithms have been around for decades, but until we had the hardware that could actually accelerate it properly, that's when it really started to take off in the mainstream. ([Time 0:04:38](https://share.snipd.com/snip/71b4df11-0920-4711-b15a-01074c1bf6fb)) - **DRA Reimagines Hardware Allocation** - Dynamic Resource Allocation (DRA) models hardware allocation after Kubernetes storage primitives like PVCs to expose specialized devices cleanly. - DRA lets users request device types and configurations rather than just a GPU count, shifting complexity into scheduler and drivers. Transcript: Kevin Klues Entire DRA model is sort of built around a similar model that was designed for storage systems years ago. If you've ever worked with persistent volumes, persistent volume claims in Kubernetes, the abstractions that DRA provides look very similar to that. You create this thing called a resource claim. You've got this thing called a device class. And so, you know, the ideas aren't really new, but we built DRA to be able to allocate and make use of generic resources, not just storage systems. And in fact, Patrick Oley, who's one of the other leads on the DRA project, he said that if he could go back in time, he would have written all of the storage system stuff in terms of DRA instead Of the way that it evolved back then. And so, you know, that's how it kind of had its start, but it's evolved into a lot more than that now. You know, Kubernetes kind of hit this point where, you know, we needed a way to not just work with fungible hardware. We needed a way to actually get access to specialized hardware and request exactly the type of hardware that we need access to. And the abstractions we had before didn't really allow us to do that. Because the only knob you had in the old way of requesting access to resources was a simple count. You could do something like ask for nvidia.com/gpu2. You could say I want two GPUs, but you couldn't say what type of GPU. You couldn't say how you might want that GPU to be configured once it's given to you. And DRA really allows you as an end user to express these more complex constraints on what you want access to. And then, you know, the complexity of how to actually make that happen is built into the Kubernetes scheduler, into these vendor specific plugins that are running on individual nodes To make all of that kind of transparent to the end user. And at the end of the day, all they see is that their workload now has access to the hardware, exactly what they asked for. ([Time 0:10:31](https://share.snipd.com/snip/4d5a0631-50f6-4a15-af4c-340d618eb51d)) - **Adopt DRA Instead Of Custom Controllers** - Use DRA when available so you avoid writing custom controllers and CRDs to claim specialized hardware. - Follow cloud vendor guides (e.g., AWS blog) to integrate DRA in a few YAML lines instead of reinventing scheduling logic. Transcript: Jesse Butler Use Kubernetes and DRA. Here's a blog. It shows you how to do it in three steps. There's like 18 lines of YAML. You don't have to write a CRD. You don't have to figure out those things. The really nice thing and elegant thing about it is that you can configure it if you need to. You can get in really deep. There's granular configuration that's available to you, which kind of gets back to that thing of abstractions. Abstractions still need to respect the power that they're abstracting, right? If you want to build a fleet of agents that are going to run, that's probably going to need like inference. You're going to need a model. You're going to need GPUs to run that. Kubernetes is where people are doing that and DRA and that work that Kevin and his team did enable that. ([Time 0:13:40](https://share.snipd.com/snip/3d350c94-7786-4e0b-a093-0e74d1ffe50a)) - **Workload Abstraction For Gang Scheduling** - Many AI workloads require multi-pod, multi-node coordination that single-pod scheduling can't express. - Kubernetes' upcoming workload CRD will let users gang-schedule pod groups with topology and strict start-all-or-none rules. Transcript: Kevin Klues Lots of times you want multiple pods in a multi-node workload to come up simultaneously. You want to be able to express things like, I can have some subset of these pods come up, but if I can't get all of them or I can't get exactly that amount, I don't want any of them to come up. World. And so we're working on a new abstraction called the workload abstraction, which, you know, DRA reached GA in Kubernetes 134 back in August. There's obviously still lots of work to do in that area, but this new workload abstraction that we're working on is kind of the next big thing that we really need to pour a lot of resources Into to make it a reality in Kubernetes. What is it exactly then, a workload abstraction? It's going to be a CRD. So it's a way to express both a grouping of pods that have some type of scheduling constraints put on it. So you might say I want this group of pods to be gang scheduled. They all have to come online or none of them should come online. And on top of that you can specify topology constraints there. So you can think of it kind of as a node selector on steroids. Instead of just saying this pod needs to land on this node, you can say this group of pods needs to land within these topological constraints subject to the scheduling constraints you Put on it, like whether you want it getting scheduled or not. There's, you know, none of this is, so a very basic implementation of this entered, or is about to be included in Kubernetes 135 coming out next month. ([Time 0:16:07](https://share.snipd.com/snip/518cadd1-5a8b-4905-a52e-8eb44b0c672a)) - **Agents Will Be Treated Like Microservices** - Agentic AI will push Kubernetes users to treat fleets of agents like microservices that must interoperate reliably. - Infrastructure must provide seamless substrate (scheduling, resources, topology) for agents to scale and communicate. Transcript: Jesse Butler We're now looking at inference and customers wanting to build agents, fleets of agents, making them work together. We're sort of seeing like the LLM having moved the stack up. And so I think customers are starting to realize that agents are sort of like a microservice. So you can start architecting sort of whole applications or whole systems using agents where you would have maybe had a microservices architecture. What that means is those need to work seamlessly. So what we don't want to do is walk away from all the problems we've solved over the last 10 years with getting processes up and running on CPUs through Kubernetes. So looking at things like not only DRA for resource allocation, but also like new types of scheduling, new types of like defining workloads are going to be really, really important Because we don't really know what customers are going to build over the next 12 to 18 months or what those patterns are going to be. But we're pretty sure that it's going to be something around agentic AI and fleets of agents working together, sort of informing each other through context, having shared tools, looking At things like A to A, agent to agent communication. All of the substrate below that needs to just work. ([Time 0:18:30](https://share.snipd.com/snip/f8d33a1d-1645-48de-9e8e-ecf3a8dacb23)) - **Preserve Legacy Requests During Migration** - Ensure DRA drivers can honor legacy GPU resource requests to avoid breaking existing workloads. - Prioritize backward compatibility to accelerate DRA adoption across users and environments. Transcript: Kevin Klues My perspective, the most critical feature which hasn't kind of reached a level of maturity that it needs to in the DRA space is this ability for DRA drivers to honor the old-style resource Requests. Because until we have that capability, DRA is not going to gain widespread adoption. Because not everyone is going to want, because DRA is a different model for how you request access to resources. And if you don't have a way for people to take their existing workloads and run them on a DRA enabled system, it's going to be really slow for it to gain traction. And so we have a feature that we're working on right now. It's still in alpha, even in 135 coming up, but we're hoping to graduate it to beta in 136. And with that in place, it'll be transparent to end users, whether they're running a DRA, whether their admins have decided to run a DRA driver on the system or the device plugin, their Workloads will just continue to work even if they're asking for resources without this more complex, well, I shouldn't say complex, this more feature-rich API that DRA provides them. Alex Williams Because people are working on all kinds of, there's all kinds of resources out there. And so you need to be able to have that capability to... Kevin Klues Well, I think it's more that people are slow to migrate to the new APIs. They want to be able to continue running the things they've been running with the exact same pod specs, manifests that they've had up until now, and they don't want to change everything They're doing just to accommodate DRA. ([Time 0:21:20](https://share.snipd.com/snip/07074804-9699-4a97-9423-246aa9e0d4ec)) - **Topology Standardization Matters** - Topology keys and conventions are currently inconsistent across environments, complicating portability. - Standardizing topology and conformance (including DRA driver conformance) is essential for enterprise portability. Transcript: Jesse Butler Topology has sort of been the Wild West for a while. And so if you look from sort of environment to environment or product to product, we've all kind of come up with our own topology keys. And as kind of boring as that sounds, it's really chaotic for trying to like make something work in multiple environments. So again, kind of respecting the reason for Kubernetes like as a standard, you want to adopt it, especially as an enterprise, it's portability, right? It's being able to just use these APIs. You don't want these differences between environment to environment. So those are some of the things that we're looking at trying to kind of just unify for our part in the community. ([Time 0:23:30](https://share.snipd.com/snip/499069d6-4054-489d-abfb-e1cde40a2511)) - **Slice GPUs Dynamically With MIG** - Use GPU slicing (MIG) to run many small models efficiently on shared hardware. - Let DRA dynamically create MIG devices so workloads get sized GPUs without manual reconfiguration. Transcript: Kevin Klues Optimizing across small language models versus large language models? Yeah, sure. So I don't know how familiar you are with a feature called MIG that NVIDIA GPUs provide. It's a way to take a full GPU and subdivide it into a smaller set of mini GPUs you can think of it as that are fully isolated from one another in a way where you can actually, you know, they're Capable of running multi-tenant workloads. And one of the things that we're doing with DRA, which was actually demoed earlier today in a presentation from one of my colleagues, is the ability to dynamically create these MIG devices Based on workload requests that are coming in. And so if you have a smaller workload, or a workload that needs a smaller GPU, you put a request for that in, and the system will figure out where there's a GPU that's capable of being sliced Up into this smaller piece. It'll create that smaller GPU on it and then start running you there. And then once your job's done, that GPU becomes a free agent again to potentially be a full GPU or some other combination smaller MIG devices over time. ([Time 0:29:30](https://share.snipd.com/snip/380d663c-3830-472c-9b98-97c352ab8b80))