Google Cloud Therapist on Bringing AI to Cloud Native Infrastructure  ## Metadata - Author: [[The New Stack Podcast]] - Full Title: Google Cloud Therapist on Bringing AI to Cloud Native Infrastructure - Category: #podcasts - URL: https://share.snipd.com/episode/ef3713ab-9fc0-468f-b773-6a3186e00099 ## Document Notes ## Highlights - **Bobby Allen's Cloud Therapist Role** - Bobby Allen created the title "cloud therapist" to emphasize listening over pushing products. - He compares it to Spider-Man's dual identity, highlighting empathy in tech roles. Transcript: Bobby Allen So the cloud therapist thing, by the way, I actually do have cloudtherapist at google.com. That's my email alias. But cloud therapist is a title that I made up partly because I feel like we're making up a lot of stuff as we go along now in tech. So think of this as like, this is scar tissue from doing 10 years of startups before coming to Google. And what I found was a lot of product managers, which is my official job, right? So think of like, think of cloud therapists like Spider-Man and group product managers like Peter Parker, right? Like my real identity. But a lot of people that are product managers, Frederick, try to solve everything with technology. And I want people to have a bias that I'm there to listen most to what their problem is, not trying to make everything feel like where my product is a solution for everything that ails them. I want to listen first and try to meet them where they are. So it lets me define myself as opposed to leading with, okay, I'm a product manager for GKE and that's all that I care about. No, I care about you. And I may not be the best person to help you, but I want to listen and get you connected to whoever else on the Google team in addition to me or besides me can help you get where you want to go. Do they accept that therapy usually? They do. And actually some people ask for therapy. They say, I need to lay on the couch and cry first because the other part of that, Frederick, is there are a lot of good and bad things that happened before Google ever showed up and before I showed up. And like respecting the history is important. So one of the mantras I have, the short version is technology is the easy part of tech. The long version is tech is the easy part. People are the best part. Behavior is the hard part. Humility is the worst part. And so I want to respect all the other things. Think of technology like the part above the waterline and everything else is like the iceberg below that you can't see. ([Time 0:00:51](https://share.snipd.com/snip/918c34fc-560e-46bc-8412-65823eebe706)) - **GKE's Foundation for AI** - GKE underpins AI workloads by offering scalability, elasticity, and support for GPUs and TPUs. - Kubernetes' built-in capabilities like high availability make it ideal for AI deployment. Transcript: Frédéric Lardinois A little bit of AI. Bobby Allen Yeah. So, I mean, GKE is a phenomenal platform for AI in so many ways, directly and indirectly, right? So let's hit the indirectly first. So indirectly, GKE powers things like Vertex AI, right? So Vertex AI runs directly on GKE, which to me is part of our superpower, Frederick, because when you think about us drinking our own champagne, if you will, we have customers like DeepMind And like Vertex AI that leverage directly or indirectly our technology and our expertise. So they push us to scale and to mature and to make things robust to support the things that they're doing. And then directly, we have major customers and logos, I won't go through naming them, that run mature AI practices, training and inference on top of GKE and have for years and have kind Of grown with us over the years. So it's really hard to get away from GKA in terms of what happens, whether it's gaming, whether it's healthcare, financial services, retail, life sciences, obviously a lot of digital Natives and startups. You really kind of trust GKA to be the way that they orchestrate a lot of the newer types of resources they need. So I could go on and on, but the short story is we want to be able to service a lot of different people. And GKE is such a great platform because you can tune it or customize it to what you need for your particular use cases. Frédéric Lardinois Right, because the best way to think of it is just as a primitive that underlies everything else, right? Exactly. How has this AI use case changed, though, how you're thinking about GKE? Bobby Allen It's changed how we think about GKE because I'm going to borrow from some of our killer engineers, people like Tim Hocken, Clayton Coleman. So if you look at where we were in the computing industry before, if you go way back, right, we had specialized hardware. And then we went to more commoditized hardware. And now we're swinging back to specialized hardware again. And again, GKE is kind of a thread over the past 10 years as we celebrate 10 years of GKE and 10 years of Kubernetes last year. That's been a consistent theme as the pendulum has kind of swung back and forth that you need an orchestrator to organize those resources and use them efficiently. So AI is special, but it's not. So I'm kind of contradicting myself because we see AI as another type of modern workload. So if you think about things that are cloud native, that are scalable, that need to be elastic, AI has a lot of those characteristics. But it also has some other distinguishing characteristics like leveraging hardware accelerators like GPUs and TPUs. Sure. Kind of sitting at the intersection of like high availability, HPC and a lot of innovation. GKE underpins all well because a lot of the things that GKE naturally does, I mean, it's literally built for a lot of the types of things we want to do, defining your environments, defining Your explicit configurations of what you want to do. Like there's so many primitives that are built into Kubernetes that are just tailor-made for people to tune the dials that they need for their AI workloads. Because the one thing that we are seeing a lot, you need a lot of control when it comes to AI, right? So if I'm doing a certain level of training, I want to have, you know, co-located GPUs that are grouped into a certain configuration or a certain kind of cluster. ([Time 0:02:45](https://share.snipd.com/snip/f4eb02bd-ab16-44fb-b6c8-e91d4e63927f)) - **AI Is Still In Early Stage** - AI is still in early stages, similar to a teething infant in tech maturity. - There's plenty of room for newcomers to engage with AI development and deployment. Transcript: Bobby Allen My analogy for this, Frederick, is I used to say Cloud was a teenager that just learned how to drive. If that's the case, AI just got potty trained. Like it's real, it's maybe teething. It's somewhere in there. It's real, real early. I think that's great because I hope your listeners hear that if they're not deep into the AI rapid hole yet, it's early. And so there's definitely going to be room and on ramps for them to get involved and bring new expertise. But a lot of what happens is the things that we're doing now definitely connect to the past. ([Time 0:05:54](https://share.snipd.com/snip/7ab1a6b2-e632-4060-a54b-ddcbf5a354c9)) - **Mastering The Remix Strategy** - Master the remix: combine new AI innovations with proven cloud practices. - Balance control and automation by offering both customizable and hands-off AI workload options. Transcript: Bobby Allen Prior to AI, we had to solve issues related to scaling. We had to solve issues related to control, related to security, related to different modes of operation. So one of the things we see is because GK is such a wide platform, the customers are not a monolith. Some want full control, expose every single dial, and then other customers want sensible defaults and kind of automatic configuration baked in. And I think AI is very similar to that, right? Some people want everything at their fingertips to control and mess up and break as they please. And other people want us to protect them from themselves. That's a history that started before AI. So what we're seeing, for example, you have people that are running AI workloads on GKE Standard and GKE Autopilot, sensible defaults, best practices built in, and standard where They want full control. So that's one of the things that we see. The connection between the workloads running in GKE and the other kind of adjacent systems, storage, networking, security, databases, all of those things are things that we're kind Of doing more of in the AI world. And then honestly, just being responsive to customers, right? Interesting is a lot of customers are figuring out what they want. I think prior to AI, if we think of GKEs, product managers, and engineers almost like a chef, right? Pretend we're Wolfgang Puck for a second. What do you want for dinner? Well, you would tell us and we would make it. Now there's a lot more experimenting, right? Let me bring you a slew of entre or slew of appetizers or tapas, if you will. So you try those different things and then, yep, I want you to double down on this. That's a lot of what we're seeing now, Frederick, is that people are trying or taste testing different things. And then we're maybe going deeper into certain things with one customer. ([Time 0:07:00](https://share.snipd.com/snip/cc2ed3aa-65b7-4fe4-ae66-ff321b1d9977)) - **AI And Cloud Migration Overlap** - AI adoption often coincides with cloud migration; many prioritize new AI projects over legacy migration. - AI use spans building models, integrating AI into development, and embedding AI in operations. Transcript: Bobby Allen I think they're happening at the same time. I mean, everything's been accelerated in the past couple of years. So, yes, there are companies that are moving to the cloud. I think there are companies that have almost paused the migrations and the rationalizations because they figured that I don't have time to fix all the old stuff. I need to build the new stuff. So I think if anything, they've accelerated the newer things that they want to do because everyone's got an AI mandate. And if I could, Frederick, I think the thesis that a lot of us are trying to solve is how to get value from AI. And actually, maybe that's a good point to step back. I bucketize AI into kind of three different broad strokes so that it's not as much of a junk drawer. I think there's what you build, how you build, and how you operate, right? There are AI workloads or AI models that you might be building, right? Think training, inference, fine-tuning. There's how you build, where you're baking AI into your software development life cycle, and there's how you operate, where you're baking AI into how you operate your platform. All three of those are things that are advantageous to customers, but the how you operate, for example, is one of the low-hanging fruit pieces we're seeing because even if I'm not running AI workloads, I want AI assistance to help me optimize costs or find security gaps to help me optimize troubleshooting. So depending on where people are, they may play in all three of those buckets or just one of those right now. ([Time 0:09:03](https://share.snipd.com/snip/b40ca3ce-4cfb-428d-927a-70411df33e79)) - **Multiple AI Options Within Organizations** - Different users within organizations adopt diverse AI approaches simultaneously. - GKE provides control to platform teams while Vertex AI and Cloud Run meet less technical needs. Transcript: Frédéric Lardinois How do you think about that and where does GKE fit in there in the overall platform? Bobby Allen I think GKE is the most control with the least technical debt, in my opinion, right? Because it's future-proofed in so many ways because we know it can scale. We know it can grow. We know the commitment that we have in the open source community where Google has continued to invest at a very aggressive rate in the future, even with things that we announced at KubeCon Last week. And so I think it gives you control, scalability, and headroom for the future. But at the same time, to your point, we do want to acknowledge some people want batteries included, right? I don't want to put together the model car. I want to buy the thing that's fully there like a Vertex. I may want something that's a quicker path to get started like a Cloud Run, which now supports GPUs to do inference, for example. What's interesting, Frederick, I think before talking to more customers, I would have said people are choosing this, this, or this. And the reality is we have people that are choosing two or three of those choices at the same time. Because as our customers are not monoliths, the roles in that customer are not monoliths, right? So the platform team might have a GKE first philosophy because they want to stand up a platform to standardize and harden and so forth. But there may be a lab that wants to do things with the cloud run, or there may be an HR marketing department that wants to experiment with the Vertex because they might not have developers Or platform folks. ([Time 0:10:43](https://share.snipd.com/snip/a9815765-0ca3-4f76-8e20-4e9c69cd1eff)) - **GKE Evolving For AI Routing** - AI workloads demand specialized orchestration due to resource heterogeneity and performance variability. - GKE evolves into a model router, routing requests to AI models by purpose rather than brand. Transcript: Frédéric Lardinois They put different stresses on the whole system. They do. Is AI kind of one of the main drivers right now for how you're thinking about what's next? Bobby Allen Absolutely. I mean, AI is a huge driver for us in terms of product roadmap, in terms of how we talk to customers. And it's a balance, Frederick, because what I'll say to try to keep us on is there are a lot of customers who are talking about AI but not doing AI yet. Explain, explain. So what I mean by that is, let's talk training for a second, right? If we had 100 customers, maybe five of them are training their own models from scratch, right? That's what we hear a lot and what we think about a lot when we talk about AI. Obviously, the Googles, the Metas, other companies like that. Honestly, that's a higher number than I would have thought because not many people are training anymore at this point. Somewhere between 2% and 5%, right? Being generous, right? People that are doing training or have aspirations. And so what's interesting is those folks can dominate our mind share because they have so much momentum behind what they're doing. You know, they're putting out new models on Hugging Face and so forth. But the other 95 to 97 customers out of that hundred are not doing their own models, right? They want to know things like, how do I evaluate which model is for me? And so let's go down that path for a second. If I'm an enterprise customer, I don't want to train models. I want to experiment with models. I want to figure out how to expose those models to my team. And so that's where concepts like model as a service come into play. So the thing that we're seeing is my thought about this is a lot of enterprise customers will have a small team in many models. So think like an architectural team that evaluates a bunch of different models, and that may evolve into many people fewer models, right? So a Tiger team may evaluate 100 different models, but you're not going to expose 100 different models to 10,000 developers necessarily, right? And so that scratch test or that sanity check that the Tiger team is doing may yield to, we're going to have five vetted models that we're exposing to our development community. But I think what's also going to happen is models are going to be identified by purpose, not by brand, if you will. I want a model for image generation. And so on that model as a service offering, GKE will support the ability to route you to the model based on the purpose, not necessarily the name. So maybe you want meta, maybe you want Gemma or Gemini. Maybe you just want to say, hit this inference endpoint if you're a developer for image generation, and I don't care what's underneath, right? GKA will allow you to set up that scaffolding to give people the output they want without knowing the brand or the version of the model. We're announcing a bunch of things even here at Next around how to do smarter routing of models, lore adapters. One big thing we're excited about is the GK Inference gateway to do smarter routing, and we can go deeper into that later maybe. ([Time 0:12:17](https://share.snipd.com/snip/b66eab68-61c3-43f6-b893-56c30e308f93)) - **Optimize AI Inference With GKE** - Use GKE inference quick start to select hardware and deployment that matches model performance needs. - Employ GKE Inference Gateway for smarter, LLM-aware load balancing to improve app responsiveness. Transcript: Bobby Allen So we have two major things. So we have kind of this inference capabilities bucket. Think of that as the label. And there are two big things underneath that that we're unveiling. Both I'm really excited about. One is what we call the GKE inference quick start. So let me explain this one. So if you go to deploy a model, DeepSeq, Llama, Gemma, et cetera, it's one thing for someone to tell you what resources do I need to run that model. But what about the other side of that? What are you going to get in terms of metrics, queries per second, time per hour per token, time to first token, et cetera. Right now, there's really not very prescriptive guidance around that until the GKE inference quick start. So what you're going to get now is Google's got a regular set of benchmarks that we run against popular models and hardware. So you'll see something, Frederick, like if you want to run DeepSeq and support this level of output tokens per second or time per output token, you need to deploy this, this, or this. So think of it as like t-shirt sizes. If you go H100, you're going to get this. A100, this. TPUs, this. So that's going to be baked into this kind of prescriptive set of benchmarking. So if you deploy these resources, you've got choices, but you know what you're going to support when people do inference against those. I love that because now we have a sense of, are we getting what we pay for, right? Frédéric Lardinois So I can make those trade-offs and decide where I pay more, be faster, pay less, take longer. Bobby Allen And you might have a different profile in development where you're playing versus in production where customers are really hitting this at scale. So the inference quick start is giving you a push in the direction of what good looks like and honestly kind of help and protect you from yourself as a user because they don't have time To do all these benchmarks as fast as these models are coming out. The turnaround time for these is actually pretty quick. So, you know, a model might come out and Google might have something out there in two to four weeks where you can show, okay, if you want to use this model, support your data scientists Or AI developers this way. A TPU will do this. These GPU models will do this, and we're actually going to demo some of that here at the show, which I'm excited about. But then the other side of that is, let's say that you do inference against a model, and you build this application that is wildly successful. People love it. It's going viral. It's growing like gangbusters. Well, you're probably going to need load balancing at some point, right? And old school load balancing doesn't do a good job of routing traffic for models because old school load balancing assumes that most requests are created equal. Right. It would be like if this were fast food and everyone kind of has a number one and number three or number five from their favorite restaurant. Well, AI is more like I need Thanksgiving dinner. Yeah. And then someone else orders a sandwich, right? If you have a Thanksgiving dinner order, you don't want to pile other orders on top of that because now you've got a bottleneck. I'm a big believer that model responsiveness leads to application responsiveness. Sure, sure. So slow models equals slow applications. So our GKE Inference Gateway does smarter kind of LLM-aware routing, so you don't route people to resources that are tied up with longer-running requests. And so you route them to the people or to the resources, the endpoints, if you will, that can serve those requests better. Then your end users have a better experience. Frédéric Lardinois Yeah, and then maybe combine that with a caching service or something on top, which that's its own rabbit hole of LLM caching. It is. Not going down there. Like in a way, you're almost building another abstraction on top of GKE then. Is that correct? Bobby Allen It would be in GKE. I mean, it's kind of the intersection of the runtime and the networking. So when you think about the load balancers have a little bit more understanding of the type of requests that are coming through. And we know this, right? Like in the software world, we know that a Java application has a different profile than a Python application or a JavaScript application. And I think what we're getting to is that all requests are not created equal, just like all types of code or applications aren't. And so we're applying that knowledge to how these things are routed. So the inference gateway is something that you deploy as part of GKE. You even access it from that part of the console. And what it does is you can see things like inference pools that were just announced at KubeCon last week. And you can kind of have a little bit of metrics and observability on how the gateway is kind of routing things to the different types of resources based on the nature of the request. ([Time 0:15:21](https://share.snipd.com/snip/1abea13d-1e78-4d19-9979-8e6b3ec96e4c)) - **Multiple Paths For Building AI Agents** - GKE, Cloud Run, and Vertex provide multiple pathways to build AI agents depending on scale and time constraints. - Customers’ diverse needs drive the availability of multiple AI development onramps within Google Cloud. Transcript: Frédéric Lardinois Yeah. What's the role of GKE there? Bobby Allen So GKE is one of the ways to build agents, certainly. There are people that are building agents in Vertex, in agent space, in Cloud Run, and in GKE. Frédéric Lardinois That's another where you've got a lot of on-ramps. Bobby Allen Yeah, there are a lot of on-ramps. And again, I think part of what Google Cloud acknowledges is that customers are not a monolith, right? Like, just think about, again, back to, I'm sorry to go to food because I'm going to probably make you hungry. Making me hungry at this point. Even if we said we want a burger, right? How many different opinions are there about the type of burger that people want? Personally, I'm a turkey burger guy, right? Specifically, turkey burgers or provolone or mustard cheese. That's my wheelhouse, but that may not be the type of burger you want. Frédéric Lardinois You do you. Bobby Allen Right, right. So we have this level of choice and diversity, and so Google wants to meet people where they are. The other thing is sometimes there's a progression, right? If you have a lot of time, right? I prefer to make my own turkey burgers at home. If I have time, I'm going to, I've got a hamburger press at home, actually, Frederick. I take the ground turkey and I ground it. If I have time, that's what I do. If I don't have time for that, I've got to do something different that's more convenient. So we know that people are not a monolith. And even at the same customer in the same role, depending on the time they have or how much they're on fire, they might have to do something different than they would have normally. And so I think that choice is part of what we want to embrace, that people may do something quick and dirty in Cloud Run right now. But then when they want to scale this out in terms of how people build and deploy agents at scale, they might use something more like a GK. ([Time 0:19:42](https://share.snipd.com/snip/ec6668df-f1e6-4bda-aa7f-4ce9e950222e)) - **Simplify AI Resource Management** - Simplify shifting between GPUs and TPUs with Custom Compute Class to ease resource management. - Implement dynamic workload scheduling to optimize resource availability and prevent wasted time. Transcript: Frédéric Lardinois Looking forward a little bit, without giving away your roadmap, obviously, but what are your customers looking for right now? What is top of mind for them? Bobby Allen A couple things are top of mind for them. So one, hardware accelerator fungibility. Make it really easier for me to switch from GPUs to TPUs, right? So I've been doing all this stuff in GPUs, you know, do the framework support it, do the node API support it. We have something called custom compute class or CCC that will allow you to change a few lines of code and decide that I want to scale this on TPUs versus GPUs. You can look at things like reserved instances, use something called dynamic workload scheduler, because as you know, obtainability is a big challenge in this space. So what if I want to run a job and all the resources aren't ready? DWS will collect all those resources and then kind of align to kick off the job when the resources are ready so that you don't waste time with part of what you need to finish the job. Got it. So there's so many different ways that we can really look at this. Obtainability is a big one, like I said. Resource fungibility is a big one. I think the ability to run heterogeneous environments, right? So people don't necessarily want to carve off training from inference from non-AI workloads. They want to run all of those on the same platform, GKE, and they're looking for best practices around how to do that. Frédéric Lardinois Right. To make it easier for the ops teams. Bobby Allen Because it's the same platform teams that are responsible in many ways for supporting that. So like, it would almost be like exiting one house and going in another house. Like, no, there are teenagers and adults and babies and grandparents that can all be under the same roof. How do we serve all of those folks well? That's what I think we're going to see more investments in is best practices for people to orchestrate different types of workloads. Because there are some things across those workloads that are similar, right? Security configuration, storage profiles, observability. How do we make that easy, but then acknowledge the distinction? Because even people under the same roof might have different dietary preferences, different temperature preferences. ([Time 0:21:11](https://share.snipd.com/snip/55d4e52c-9a8a-4235-96d9-685deb76dbaf))