Do All Your AI Workloads Actually Require Expensive GPUs?  ## Document Notes ## Highlights - **Axion Targets Cost And Power Efficiency** - Google built Axion CPUs to deliver higher performance at lower cost and power for large-scale cloud workloads. - Customers can get more compute and energy efficiency by shifting certain workloads from x86 to ARM-based instances. Transcript: Andrei Gueletii Yeah, absolutely. Thank you. So Google has always had the history of custom silicon. We are dating back many, many years through our famous TPUs that were specifically designed to handle infrastructure for machine learning. But we saw a shift in data center needs and also a shift in public consumption. People wanted to have more performance at the lower cost, at the lower power consumption, so more data centers can have more compute. And as a result of it, we've invested many years ago into our first generation machine, which was ARM-based and called a T2A as an experiment for people to give it a try. But now we're seeing that our customers are ready to shift over to ARM, take advantage of better performance, pay less for the same performance, still get great reliability, and effectively Get more for less. ([Time 0:03:34](https://share.snipd.com/snip/53be143f-45de-45c3-b5c5-eb514a6bda20)) - **Right-Size With Custom Machine Shapes** - Use N4A custom machine shapes to right-size memory and CPU per workload instead of fixed sizes. - Tailor instance shapes to microservices and general-purpose workloads to improve price performance. Transcript: Gari Singh Thing from the N4A perspective, from my perspective, especially coming from the kind of Kubernetes side of the house, right, where people are very, love to mix and match shapes of your Pods that you're deploying out there. I think the great thing about the N4A really is, we've introduced these things called custom machine shapes, if you will, or types, allowing you to basically mix and match how much memory And RAM you want, so no longer necessarily having fixed sizes. I know we'll talk a little bit more probably later about some FinOps and things like that, but that's really one of the great benefits. And of course, it's just a general purpose, not just, but for general purpose workloads, right? So you're getting price performance for your standard microservices type workloads, which makes it super simple to switch it out for a majority of the applications that people are Using today. ([Time 0:04:40](https://share.snipd.com/snip/47635caf-1225-4df4-8661-f9f1bebfcec0)) - **Platform Engineering Meets FinOps** - Platform engineering teams must optimize centralized platforms for price performance and FinOps as they scale. - Choosing cost-effective compute under the platform improves developer experience and overall efficiency. Transcript: Gari Singh Yeah, I think, you know, a lot of times, so there's multiple pieces, I think, to platform engineering. When it first sort of starts out, people are building, like what's the developer experience, what's the overall kind of platform that we're running on. But over time, as it grows, and if you're successful, one of the ultimate things that comes on is price, performance, fin ops, et cetera, becomes one of those key things, like you're Supposed to be building this centralized platform, optimizing all its usage, what's the cost of this, how are you optimizing price performance, how are you abstracting this from The developers. ([Time 0:05:46](https://share.snipd.com/snip/184f8046-3592-4fda-8c4f-0df6c5370511)) - **ZoomInfo's 60% Price-Performance Win** - ZoomInfo publicly reported a 60% improvement in price performance after moving workloads to ARM instances. - Data analytics and ETL pipelines are natural fits for ARM-based CPU instances in the cloud. Transcript: Andrei Gueletii Yeah, I believe they're our first reference customer who has gone publicly to say that they saw a 60% improvement in overall price performance ratio. It's also worth noting, while they are one of the customers who is coming forward, generally speaking, data analytics workload, ETL pipelines are all CPU heavy driven, and they're A perfect fit for an arm architecture so a lot of our partners in the ecosystem who depend on Kubernetes to run their ETL pipelines have adopted C4A's and then 4A's for improved performance As well as our own native cloud workloads such as Dataflow which is our ETL pipeline, product platform as a service, as well as Dataproc, do take advantage of C4A, and we do see a performance Increase as well in that particular space. ([Time 0:14:31](https://share.snipd.com/snip/a37fef88-2763-44c6-a71c-a8eb26a761c6)) - **C4A Suits Databases And In-Memory Caches** - C4A instances provide dedicated high-frequency physical cores and high-memory options suited to databases and caches. - ARM's memory and core model yields faster retrieval and better per-core performance for DB workloads. Transcript: Andrei Gueletii Me out a little bit about the magic of the memory of the memory on our secret sauce which is shared by you but again the magic starts for the databases mostly most database engine if you Look at the the commercial ones as well as the non-commercial ones they don't necessarily benefit a lot from a hyper thread so a c4a instance is always dedicated high frequency core, Dedicated core. The second component of this in C4A do allow you to have high memory instances, which allows you to cache a lot of stuff in RAM. And due to the power of the memory, that is a secret sauce of ARM, it allows you to also retrieve it a little bit faster. Yeah, yeah. Pranay Bakre And I think, yeah, add on to that, right? So the one point that Andrey mentioned about hyper threading. So with ARM, like I said in the beginning, that you get like a physical dedicated core for any of your workloads, right? So you essentially get more compute to kind of play around, and it's efficient by design, so you get essentially better performance. So that kind of contributes to that. ([Time 0:15:47](https://share.snipd.com/snip/88aa3d14-e37d-4738-b8a5-27d473714e4e)) - **Use CPUs For Many Inference Tasks** - Decouple workloads: run small-model inference, batch jobs, embeddings, and pre-processing on CPUs instead of GPUs when latency and VRAM needs are low. - Use GPUs only when model size or low-latency requirements justify the cost and memory footprint. Transcript: Speaker 3 We were talking about, I was talking a little bit earlier. It's come up a few times at the conference already and how inferencing is going to be done by millions and millions of companies at some point. We'll all be kind of thinking about how do you do inferencing. Andrei Gueletii So I'm thinking about how does Axiom fit into the AI ML world. Yeah, absolutely, great question. So the latest, as you probably know, is agents. Everybody wants to run an agent. Yeah. Everybody wants an LLM somewhere or some sort of machine learning model. What not everybody's realizing is part of your agents and your agent orchestration systems, agents will have tools, and those tools will call our things, potentially just classical Machine learning models that technically don't always need to have a GPU behind them. Secondarily, from an inferencing perspective, GPU comes with VRAM, and sometimes that VRAM is not sufficient for your workload, or it's the opposite, too big. Cloud, after all, is all about efficiencies and gains. So if you're seeing an underutilized GPU, there's a lot of workloads that can benefit, specifically smaller LLMs that could be influencing on CPUs. If you don't have a clear need for low latency out of a response and more of a job that relies on batch, you will potentially see that the C4A instance will be a much cheaper alternative Versus you renting a GPU. Similar fashion, if you are running tools or you're running a RAG or if you're running embeddings, a CPU is just as efficient to be able to do the pre-work, the pre-queuing before it sends Data to the LLM, and again, you're kind of decoupling your tools and processes and only using the right things when it matters. And not necessarily just throwing everything all to the GPU or suffering the performance downgrades of a CPU-based inference. ([Time 0:19:14](https://share.snipd.com/snip/06fc2160-1735-4678-8a67-d5a0873f6118)) - **ARM Excels Beyond Small LLMs** - Many traditional ML workloads like recommendation engines are highly optimized for ARM and gain lower latency on Axion. - ARM performance and efficiency fit a broad spectrum of AI/ML use cases beyond huge LLMs. Transcript: Pranay Bakre Would you like to add anything to that? Yeah, I think he covered it pretty perfectly. But I think one more thing is that with inferencing, and especially that's the message we want to land, that whenever you hear AI, you don't need to run a 3 billion parameter model on a GPU. You can do that very well on a CPU, and you can go beyond that. So inferencing is well supported. So one more thing in addition to that, other than LLMs, is about any traditional use cases, or MLPer, for example, recommendation engines like DLRMv2, they are highly optimized for ARM, so you get much lower latency with Axion, and you get better performance in there. ([Time 0:21:04](https://share.snipd.com/snip/8a438533-99b3-4d3d-a13c-7c902978be60)) - **ARM Spans Edge To Cloud** - ARM is already widely used at the edge and on devices like Pixel phones to run local models efficiently. - Google supports multi-tier deployments from phones to distributed cloud combining CPU and GPU where needed. Transcript: Andrei Gueletii I just wanted to add also, Google has been famous for the use case that you're describing. I believe everybody knows we own Android. Right. Android has been making strides and running local LLMs on top of cell phones. Right. So if you look at the Pixel 9 and now 10 they will run a local version of our model paired with a CPU that's ARM based. Yes. And most importantly we also have MediaPi which will allow you to adopt the same technology whether it lands on top of an Android or an iOS. And we do see quite a few customers asking for this use cage. Google's belief in that particular space is multi-tiered I would say. So there are certain things that would be great on top of a cell phone. Then more powerful devices might be needed from our partners. ([Time 0:23:08](https://share.snipd.com/snip/01664f05-094e-4423-8457-8cac49850c60)) - **Leverage Auto-Provisioning To Reduce Waste** - Use custom machine types and GKE node auto-provisioning to avoid over-provisioning and reduce costs. - Annotate pod specs and let the cluster pick the best ARM node shapes to match real pod resource needs. Transcript: Speaker 3 What makes them interesting for using them in the M4A? Gari Singh Yeah, and we were, I think we alluded to this a little bit earlier, but, you know, again, we've talked a lot about what people really want to be able to do is be able to right size, to compute For their kind of workloads, right? And workloads come in all kinds of different sizes and shapes, right? And you also, as you mentioned, we want to be able to provision, you know, more on demand, right? And not have to pre-provision, right? So with custom machine types, we're able to basically kind of mix and match that sizes, right? You're no longer limited to fixed shapes. This makes it really easy for us, for like from a Kubernetes perspective, to pick a node that's going to perfectly match whatever pods we needed scheduled, which is going to, of course, We already talked about the price performance, in general, but now we're reducing any extra spend, you know, extra cores or over-provisioning, and then we're able to provision more And more of those. And the other nice thing, I guess, to chop up the Kubernetes side on there, is that one of the big things that we have been working on on for a long time, is what we call node auto provisioning, Right? And very much like it used to be in the old days, where people picked, you buy 15 servers from somebody that are of this shape, and you just provision them out there, that's kind of how people Look at things from a Kubernetes perspective. I'm going to use this node, and it's going to have 32 ARM cores and 64 gigs of memory, even if I only deploy a pod with one core or whatever and I've got 63 extra cores left. So we've always been the ability to say, hey, based on your workload, we will find and pick the right machine shape for those workloads that are in the backlog. Now we have the ability to pick, to customize that shape even more, which is going to greatly reduce your cost. ([Time 0:24:11](https://share.snipd.com/snip/60457859-a3b1-401e-b66f-c59bbb547d7d)) - **Make Images Multi-Architecture** - Build multi-architecture container images or recompile binaries to support both x86 and ARM64 for seamless deployment. - Migrating to ARM is generally straightforward and many Google services already run on ARM internally. Transcript: Pranay Bakre It's single line change, can I say that? No, just wondering. No, but it's very simple. I think the notion comes from it where people who are on the legacy architectures have been in the past. They feel that resistance moving on to a new architecture. There's going to be a much more heavy lift compared to a straightforward answer. It is, in majority of the cases, is a straightforward thing. If you are building your images in a multi-architecture fashion, that can go to both x86 and ARM64. You can deploy it seamlessly. If you are cross-compiling your applications from one architecture to other, there are, let's say, a Golang-based application or a Python-based application. You just have to recompile it, make sure that there is an ARM binary that can exist and can be deployed. It's not a heavy lift. We did a workshop yesterday, essentially which we were talking about, that migrating to ARM or Axion, it's not that hard. You just need to start and then just go forward. And we have so many resources. We can help with that transition too. And I'm sure the Google guys can. Andrei Gueletii Yeah, I just wanted to also note that we're absolutely agree with you saying Google has been on its journey to transform to ARM as well. A lot of our services that we pick to battle test ARM for our native services have gone through your journey. We're seeing tremendous results. So it's not only you consuming it in Kubernetes or other services in the cloud. ([Time 0:26:23](https://share.snipd.com/snip/a41fbdee-0950-4f1f-b537-ef1816c6f420))