Arm’s Open Source Leader on Meeting the AI Challenge  ## Metadata - Author: [[The New Stack Podcast]] - Full Title: Arm’s Open Source Leader on Meeting the AI Challenge - Category: #podcasts - URL: https://share.snipd.com/episode/04ca0392-fcb7-4a3d-8fae-ae10ccd9c9ae ## Document Notes ## Highlights - **Arm's Open Source Commitment** - Arm has shifted from being a passive open source user to an active contributor and leader. - Open source is the default approach at Arm, requiring special justification for any proprietary software. Transcript: Andrew Wafaa I would say, maybe probably about a decade ago, Arm was always viewed as a pretty bad open source citizen. We were a consumer of open source, as everyone is. But we never really participated as we should have. And definitely over the last 10, maybe slightly longer, we have upped our open source game. We are very much an active member of the open source community. We employ a number of maintainers across various different projects, kernel, GCC compiler, LLVM compiler, and a number of other projects. And for us, open source is the default for our software, right? If an engineering group wants to release some software, proprietary, whatever else, that's the exception. They have to justify why it's not open source. I head up our OSPO. We call it an open source office. We drop program from it because open source is the default. It's not a program. It is bread and butter DNA that you have to do in open source. ([Time 0:02:20](https://share.snipd.com/snip/bab2a42f-959e-42d3-bd31-388e103ddeb5)) - **Arm's Open Source Project Focus** - Arm strategically contributes to kernel, compiler, and AI projects to strengthen its architecture support. - PyTorch is becoming a central open source AI ecosystem, akin to Kubernetes in scale and community. Transcript: Andrew Wafaa So we're heavily invested in the kernel and the compiler because without kernel and compiler, support for our architecture is somewhat going to be limited. So we have to invest in those two projects especially. But we work across the board, everything from firmware, kernel, compiler, right up the way to applications. We've contributed code to things like new vector from a security perspective because it's something that we want to use. What better way to use it is contribute code, make sure it works well, give that back to the community and engage like that. We've contributed to Kubernetes, enabling ARM support, work with the community, enabling CICD. I think that's probably an area of where we can make the largest impact is around the CICD space. Because a lot of projects don't have access necessarily. I mean, it's easier now with GitHub actions, you can actually get runners on ARM direct from GitHub. But until late last year, you couldn't do that easily. So we would stand up runners to plug into people's GitHub projects and such like. So we actively participate in all sorts of areas. We contribute to things like the UXL Foundation, OneDNN, the old Intel OneAPI stack. We contribute heavily to, like I say, OneDNN. We're working towards OneDAL with partners. There's a number of other pieces there. PyTorch, we've contributed a lot to PyTorch. I think PyTorch is going to be an area of significant contribution, not just from ARM, but from the wider ecosystem. I wouldn't be surprised if PyTorch became the new Kubernetes in that respect. So, I mean, there's already the PyTorch foundation. I think it's going to grow a much larger ecosystem around that initial project of PyTorch. You're going to see a lot of other things that interface with PyTorch, interact with PyTorch, leverage PyTorch, building that ecosystem up around that. ([Time 0:03:36](https://share.snipd.com/snip/72c23eeb-0697-415b-b7ef-7facdfe84f18)) - **Support PyTorch Project Health** - Enhance continuous integration and delivery (CICD) to improve project health and community involvement in PyTorch. - Grow the maintainer base outside Meta to ensure PyTorch's long-term openness and sustainability. Transcript: Andrew Wafaa So PyTorch, it's a Linux foundation project. It has its own foundation. It was donated by Meta. And when a project goes from being a solely controlled project to being a truly open project, there's a lot of work involved. And to Meta's credit, they've done a huge amount of work to get that project opened up, get whole raft contributions, etc. I think there are still some steps to go, but you've got to give people the kudos that have done that work and they're doing it. I think from PyTorch's perspective, enhanced CICD I think is going to be a big thing. That's something that affects all projects at the end of the day. It would be great to grow the maintainer base outside of Meta as it stands, just to ensure that it's an open source project and so there's no single benevolent dictator, if you will, whatnot. Not that I'm saying Meta's being that evil corporation or anything like that, far from it, but just from a project health perspective. But that takes trust, that takes a bit of time, so we understand it's not going to happen overnight. ([Time 0:07:53](https://share.snipd.com/snip/e91cb795-4c9d-4e0d-a443-b6fbd0f18572)) - **CPUs vs GPUs for AI Inference** - CPUs, especially with Arm's scalable vector and matrix extensions, are often better suited for AI inference than GPUs. - CPUs offer more flexibility and can reduce the need for costly GPU fleets in AI workloads. Transcript: Andrew Wafaa So it's kind of horses for courses, right? There's the right tool for the right job. Part of the issue with AI is that we don't really know what that right job is. We have an idea, we've got implementations of stuff. There's the big move from LLMs over now to agentic workflows, etc. But I do genuinely think that for the bulk of AI workloads, especially in the inference space, your CPU is absolutely the right tool for the job. There's a story out there around GPUs. It's a well-established story, a very well-told story. It's very convincing, but it's not entirely true in my opinion. I think CPUs are just as good as long as you can do vectorization and matrices, those sort of stuff, then your CPU's fine. A GPU's excellent at doing vectorization and matrix workloads because that's all what it can do. A CPU can do far more, and things like Clyde AI and others can actually leverage some of the architectural features that we are putting out there that enhance that vector support and Matrices support. We've got a number of new architectures or new technologies coming out in our architecture. So we've got SVE currently, SVE2 is coming out. We've got SME also in the pipeline. So, SVE and SVE2, that's scalable vector extensions and SME, scalable matrix extensions. And so, we're providing the tooling and the capabilities there from a CPU perspective that minimize the amount of GPUs that you need to buy. There are going to be some workloads where actually, yes, a GPU is needed. ([Time 0:09:22](https://share.snipd.com/snip/7ffa33ce-dbbb-4a4b-bda6-c07f38212c42)) - **Migrate Workloads to Arm** - Use Arm's total migration program to move workloads from incumbent architectures to Arm easily. - Leveraging cloud Arm instances can optimize performance and reduce costs. Transcript: Andrew Wafaa So we've got a number of programs out there to help people with developing an ARM. We understand that we're coming from left field, if you will, somewhat. The incumbent architectures are very similar, so it's quite easy to work with one or the other kind of thing. It's a similar architecture set, that sort of stuff. In the context of Kubernetes, where we are at KubeCon, we've recently just announced our total migration program. So that allows and helps people to migrate their workloads from incumbent architectures to the ARM architecture, whether it be on AWS Graviton, whether it be on Google Axiom or Microsoft Cobalt within the Azure and GCP space, respectively, and whatnot. So that's a recent announcement that goes into a lot of detail and helped people to actually move their workloads to the ARM architecture. Because we know it's somewhat different. There are some quirks there. ([Time 0:11:56](https://share.snipd.com/snip/e4920630-f822-4441-9df6-b2ec11c0a11d)) - **Arm's Broad Architecture Benefits** - Arm's architecture supports a broad range of devices, from mobile to supercomputers, enabling power and cost efficiency. - Its scale-out capability makes it a natural fit for cloud and container workloads. Transcript: Andrew Wafaa From a use case perspective, I would say all use cases should move to ARM. Heather Joslin Of course, of course. Andrew Wafaa But, yeah, it's's again, horses for courses. If you look at a lot of the public cloud vendors, running on their ARM-based instances will save you significant amounts of money. If you're running on premises, you know, in your own data center, et cetera, it will save you money because power and cooling will be less. You can pack the racks far more densely with compute power than you would with other architectures. And so a big driver is cost efficiency. But also the scale-out capabilities of the architecture makes it a natural fit for those sort of workloads, especially in the container space. It's easier to scale out your workloads without too much problem, and it's a much more efficient way of using the compute available there. ([Time 0:13:04](https://share.snipd.com/snip/fe0c8b74-7365-4c67-bd60-a47fe5b18b6a)) - **Open Source Maintainer Challenges** - The maintainer problem in open source intensifies with fast-evolving AI projects like PyTorch. - Expanding maintainer bases in projects like UXL Foundation is crucial but takes time and trust. Transcript: Andrew Wafaa Mean, the maintainer problem within open source is a very real issue. And, you know, maintainers get to where they are somewhat through trust and experience and whatnot. Time doesn't stand still. Everyone gets a little bit older every day kind of thing. And I think seeing expanded maintainership in, like I say, PyTorch is an example. UXL Foundation is making great moves to expanding their maintainer base. Again, that was a single vendor project. It was Intel for a long time. They've completely opened up and Armour now on that maintainer list. Hopefully others from the likes of Fujitsu and Google and that can also help contribute there. I think that there's scope, but it takes time, unfortunately. And when you have an environment like AI that moves so quickly, it can be super frustrating because you just can't keep up with it. And it's like, just give us another five, ten minutes, and then we might be able to do something. Stop changing every minute. Heather Joslin Yeah, stop innovating everybody. ([Time 0:15:49](https://share.snipd.com/snip/24d5bff0-47df-43fe-8e6e-6cbc02d93f16))